<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">Il giorno 01 ott 2018, alle ore 09:17, Ashish Pandey <<a href="mailto:aspandey@redhat.com" class="">aspandey@redhat.com</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class=""><div class=""><br class=""></div><div class="">Ohh!! It is because brick-multiplexing is "ON" on your setup. Not sure if it is by default ON for 3.12.14 or not.<br class=""></div><div class=""><br class=""></div><div class="">See "<span style="font-variant-ligatures: no-common-ligatures;" data-mce-style="font-variant-ligatures: no-common-ligatures;" class="">cluster.brick-multiplex: on</span>" in gluster v <volname> info<br class=""></div><div class="">If brick multiplexing is ON, you will see only one process running for all the bricks on a Node.<br class=""></div><div class=""><br class=""></div><div class="">So we have to do following step to kill any one brick on a node.<br class=""></div><div class=""><br class=""></div><div class=""><strong class="">Steps to kill a brick when multiplex is on -</strong> <br class=""></div><div class=""><br class=""></div><div class=""><strong class="">Step - 1 </strong> <br class=""></div><div class="">Find <strong class="">unix domain_socket</strong> of the process on a node.<br class=""></div><div class="">Run "ps -aef | grep glusterfsd" on a node. Example : <br class=""></div><div class=""><br class=""></div><div class="">This is on my machine when I have all the bricks on same machine<br class=""></div><div class=""><br class=""></div><div class="">[root@apandey glusterfs]# ps -aef | grep glusterfsd | grep -v mnt<br class="">root 28311 1 0 11:16 ? 00:00:06 /usr/local/sbin/glusterfsd -s apandey --volfile-id vol.apandey.home-apandey-bricks-gluster-vol-1 -p /var/run/gluster/vols/vol/apandey-home-apandey-bricks-gluster-vol-1.pid -S /var/run/gluster/1259033d2ff4f4e5.socket --brick-name /home/apandey/bricks/gluster/vol-1 -l /var/log/glusterfs/bricks/home-apandey-bricks-gluster-vol-1.log --xlator-option *-posix.glusterd-uuid=61b4524c-ccf3-4219-aaff-b3497ac6dd24 --process-name brick --brick-port 49158 --xlator-option vol-server.listen-port=49158<br class=""><br class=""></div><div class="">Here, /var/run/gluster/1259033d2ff4f4e5.socket is the unix domain socket<br class=""></div><div class=""><br class=""></div><div class=""><strong class="">Step - 2</strong> <br class=""></div><div class="">Run following command to kill a brick on the same node - <br class=""></div><div class=""><br class=""></div><div class="">gf_attach -d <unix domain_socket> brick_path_on_that_node</div><div class=""><br class=""></div><div class="">Example: <br class=""><br class=""></div><div class=""><em class="">gf_attach -d /var/run/gluster/1259033d2ff4f4e5.socket /home/apandey/bricks/gluster/vol-6</em></div><div class=""><br class=""></div><div class="">Status of volume: vol<br class="">Gluster process TCP Port RDMA Port Online Pid<br class="">------------------------------------------------------------------------------<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-1 49158 0 Y 28311<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-2 49158 0 Y 28311<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-3 49158 0 Y 28311<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-4 49158 0 Y 28311<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-5 49158 0 Y 28311<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-6 49158 0 Y 28311<br class="">Self-heal Daemon on localhost N/A N/A Y 29787<br class=""> <br class="">Task Status of Volume vol<br class="">------------------------------------------------------------------------------<br class="">There are no active volume tasks<br class=""> <br class="">[root@apandey glusterfs]# <br class="">[root@apandey glusterfs]# <br class="">[root@apandey glusterfs]# gf_attach -d /var/run/gluster/1259033d2ff4f4e5.socket /home/apandey/bricks/gluster/vol-6 <br class="">OK<br class="">[root@apandey glusterfs]# gluster v status<br class="">Status of volume: vol<br class="">Gluster process TCP Port RDMA Port Online Pid<br class="">------------------------------------------------------------------------------<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-1 49158 0 Y 28311<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-2 49158 0 Y 28311<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-3 49158 0 Y 28311<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-4 49158 0 Y 28311<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-5 49158 0 Y 28311<br class="">Brick apandey:/home/apandey/bricks/gluster/<br class="">vol-6 N/A N/A N N/A <br class="">Self-heal Daemon on localhost N/A N/A Y 29787<br class=""> <br class="">Task Status of Volume vol<br class="">------------------------------------------------------------------------------<br class="">There are no active volume tasks<br class=""><br class=""></div><br class=""><div class="">To start a brick we just need to start volume using "force"<br class=""></div><div class=""><br class=""></div><div class="">gluster v start <volname> force<br class=""></div><div class=""><br class=""></div><div class="">----<br class=""></div><div class="">Ashish<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><div style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" data-mce-style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;" class=""><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" class="">aspandey@redhat.com</a>><br class=""><b class="">Cc: </b>"Gluster Users" <<a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Friday, September 28, 2018 9:25:53 PM<br class=""><b class="">Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class="">I asked you how to detect the PID of a specific brick because I see that more than one brick has the same PID (also on my virtual env).</div><div class="">If I kill one of them I risk to kill some other brick. Is it normal?</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s01 ~]# gluster vol status</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Status of volume: tier2</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Gluster process TCP Port RDMA Port Online Pid</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">------------------------------------------------------------------------------</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt1/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt1/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt1/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt2/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt2/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt2/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt3/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt3/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt3/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt4/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt4/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt4/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt5/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt5/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt5/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt6/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt6/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt6/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt7/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt7/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt7/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt8/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt8/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt8/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt9/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt9/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt9/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt10/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt10/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt10/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt11/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt11/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt11/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s01-stg:/gluster/mnt12/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s02-stg:/gluster/mnt12/brick 49153 0 Y 3956 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s03-stg:/gluster/mnt12/brick 49153 0 Y 3953 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt1/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt2/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt3/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt4/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt5/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt6/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt7/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt8/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt9/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt10/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt11/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s04-stg:/gluster/mnt12/brick 49153 0 Y 3433 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt1/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt2/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt3/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt4/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt5/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt6/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt7/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt8/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt9/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt10/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt11/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s05-stg:/gluster/mnt12/brick 49153 0 Y 3709 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt1/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt2/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt3/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt4/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt5/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt6/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt7/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt8/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt9/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt10/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt11/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick s06-stg:/gluster/mnt12/brick 49153 0 Y 3644 </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Self-heal Daemon on localhost N/A N/A Y 79376</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Quota Daemon on localhost N/A N/A Y 79472</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Bitrot Daemon on localhost N/A N/A Y 79485</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Scrubber Daemon on localhost N/A N/A Y 79505</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Self-heal Daemon on s03-stg N/A N/A Y 77073</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Quota Daemon on s03-stg N/A N/A Y 77148</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Bitrot Daemon on s03-stg N/A N/A Y 77160</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Scrubber Daemon on s03-stg N/A N/A Y 77191</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Self-heal Daemon on s02-stg N/A N/A Y 80150</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Quota Daemon on s02-stg N/A N/A Y 80226</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Bitrot Daemon on s02-stg N/A N/A Y 80238</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Scrubber Daemon on s02-stg N/A N/A Y 80269</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Self-heal Daemon on s04-stg N/A N/A Y 106815</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Quota Daemon on s04-stg N/A N/A Y 106866</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Bitrot Daemon on s04-stg N/A N/A Y 106878</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Scrubber Daemon on s04-stg N/A N/A Y 106897</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Self-heal Daemon on s05-stg N/A N/A Y 130807</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Quota Daemon on s05-stg N/A N/A Y 130884</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Bitrot Daemon on s05-stg N/A N/A Y 130896</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Scrubber Daemon on s05-stg N/A N/A Y 130927</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Self-heal Daemon on s06-stg N/A N/A Y 157146</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Quota Daemon on s06-stg N/A N/A Y 157239</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Bitrot Daemon on s06-stg N/A N/A Y 157252</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Scrubber Daemon on s06-stg N/A N/A Y 157288</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0); min-height: 18px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> </span><br class="webkit-block-placeholder"></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Task Status of Volume tier2</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">------------------------------------------------------------------------------</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Task : Remove brick </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">ID : 06ec63bb-a441-4b85-b3cf-ac8e9df4830f</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Removed bricks: </span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s04-stg:/gluster/mnt1/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s04-stg:/gluster/mnt2/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s04-stg:/gluster/mnt3/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s04-stg:/gluster/mnt4/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s04-stg:/gluster/mnt5/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s04-stg:/gluster/mnt6/brick</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Status : in progress </span></div></div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s01 ~]# ps -ef|grep glusterfs</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">root 3956 1 79 set25 ? 2-14:33:57 /usr/sbin/</span><span style="font-variant-ligatures: no-common-ligatures; color: #c33720" class="" data-mce-style="font-variant-ligatures: no-common-ligatures; color: #c33720;"><b class="">glusterfs</b></span><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">d -s s01-stg --volfile-id tier2.s01-stg.gluster-mnt1-brick -p /var/run/gluster/vols/tier2/s01-stg-gluster-mnt1-brick.pid -S /var/run/gluster/a889b8a21ac2afcbfa0563b9dd4db265.socket --brick-name /gluster/mnt1/brick -l /var/log/</span><span style="font-variant-ligatures: no-common-ligatures; color: #c33720" class="" data-mce-style="font-variant-ligatures: no-common-ligatures; color: #c33720;"><b class="">glusterfs</b></span><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/bricks/gluster-mnt1-brick.log --xlator-option *-posix.glusterd-uuid=b734b083-4630-4523-9402-05d03565efee --brick-port 49153 --xlator-option tier2-server.listen-port=49153</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">root 79376 1 0 09:16 ? 00:04:16 /usr/sbin/</span><span style="font-variant-ligatures: no-common-ligatures; color: #c33720" class="" data-mce-style="font-variant-ligatures: no-common-ligatures; color: #c33720;"><b class="">glusterfs</b></span><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/</span><span style="font-variant-ligatures: no-common-ligatures; color: #c33720" class="" data-mce-style="font-variant-ligatures: no-common-ligatures; color: #c33720;"><b class="">glusterfs</b></span><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/glustershd.log -S /var/run/gluster/4fab1a27e6ee700b3b9a3b3393ab7445.socket --xlator-option *replicate*.node-uuid=b734b083-4630-4523-9402-05d03565efee</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">root 79472 1 0 09:16 ? 00:00:42 /usr/sbin/</span><span style="font-variant-ligatures: no-common-ligatures; color: #c33720" class="" data-mce-style="font-variant-ligatures: no-common-ligatures; color: #c33720;"><b class="">glusterfs</b></span><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> -s localhost --volfile-id gluster/quotad -p /var/run/gluster/quotad/quotad.pid -l /var/log/</span><span style="font-variant-ligatures: no-common-ligatures; color: #c33720" class="" data-mce-style="font-variant-ligatures: no-common-ligatures; color: #c33720;"><b class="">glusterfs</b></span><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/quotad.log -S /var/run/gluster/958ab34799fc58f4dfe20e5732eea70b.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">root 79485 1 7 09:16 ? 00:40:43 /usr/sbin/</span><span style="font-variant-ligatures: no-common-ligatures; color: #c33720" class="" data-mce-style="font-variant-ligatures: no-common-ligatures; color: #c33720;"><b class="">glusterfs</b></span><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> -s localhost --volfile-id gluster/bitd -p /var/run/gluster/bitd/bitd.pid -l /var/log/</span><span style="font-variant-ligatures: no-common-ligatures; color: #c33720" class="" data-mce-style="font-variant-ligatures: no-common-ligatures; color: #c33720;"><b class="">glusterfs</b></span><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/bitd.log -S /var/run/gluster/b2ea9da593fae1bc4d94e65aefdbdda9.socket --global-timer-wheel</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">root 79505 1 0 09:16 ? 00:00:01 /usr/sbin/</span><span style="font-variant-ligatures: no-common-ligatures; color: #c33720" class="" data-mce-style="font-variant-ligatures: no-common-ligatures; color: #c33720;"><b class="">glusterfs</b></span><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> -s localhost --volfile-id gluster/scrub -p /var/run/gluster/scrub/scrub.pid -l /var/log</span><span style="font-variant-ligatures: no-common-ligatures; color: #c33720" class="" data-mce-style="font-variant-ligatures: no-common-ligatures; color: #c33720;"><b class="">glusterfs</b></span><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/scrub.log -S /var/run/gluster/ee7886cbcf8d2adf261084b608c905d5.socket --global-timer-wheel</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures: no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">root 137362 137225 0 17:53 pts/0 00:00:00 grep --color=auto </span><span style="font-variant-ligatures: no-common-ligatures; color: #c33720" class="" data-mce-style="font-variant-ligatures: no-common-ligatures; color: #c33720;"><b class="">glusterfs</b></span></div></div><br class=""><div class=""><blockquote class=""><div class="">Il giorno 28 set 2018, alle ore 17:47, Ashish Pandey <<a href="mailto:aspandey@redhat.com" class="" target="_blank" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class=""><br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><div style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" class="" target="_blank" data-mce-href="mailto:mauro.tridici@cmcc.it">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" class="" target="_blank" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>><br class=""><b class="">Cc: </b>"Gluster Users" <<a href="mailto:gluster-users@gluster.org" class="" target="_blank" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Friday, September 28, 2018 9:08:52 PM<br class=""><b class="">Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br class=""><div class=""><br class=""></div><div dir="auto" class=""><div class="">Thank you, Ashish.<div dir="auto" class=""><br class=""></div><div dir="auto" class="">I will study and try your solution on my virtual env.</div><div dir="auto" class="">How I can detect the process of a brick on gluster server?</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Many Thanks,</div><div dir="auto" class="">Mauro</div><div dir="auto" class=""><br class=""></div><br class=""><div class="">gluster v status <volname> will give you the list of bricks and the respective process id.<br class=""></div><div class="">Also, you can use "ps aux | grep glusterfs" to see all the processes on a node but I think the above step also do the same.<br class=""></div><div class=""><br class=""></div><div class="">---<br class=""></div><div class="">Ashish<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="gmail_quote"><div dir="ltr" class="">Il ven 28 set 2018 16:39 Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>> ha scritto:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class=""><br class=""></div><div class=""><br class=""></div><hr id="m_4766848556606636788zwchr" class=""><div style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:mauro.tridici@cmcc.it">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>><br class=""><b class="">Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Friday, September 28, 2018 7:08:41 PM<br class=""><b class="">Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br class=""><div class=""><br class=""></div><div class=""><div class=""><br class=""></div><div class="">Dear Ashish,</div><div class=""><br class=""></div><div class="">please excuse me, I'm very sorry for misunderstanding.</div><div class="">Before contacting you during last days, we checked all network devices (switch 10GbE, cables, NICs, servers ports, and so on), operating systems version and settings, network bonding configuration, gluster packages versions, tuning profiles, etc. but everything seems to be ok. The first 3 servers (and volume) operated without problem for one year. After we added the new 3 servers we noticed something wrong.</div><div class="">Fortunately, yesterday you gave me an hand to understand where is (or could be) the problem. </div><div class=""><br class=""></div><div class="">At this moment, after we re-launched the remove-brick command, it seems that the rebalance is going ahead without errors, but it is only scanning the files.</div><div class="">May be that during the future data movement some errors could appear.</div><div class=""><br class=""></div><div class="">For this reason, it could be useful to know how to proceed in case of a new failure: insist with approach n.1 or change the strategy?</div><div class="">We are thinking to try to complete the running remove-brick procedure and make a decision based on the outcome.</div><div class=""><br class=""></div><div class="">Question: could we start approach n.2 also after having successfully removed the V1 subvolume?!</div><div class=""><br class=""></div><div class="">>>> Yes, we can do that. My idea is to use replace-brick command. <br class=""></div><div class="">We will kill "ONLY" one brick process on s06. We will format this brick. Then use replace-brick command to replace brick of a volume on s05 with this formatted brick.<br class=""></div><div class="">heal will be triggered and data of the respective volume will be placed on this brick.<br class=""></div><div class=""><br class=""></div><div class="">Now, we can format the brick which got freed up on s05 and replace the brick which we killed on s06 to s05.<br class=""></div><div class="">During this process, we have to make sure heal completed before trying any other replace/kill brick. <br class=""></div><div class=""><br class=""></div><div class="">It is tricky but looks doable. Think about it and try to perform it on your virtual environment first before trying on production.</div><div class="">-------<br class=""></div><div class=""><br class=""></div><div class="">If it is still possible, could you please illustrate the approach n.2 even if I dont have free disks?</div><div class="">I would like to start thinking about it and test it on a virtual environment.</div><div class=""><br class=""></div><div class="">Thank you in advance for your help and patience.</div><div class="">Regards,</div><div class="">Mauro</div></div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div class=""><blockquote class=""><div class="">Il giorno 28 set 2018, alle ore 14:36, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>> ha scritto:</div><br class="m_4766848556606636788Apple-interchange-newline"><div class=""><div class=""><div style="font-family:'times new roman','new york',times,serif;font-size:12pt" class="" data-mce-style="font-family: 'times new roman','new york',times,serif; font-size: 12pt;"><br class="">We could have taken approach -2 even if you did not have free disks. You should have told me why are you<br class="">opting Approach-1 or perhaps I should have asked.<br class="">I was wondering for approach 1 because sometimes re-balance takes time depending upon the data size.<br class=""><div class=""><br class=""></div>Anyway, I hope whole setup is stable, I mean it is not in the middle of something which we can not stop.<br class="">If free disks are the only concern I will give you some more steps to deal with it and follow the approach 2.<br class=""><div class=""><br class=""></div>Let me know once you think everything is fine with the system and there is nothing to heal.<br class=""><div class=""><br class=""></div>---<br class="">Ashish<br class=""><div class=""><br class=""></div><hr id="m_4766848556606636788zwchr" class=""><div style="font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt" class="" data-mce-style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:mauro.tridici@cmcc.it">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>><br class=""><b class="">Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Friday, September 28, 2018 4:21:03 PM<br class=""><b class="">Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="">Hi Ashish,</div><div class=""><br class=""></div><div class="">as I said in my previous message, we adopted the first approach you suggested (setting network.ping-timeout option to 0).</div><div class="">This choice was due to the absence of empty brick to be used as indicated in the second approach.</div><div class=""><br class=""></div><div class="">So, we launched remove-brick command on the first subvolume (V1, bricks 1,2,3,4,5,6 on server s04).</div><div class="">Rebalance started moving the data across the other bricks, but, after about 3TB of moved data, rebalance speed slowed down and some transfer errors appeared in the rebalance.log of server s04.</div><div class="">At this point, since remaining 1,8TB need to be moved in order to complete the step, we decided to stop the remove-brick execution and start it again (I hope it doesn’t stop again before complete the rebalance)</div><div class=""><br class=""></div><div class="">Now rebalance is not moving data, it’s only scanning files (please, take a look to the following output)</div></div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s01 ~]# gluster volume remove-brick tier2 s04-stg:/gluster/mnt1/brick s04-stg:/gluster/mnt2/brick s04-stg:/gluster/mnt3/brick s04-stg:/gluster/mnt4/brick s04-stg:/gluster/mnt5/brick s04-stg:/gluster/mnt6/brick status</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> s04-stg 0 0Bytes 182008 0 0 in progress 3:08:09</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Estimated time left for rebalance to complete : 442:45:06</span></div></div><div class=""><br class=""></div><div class=""><div class="">If I’m not wrong, remove-brick rebalances entire cluster each time it start.</div><div class="">Is there a way to speed up this procedure? Do you have some other suggestion that, in this particular case, could be useful to reduce errors (I know that they are related to the current volume configuration) and improve rebalance performance avoiding to rebalance the entire cluster?</div></div><div class=""><br class=""></div><div class="">Thank you in advance,</div><div class="">Mauro</div><br class=""><div class=""><blockquote class=""><div class="">Il giorno 27 set 2018, alle ore 13:14, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>> ha scritto:</div><br class="m_4766848556606636788Apple-interchange-newline"><div class=""><div class=""><div style="font-family:'times new roman','new york',times,serif;font-size:12pt" class="" data-mce-style="font-family: 'times new roman','new york',times,serif; font-size: 12pt;"><div class=""><br class=""></div><div class="">Yes, you can.</div><div class="">If not me others may also reply.<br class=""></div><div class=""><br class=""></div><div class="">---<br class=""></div><div class="">Ashish<br class=""></div><div class=""><br class=""></div><hr id="m_4766848556606636788zwchr" class=""><div style="font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt" class="" data-mce-style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:mauro.tridici@cmcc.it">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>><br class=""><b class="">Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Thursday, September 27, 2018 4:24:12 PM<br class=""><b class="">Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Dear Ashish,</div><div class=""><br class=""></div><div class="">I can not thank you enough!</div><div class="">Your procedure and description is very detailed.</div><div class="">I think to follow the first approach after setting network.ping-timeout option to 0 (If I’m not wrong “0" means “infinite”...I noticed that this value reduced rebalance errors).</div><div class="">After the fix I will set network.ping-timeout option to default value.</div><div class=""><br class=""></div><div class="">Could I contact you again if I need some kind of suggestion?</div><div class=""><br class=""></div><div class="">Thank you very much again.</div><div class="">Have a good day,</div><div class="">Mauro</div><div class=""><br class=""></div><br class=""><div class=""><blockquote class=""><div class="">Il giorno 27 set 2018, alle ore 12:38, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>> ha scritto:</div><br class="m_4766848556606636788Apple-interchange-newline"><div class=""><div class=""><div style="font-family:'times new roman','new york',times,serif;font-size:12pt" class="" data-mce-style="font-family: 'times new roman','new york',times,serif; font-size: 12pt;"><div class=""><br class=""></div><div class=""><div class="">Hi Mauro,</div><div class=""><br class=""></div><div class="">We can divide the 36 newly added bricks into 6 set of 6 bricks each starting from brick37.</div><div class="">That means, there are 6 ec subvolumes and we have to deal with one sub volume at a time.</div><div class="">I have named it V1 to V6.</div><div class=""><br class=""></div><div class="">Problem:</div><div class="">Take the case of V1.</div><div class="">The best configuration/setup would be to have all the 6 bricks of V1 on 6 different nodes.</div><div class="">However, in your case you have added 3 new nodes. So, at least we should have 2 bricks on 3 different newly added nodes.</div><div class="">This way, in 4+2 EC configuration, even if one node goes down you will have 4 other bricks of that volume and the data on that volume would be accessible.</div><div class="">In current setup if <span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s04-stg</span> goes down, you will loose all the data on V1 and V2 as all the bricks will be down. We want to avoid and correct it.</div><div class=""><br class=""></div><div class="">Now, we can have two approach to correct/modify this setup.</div><div class=""><br class=""></div><div class=""><strong class="">Approach 1</strong></div><div class="">We have to remove all the newly added bricks in a set of 6 bricks. This will trigger re- balance and move whole data to other sub volumes.</div><div class="">Repeat the above step and then once all the bricks are removed, add those bricks again in a set of 6 bricks, this time have 2 bricks from each of the 3 newly added Nodes.</div><div class=""><br class=""></div><div class="">While this is a valid and working approach, I personally think that this will take long time and also require lot of movement of data.</div><div class=""><br class=""></div><div class=""><div class=""><strong class="">Approach 2</strong></div><div class=""><br class=""></div><div class="">In this approach we can use the heal process. We have to deal with all the volumes (V1 to V6) one by one. Following are the steps for V1-</div><div class=""><br class=""></div><div class=""><strong class="">Step 1 - </strong></div><div class="">Use replace-brick command to move following bricks on <strong class="">s05-stg</strong> node <strong class="">one by one (heal should be completed after every replace brick command)</strong></div><div class=""><br class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><strong class=""><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick39: s04-stg:/gluster/mnt3/brick to s05-stg/<brick which is free><br class=""></span></strong></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><strong class=""><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick40: s04-stg:/gluster/mnt4/brick <span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">to s05-stg/<other brick which is free></span><br class=""></span></strong></div></div><div class=""><br class=""></div><div class="">Command :</div><div class="">gluster v replace-brick <volname> <strong class=""><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s04-stg:/gluster/mnt3/brick</span></strong> <strong class=""><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s05-stg:/<brick which is free></span></strong> commit force</div><div class="">Try to give names to the bricks so that you can identify which 6 bricks belongs to same ec subvolume</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="">Use replace-brick command to move following bricks on <strong class="">s06-stg</strong> node one by one</div><br class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick41: s04-stg:/gluster/mnt5/brick to <strong class=""><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s06-stg/<brick which is free></span></strong><br class=""></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick42: s04-stg:/gluster/mnt6/brick <span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">to <strong class=""><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s06-stg/<other brick which is free></span></strong></span><br class=""></span></div></div><div class=""><br class=""></div><div class=""><div class=""><br class=""></div><div class=""><strong class="">Step 2</strong> - After, every replace-brick command, you have to wait for heal to be completed.</div><div class="">check <strong class="">"gluster v heal <volname> info "</strong> if it shows any entry you have to wait for it to be completed.</div><div class=""><br class=""></div></div><div class="">After successful step 1 and step 2, setup for sub volume V1 will be fixed. The same steps you have to perform for other volumes. Only thing is that</div><div class="">the nodes would be different on which you have to move the bricks.</div><div class=""><br class=""></div><div class=""><br class=""></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">V1<br class=""></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick37: s04-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick38: s04-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick39: s04-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick40: s04-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick41: s04-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick42: s04-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">V2</span></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick43: s04-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick44: s04-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick45: s04-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick46: s04-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick47: s04-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick48: s04-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">V3</span></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick49: s05-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick50: s05-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick51: s05-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick52: s05-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick53: s05-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick54: s05-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">V4</span></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick55: s05-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick56: s05-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick57: s05-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick58: s05-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick59: s05-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick60: s05-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">V5</span></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick61: s06-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick62: s06-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick63: s06-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick64: s06-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick65: s06-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick66: s06-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> </span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">V6</span></span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick67: s06-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick68: s06-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick69: s06-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick70: s06-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick71: s06-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick72: s06-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> </span></div></div><div class=""><br class=""></div><div class="">Just a note that these steps need movement of data.</div><div class="">Be careful while performing these steps and do one replace brick at a time and only after heal completion go to next.</div><div class="">Let me know if you have any issues.</div><div class=""><br class=""></div><div class="">---</div><div class="">Ashish</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div></div><hr id="m_4766848556606636788zwchr" class=""><div style="font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt" class="" data-mce-style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:mauro.tridici@cmcc.it">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>><br class=""><b class="">Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Thursday, September 27, 2018 4:03:04 PM<br class=""><b class="">Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Dear Ashish,</div><div class=""><br class=""></div><div class="">I hope I don’t disturb you so much, but I would like to ask you if you had some time to dedicate to our problem.</div><div class="">Please, forgive my insistence.</div><div class=""><br class=""></div><div class="">Thank you in advance,</div><div class="">Mauro</div><br class=""><div class=""><blockquote class=""><div class="">Il giorno 26 set 2018, alle ore 19:56, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:mauro.tridici@cmcc.it">mauro.tridici@cmcc.it</a>> ha scritto:</div><br class="m_4766848556606636788Apple-interchange-newline"><div class=""><div style="word-wrap:break-word" class="" data-mce-style="word-wrap: break-word;"><div class="">Hi Ashish,</div><div class=""><br class=""></div><div class="">sure, no problem! We are a little bit worried, but we can wait :-)</div><div class="">Thank you very much for your support and your availability.</div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Mauro</div><div class=""><br class=""></div><br class=""><div class=""><blockquote class=""><div class="">Il giorno 26 set 2018, alle ore 19:33, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>> ha scritto:</div><br class="m_4766848556606636788Apple-interchange-newline"><div class=""><div class=""><div style="font-family:'times new roman','new york',times,serif;font-size:12pt" class="" data-mce-style="font-family: 'times new roman','new york',times,serif; font-size: 12pt;"><div class="">Hi Mauro,<br class=""></div><div class=""><br class=""></div><div class="">Yes, I can provide you step by step procedure to correct it. <br class=""></div><div class="">Is it fine If i provide you the steps tomorrow as it is quite late over here and I don't want to miss anything in hurry?<br class=""></div><div class=""><br class=""></div><div class="">---<br class=""></div><div class="">Ashish<br class=""></div><div class=""><br class=""></div><hr id="m_4766848556606636788zwchr" class=""><div style="font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt" class="" data-mce-style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:mauro.tridici@cmcc.it">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>><br class=""><b class="">Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Wednesday, September 26, 2018 6:54:19 PM<br class=""><b class="">Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Hi Ashish,</div><div class=""><br class=""></div><div class="">in attachment you can find the rebalance log file and the last updated brick log file (the other files in /var/log/glusterfs/bricks directory seem to be too old).</div><div class="">I just stopped the running rebalance (as you can see at the bottom of the rebalance log file).</div><div class="">So, if exists a safe procedure to correct the problem I would like execute it.</div><div class=""><br class=""></div><div class="">I don’t know if I can ask you it, but, if it is possible, could you please describe me step by step the right procedure to remove the newly added bricks without losing the data that have been already rebalanced?</div><div class=""><br class=""></div><div class="">The following outputs show the result of “df -h” command executed on one of the first 3 nodes (s01, s02, s03) already existing and on one of the last 3 nodes (s04, s05, s06) added recently.</div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s06 bricks]# df -h</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">File system Dim. Usati Dispon. Uso% Montato su</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s06-root 100G 2,1G 98G 3% /</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">devtmpfs 32G 0 32G 0% /dev</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 4,0K 32G 1% /dev/shm</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 26M 32G 1% /run</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 0 32G 0% /sys/fs/cgroup</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s06-var 100G 2,0G 99G 2% /var</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s06-gluster 100G 33M 100G 1% /gluster</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/sda1 1014M 152M 863M 15% /boot</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgd-gluster_lvd 9,0T 807G 8,3T 9% /gluster/mnt3</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgg-gluster_lvg 9,0T 807G 8,3T 9% /gluster/mnt6</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgc-gluster_lvc 9,0T 807G 8,3T 9% /gluster/mnt2</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vge-gluster_lve 9,0T 807G 8,3T 9% /gluster/mnt4</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgj-gluster_lvj 9,0T 887G 8,2T 10% /gluster/mnt9</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgb-gluster_lvb 9,0T 807G 8,3T 9% /gluster/mnt1</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgh-gluster_lvh 9,0T 887G 8,2T 10% /gluster/mnt7</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgf-gluster_lvf 9,0T 807G 8,3T 9% /gluster/mnt5</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgi-gluster_lvi 9,0T 887G 8,2T 10% /gluster/mnt8</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgl-gluster_lvl 9,0T 887G 8,2T 10% /gluster/mnt11</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgk-gluster_lvk 9,0T 887G 8,2T 10% /gluster/mnt10</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgm-gluster_lvm 9,0T 887G 8,2T 10% /gluster/mnt12</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 6,3G 0 6,3G 0% /run/user/0</span></div></div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s01 ~]# df -h</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">File system Dim. Usati Dispon. Uso% Montato su</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s01-root 100G 5,3G 95G 6% /</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">devtmpfs 32G 0 32G 0% /dev</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 39M 32G 1% /dev/shm</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 26M 32G 1% /run</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 32G 0 32G 0% /sys/fs/cgroup</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s01-var 100G 11G 90G 11% /var</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/md127 1015M 151M 865M 15% /boot</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/cl_s01-gluster 100G 33M 100G 1% /gluster</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgi-gluster_lvi 9,0T 5,5T 3,6T 61% /gluster/mnt7</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgm-gluster_lvm 9,0T 5,4T 3,6T 61% /gluster/mnt11</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgf-gluster_lvf 9,0T 5,7T 3,4T 63% /gluster/mnt4</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgl-gluster_lvl 9,0T 5,8T 3,3T 64% /gluster/mnt10</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgj-gluster_lvj 9,0T 5,5T 3,6T 61% /gluster/mnt8</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgn-gluster_lvn 9,0T 5,4T 3,6T 61% /gluster/mnt12</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgk-gluster_lvk 9,0T 5,8T 3,3T 64% /gluster/mnt9</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgh-gluster_lvh 9,0T 5,6T 3,5T 63% /gluster/mnt6</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgg-gluster_lvg 9,0T 5,6T 3,5T 63% /gluster/mnt5</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vge-gluster_lve 9,0T 5,7T 3,4T 63% /gluster/mnt3</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgc-gluster_lvc 9,0T 5,6T 3,5T 62% /gluster/mnt1</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">/dev/mapper/gluster_vgd-gluster_lvd 9,0T 5,6T 3,5T 62% /gluster/mnt2</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">tmpfs 6,3G 0 6,3G 0% /run/user/0</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s01-stg:tier2 420T 159T 262T 38% /tier2</span></div></div><div class=""><br class=""></div><div class="">As you can see, used space value of each brick of the last servers is about 800GB.</div><div class=""><br class=""></div><div class="">Thank you,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div class=""><br class=""></div><br class=""><div class=""><br class=""></div><br class=""><div class=""><blockquote class=""><div class="">Il giorno 26 set 2018, alle ore 14:51, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>> ha scritto:</div><br class="m_4766848556606636788Apple-interchange-newline"><div class=""><div class=""><div style="font-family:'times new roman','new york',times,serif;font-size:12pt" class="" data-mce-style="font-family: 'times new roman','new york',times,serif; font-size: 12pt;"><div class="">Hi Mauro,<br class=""></div><div class=""><br class=""></div><div class="">rebalance and brick logs should be the first thing we should go through.<br class=""></div><div class=""><br class=""></div><div class="">There is a procedure to correct the configuration/setup but the situation you are in is difficult to follow that procedure.<br class=""></div><div class="">You should have added the bricks hosted on <span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s04-stg</span>, <span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s05-stg and <span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s06-stg the same way you had the previous configuration.</span></span><br class=""></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">That means 2 bricks on each node for one subvolume.<br class=""></span></span></div><div class="">The procedure will require a lot of replace bricks which will again need healing and all. In addition to that we have to wait for re-balance to complete.<br class=""></div><div class=""><br class=""></div><div class="">I would suggest that if whole data has not been rebalanced and if you can stop the rebalance and remove these newly added bricks properly then you should remove these newly added bricks.<br class=""></div><div class="">After that, add these bricks so that you have 2 bricks of each volume on 3 newly added nodes.<br class=""></div><div class=""><br class=""></div><div class="">Yes, it is like undoing whole effort but it is better to do it now then facing issues in future when it will be almost impossible to correct these things if you have lots of data.</div><div class=""><br class=""></div><div class="">---<br class=""></div><div class="">Ashish<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><hr id="m_4766848556606636788zwchr" class=""><div style="font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt" class="" data-mce-style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:mauro.tridici@cmcc.it">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>><br class=""><b class="">Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Wednesday, September 26, 2018 5:55:02 PM<br class=""><b class="">Subject: </b>Re: [Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Dear Ashish,</div><div class=""><br class=""></div><div class="">thank you for you answer.</div><div class="">I could provide you the entire log file related to glusterd, glusterfsd and rebalance.</div><div class="">Please, could you indicate which one you need first?</div><div class=""><br class=""></div><div class="">Yes, we added the last 36 bricks after creating vol. Is there a procedure to correct this error? Is it still possible to do it?</div><div class=""><br class=""></div><div class="">Many thanks,</div><div class="">Mauro</div><br class=""><div class=""><blockquote class=""><div class="">Il giorno 26 set 2018, alle ore 14:13, Ashish Pandey <<a href="mailto:aspandey@redhat.com" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:aspandey@redhat.com">aspandey@redhat.com</a>> ha scritto:</div><br class="m_4766848556606636788Apple-interchange-newline"><div class=""><div class=""><div style="font-family:'times new roman','new york',times,serif;font-size:12pt" class="" data-mce-style="font-family: 'times new roman','new york',times,serif; font-size: 12pt;"><div class=""><br class=""></div><div class="">I think we don't have enough logs to debug this so I would suggest you to provide more logs/info.<br class=""></div><div class="">I have also observed that the configuration and setup of your volume is not very efficient.<br class=""></div><div class=""><br class=""></div><div class="">For example: <br class=""></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick37: s04-stg:/gluster/mnt1/brick</span><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick38: s04-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick39: s04-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick40: s04-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick41: s04-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick42: s04-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick43: s04-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick44: s04-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick45: s04-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick46: s04-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick47: s04-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:#b8b43b;background-color:#000000" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Brick48: s04-stg:/gluster/mnt12/brick</span></div></div><div class=""><br class=""></div><div class="">These 12 bricks are on same node and the sub volume made up of these bricks will be of same subvolume, which is not good. Same is true for the bricks hosted on <span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s05-stg and <span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">s06-stg</span></span><br class=""></div><div class="">I think you have added these bricks after creating vol. The probability of disruption in connection of these bricks will be higher in this case.<br class=""></div><div class=""><br class=""></div><div class="">---<br class=""></div><div class="">Ashish<br class=""></div><div class=""><br class=""></div><hr id="m_4766848556606636788zwchr" class=""><div style="font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt" class="" data-mce-style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:mauro.tridici@cmcc.it">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank" rel="noreferrer" class="" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Wednesday, September 26, 2018 3:38:35 PM<br class=""><b class="">Subject: </b>[Gluster-users] Rebalance failed on Distributed Disperse volume based on 3.12.14 version<br class=""><div class=""><br class=""></div>Dear All, Dear Nithya,<div class=""><br class=""></div><div class="">after upgrading from 3.10.5 version to 3.12.14, I tried to start a rebalance process to distribute data across the bricks, but something goes wrong.</div><div class="">Rebalance failed on different nodes and the time value needed to complete the procedure seems to be very high.</div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s01 ~]# gluster volume rebalance tier2 status</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> localhost 19 161.6GB 537 2 2 in progress 0:32:23</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> s02-stg 25 212.7GB 526 5 2 in progress 0:32:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> s03-stg 4 69.1GB 511 0 0 in progress 0:32:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> s04-stg 4 484Bytes 12283 0 3 in progress 0:32:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> s05-stg 23 484Bytes 11049 0 10 in progress 0:32:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> s06-stg 3 1.2GB 8032 11 3 failed 0:17:57</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">Estimated time left for rebalance to complete : 3601:05:41</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">volume rebalance: tier2: success</span></div></div><div class=""><br class=""></div><div class="">When rebalance processes fail, I can see the following kind of errors in /var/log/glusterfs/tier2-rebalance.log</div><div class=""><br class=""></div><div class="">Error type 1)</div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 08:50:19.872575] W [MSGID: 122053] [ec-common.c:269:ec_check_status] 0-tier2-disperse-10: Operation failed on 2 of 6 subvolumes.(up=111111, mask=100111, remaining=</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">000000, good=100111, bad=011000)</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 08:50:19.901792] W [MSGID: 122053] [ec-common.c:269:ec_check_status] 0-tier2-disperse-11: Operation failed on 1 of 6 subvolumes.(up=111111, mask=111101, remaining=</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">000000, good=111101, bad=000010)</span></div></div><div class=""><br class=""></div><div class="">Error type 2)</div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 08:53:31.566836] W [socket.c:600:__socket_rwv] 0-tier2-client-53: readv on <a href="http://192.168.0.55:49153/" target="_blank" rel="noreferrer" class="" data-mce-href="http://192.168.0.55:49153/">192.168.0.55:49153</a> failed (Connection reset by peer)</span></div></div><div class=""><br class=""></div><div class="">Error type 3)</div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 08:57:37.852590] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (10)</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 08:57:39.282306] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (10)</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 09:02:04.928408] W [MSGID: 109023] [dht-rebalance.c:1013:__dht_check_free_space] 0-tier2-dht: data movement of file {blocks:0 name:(/OPA/archive/historical/dts/MRE</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">A/Observations/Observations/MREA14/Cs-1/CMCC/raw/CS013.ext)} would result in dst node (tier2-disperse-5:2440190848) having lower disk space than the source node (tier2-dispers</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">e-11:71373083776).Skipping file.</span></div></div><div class=""><br class=""></div><div class="">Error type 4)</div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] 0-tier2-client-7: socket disconnected</span></div></div><div class=""><br class=""></div><div class="">Error type 5)</div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[2018-09-26 09:07:42.333720] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f0417e0ee25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">90086004b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55900860032b] ) 0-: received signum (15), shutting down</span></div></div><div class=""><br class=""></div><div class="">Error type 6)</div><div class=""><br class=""></div><div class="">[2018-09-25 08:09:18.340658] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-4: server <a href="http://192.168.0.52:49153/" target="_blank" rel="noreferrer" class="" data-mce-href="http://192.168.0.52:49153/">192.168.0.52:49153</a> has not responded in the last 42 seconds, disconnecting.</div><div class=""><br class=""></div><div class="">It seems that there are some network or timeout problems, but the network usage/traffic values are not so high.</div><div class="">Do you think that, in my volume configuration, I have to modify some volume options related to thread and/or network parameters?</div><div class="">Could you, please, help me to understand the cause of the problems above?</div><div class=""><br class=""></div><div class="">You can find below our volume info:</div><div class="">(volume is implemented on 6 servers; each server configuration: 2 cpu 10-cores, 64GB RAM, 1 SSD dedicated to the OS, 12 x 10TB HD)</div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;">[root@s04 ~]# gluster vol info</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0);min-height:18px" class="" data-mce-style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: #b8b43b; background-color: #000000; min-height: 18px;"><span style="font-variant-ligatures:no-common-ligatures" class="" data-mce-style="font-variant-ligatures: no-common-ligatures;"> </span><br class="m_4766848556606636788webkit-block-placeholder"></div><div style="margin: 0px; line-height: nor</div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></div></blockquote></div></div></div><br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="" target="_blank" data-mce-href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br class=""><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" class="">https://lists.gluster.org/mailman/listinfo/gluster-users</a></div><div class=""><br class=""></div></div></div></div></blockquote></div></div></div></div></div></blockquote></div><br class=""></body></html>