<div><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Sun, 5 Aug 2018 at 13:29, Yuhao Zhang <<a href="mailto:zzyzxd@gmail.com">zzyzxd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">Sorry, what I meant was, if I start the transfer now and get glusterd into zombie status, </div></blockquote><div dir="auto"><br></div><div dir="auto">glusterd or glusterfsd?</div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">it's unlikely that I can fully recover the server without a reboot.</div><div style="word-wrap:break-word;line-break:after-white-space"><br><div><br><blockquote type="cite"><div>On Aug 5, 2018, at 02:55, Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:</div><br class="m_-5646314163240277706Apple-interchange-newline"><div><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Aug 5, 2018 at 1:22 PM, Yuhao Zhang <span dir="ltr"><<a href="mailto:zzyzxd@gmail.com" target="_blank">zzyzxd@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">This is a semi-production server and I can't bring it down right now. Will try to get the monitoring output when I get a chance. </div></blockquote><div><br></div><div>Collecting top output doesn't require to bring down servers.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space"><div><br></div><div>As I recall, the high CPU processes are brick daemons (glusterfsd) and htop showed they were in status D. However, I saw zero zpool IO as clients were all hanging.<div><div class="m_-5646314163240277706h5"><br><div><br><blockquote type="cite"><div>On Aug 5, 2018, at 02:38, Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:</div><br class="m_-5646314163240277706m_-5304337048872840202Apple-interchange-newline"><div><br class="m_-5646314163240277706m_-5304337048872840202Apple-interchange-newline"><br style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div class="gmail_quote" style="font-family:Helvetica;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">On Sun, Aug 5, 2018 at 12:44 PM, Yuhao Zhang<span class="m_-5646314163240277706m_-5304337048872840202Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:zzyzxd@gmail.com" target="_blank">zzyzxd@gmail.com</a>></span><span class="m_-5646314163240277706m_-5304337048872840202Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Hi,<br><br>I am running into a situation that heavy write causes Gluster server went into zombie with many high CPU processes and all clients hangs, it is almost 100% reproducible on my machine. Hope someone can help.<br></blockquote><div><br></div><div>Can you give us the output of monitioring these processes with High cpu usage captured in the duration when your tests are running?<br></div><div><br></div><div><ul><li><span style="font-family:terminal,monaco,monospace">MON_INTERVAL=10 # can be increased for very long runs</span></li><li><span style="font-family:terminal,monaco,monospace">top -bd $MON_INTERVAL > /tmp/top_proc.${HOSTNAME}.txt # CPU utilization by process</span></li><li><span style="font-family:terminal,monaco,monospace">top -bHd $MON_INTERVAL > /tmp/top_thr.${HOSTNAME}.txt # CPU utilization by thread</span></li></ul><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><br>I started to observe this issue when running rsync to copy files from another server and I thought it might be because Gluster doesn't like rsync's delta transfer with a lot of small writes. However, I was able to reproduce this with "rsync --whole-file --inplace", or even with cp or scp. It usually appears after starting the transfer for a few hours, but sometimes can happen within several minutes.<br><br>Since this is a single node Gluster distributed volume, I tried to transfer files directly onto the server bypassing Gluster clients, but it still caused the same issue.<br><br>It is running on top of a ZFS RAIDZ2 dataset. Options are attached. Also, I attached the statedump generated when my clients hung, and volume options.<br><br>- Ubuntu 16.04 x86_64 / 4.4.0-116-generic<br>- GlusterFS 3.12.8<br><br>Thank you,<br>Yuhao<br><br><br>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div></blockquote></div><br></div></div></div></div></blockquote></div><br></div></div>
</div></blockquote></div><br></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">--Atin</div>