From spisla80 at gmail.com Fri Mar 1 07:48:42 2019 From: spisla80 at gmail.com (David Spisla) Date: Fri, 1 Mar 2019 08:48:42 +0100 Subject: [Gluster-devel] Bitrot: Time of signing depending on the file size??? Message-ID: Hello folks, I did some observations concerning the bitrot daemon. It seems to be that the bitrot signer is signing files depending on file size. I copied files with different sizes into a volume and I was wonderung because the files get their signature not the same time (I keep the expiry time default with 120). Here are some examples: 300 KB file ~2-3 m 70 MB file ~ 40 m 115 MB file ~ 1 Sh 800 MB file ~ 4,5 h What is the expected behaviour here? Why does it take so long to sign a 800MB file? What about 500GB or 1TB? Is there a way to speed up the sign process? My ambition is to understand this observation Regards David Spisla -------------- next part -------------- An HTML attachment was scrubbed... URL: From spisla80 at gmail.com Fri Mar 1 12:19:29 2019 From: spisla80 at gmail.com (David Spisla) Date: Fri, 1 Mar 2019 13:19:29 +0100 Subject: [Gluster-devel] [Gluster-users] Bitrot: Time of signing depending on the file size??? In-Reply-To: References: Message-ID: Hello Amudhan, What does exactly mean "it takes < 250KB/s"? I figured out this discussion between you and Kotresh: https://lists.gluster.org/pipermail/gluster-users/2016-September/028354.html Kotresh mentioned there that the problem is because for some files fd process are still up in the brick process list. Bitrot signer can only sign a file if the fd is closed. And according to my observations it seems to be that as bigger a file is as longer the fd is still up. I could verify this with a 500MiB file and some smaller files. After a specific time only the fd for the 500MiB was up and the file still had no signature, for the smaller files there were no fds and they already had a signature. Does anybody know what is the reason for this? For me it looks loke a bug. Regards David Am Fr., 1. M?rz 2019 um 08:58 Uhr schrieb Amudhan P : > Hi David, > > I have also tested the bitrot signature process by default it takes < 250 > KB/s. > > regards > Amudhan P > > > On Fri, Mar 1, 2019 at 1:19 PM David Spisla wrote: > >> Hello folks, >> >> I did some observations concerning the bitrot daemon. It seems to be that >> the bitrot signer is signing files depending on file size. I copied files >> with different sizes into a volume and I was wonderung because the files >> get their signature not the same time (I keep the expiry time default with >> 120). Here are some examples: >> >> 300 KB file ~2-3 m >> 70 MB file ~ 40 m >> 115 MB file ~ 1 Sh >> 800 MB file ~ 4,5 h >> >> What is the expected behaviour here? >> Why does it take so long to sign a 800MB file? >> What about 500GB or 1TB? >> Is there a way to speed up the sign process? >> >> My ambition is to understand this observation >> >> Regards >> David Spisla >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spisla80 at gmail.com Fri Mar 1 14:00:33 2019 From: spisla80 at gmail.com (David Spisla) Date: Fri, 1 Mar 2019 15:00:33 +0100 Subject: [Gluster-devel] [Gluster-users] Bitrot: Time of signing depending on the file size??? In-Reply-To: References: Message-ID: Hello Amudhan, Am Fr., 1. M?rz 2019 um 14:56 Uhr schrieb Amudhan P : > Hi David, > > Once after file write completes (fd closed) bitrot process will wait for > 120 seconds and if there no fd is opened for the file it will trigger the > signer process. > Yes, I already know this. But there is still the problem that a fd will no closed after open() as bigger s file is. See link to discussion. > > considering the signer, process start and end time file read speed was < > 250KB/s. > How can I measure this? > > To increase bitrot signer read speed need to modify the bitrot source file. > How can I do that? Regards David > > regards > Amudhan > > On Fri, Mar 1, 2019 at 5:49 PM David Spisla wrote: > >> Hello Amudhan, >> >> What does exactly mean "it takes < 250KB/s"? >> I figured out this discussion between you and Kotresh: >> https://lists.gluster.org/pipermail/gluster-users/2016-September/028354.html >> Kotresh mentioned there that the problem is because for some files fd >> process are still up in the brick process list. Bitrot signer can only sign >> a file if the fd is closed. And according to my observations it seems to be >> that as bigger a file is as longer the fd is still up. I could verify this >> with a 500MiB file and some smaller files. After a specific time only the >> fd for the 500MiB was up and the file still had no signature, for the >> smaller files there were no fds and they already had a signature. >> >> Does anybody know what is the reason for this? For me it looks loke a >> bug. >> >> Regards >> David >> >> Am Fr., 1. M?rz 2019 um 08:58 Uhr schrieb Amudhan P > >: >> >>> Hi David, >>> >>> I have also tested the bitrot signature process by default it takes < >>> 250 KB/s. >>> >>> regards >>> Amudhan P >>> >>> >>> On Fri, Mar 1, 2019 at 1:19 PM David Spisla wrote: >>> >>>> Hello folks, >>>> >>>> I did some observations concerning the bitrot daemon. It seems to be >>>> that the bitrot signer is signing files depending on file size. I copied >>>> files with different sizes into a volume and I was wonderung because the >>>> files get their signature not the same time (I keep the expiry time default >>>> with 120). Here are some examples: >>>> >>>> 300 KB file ~2-3 m >>>> 70 MB file ~ 40 m >>>> 115 MB file ~ 1 Sh >>>> 800 MB file ~ 4,5 h >>>> >>>> What is the expected behaviour here? >>>> Why does it take so long to sign a 800MB file? >>>> What about 500GB or 1TB? >>>> Is there a way to speed up the sign process? >>>> >>>> My ambition is to understand this observation >>>> >>>> Regards >>>> David Spisla >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From khiremat at redhat.com Fri Mar 1 17:29:01 2019 From: khiremat at redhat.com (Kotresh Hiremath Ravishankar) Date: Fri, 1 Mar 2019 22:59:01 +0530 Subject: [Gluster-devel] Bitrot: Time of signing depending on the file size??? In-Reply-To: References: Message-ID: Interesting observation! But as discussed in the thread bitrot signing processes depends 2 min timeout (by default) after last fd closes. It doesn't have any co-relation with the size of the file. Did you happen to verify that the fd was still open for large files for some reason? On Fri, Mar 1, 2019 at 1:19 PM David Spisla wrote: > Hello folks, > > I did some observations concerning the bitrot daemon. It seems to be that > the bitrot signer is signing files depending on file size. I copied files > with different sizes into a volume and I was wonderung because the files > get their signature not the same time (I keep the expiry time default with > 120). Here are some examples: > > 300 KB file ~2-3 m > 70 MB file ~ 40 m > 115 MB file ~ 1 Sh > 800 MB file ~ 4,5 h > > What is the expected behaviour here? > Why does it take so long to sign a 800MB file? > What about 500GB or 1TB? > Is there a way to speed up the sign process? > > My ambition is to understand this observation > > Regards > David Spisla > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel -- Thanks and Regards, Kotresh H R -------------- next part -------------- An HTML attachment was scrubbed... URL: From jenkins at build.gluster.org Mon Mar 4 01:45:02 2019 From: jenkins at build.gluster.org (jenkins at build.gluster.org) Date: Mon, 4 Mar 2019 01:45:02 +0000 (UTC) Subject: [Gluster-devel] Weekly Untriaged Bugs Message-ID: <1266425698.31.1551663902784.JavaMail.jenkins@jenkins-el7.rht.gluster.org> [...truncated 6 lines...] https://bugzilla.redhat.com/1683574 / build: gluster-server package currently requires the older userspace-rcu against expectation https://bugzilla.redhat.com/1679904 / core: client log flooding with intentional socket shutdown message when a brick is down https://bugzilla.redhat.com/1677555 / core: Glusterfs brick is crashed due to segfault caused by broken gfid symlink https://bugzilla.redhat.com/1674412 / core: listing a file while writing to it causes deadlock https://bugzilla.redhat.com/1683815 / core: Memory leak when peer detach fails https://bugzilla.redhat.com/1679744 / core: Minio gateway nas does not work with 2 + 1 dispersed volumes https://bugzilla.redhat.com/1673058 / core: Network throughput usage increased x5 https://bugzilla.redhat.com/1678640 / core: Running 'control-cpu-load.sh' prevents CTDB starting https://bugzilla.redhat.com/1684569 / core: Upgrade from 4.1 and 5 is broken https://bugzilla.redhat.com/1676429 / distribute: distribute: Perf regression in mkdir path https://bugzilla.redhat.com/1672656 / eventsapi: glustereventsd: crash, ABRT report for package glusterfs has reached 100 occurrences https://bugzilla.redhat.com/1672258 / fuse: fuse takes memory and doesn't free https://bugzilla.redhat.com/1679892 / glusterd: assertion failure log in glusterd.log file when a volume start is triggered https://bugzilla.redhat.com/1683526 / glusterd: rebalance start command doesn't throw up error message if the command fails https://bugzilla.redhat.com/1679169 / md-cache: Integer Overflow possible in md-cache.c due to data type inconsistency https://bugzilla.redhat.com/1679170 / md-cache: Integer Overflow possible in md-cache.c due to data type inconsistency https://bugzilla.redhat.com/1677557 / nfs: gNFS crashed when processing "gluster v profile [vol] info nfs" https://bugzilla.redhat.com/1677804 / posix-acl: POSIX ACLs are absent on FUSE-mounted volume using tmpfs bricks (posix-acl-autoload usually returns -1) https://bugzilla.redhat.com/1678378 / project-infrastructure: Add a nightly build verification job in Jenkins for release-6 https://bugzilla.redhat.com/1682925 / replicate: Gluster volumes never heal during oVirt 4.2->4.3 upgrade https://bugzilla.redhat.com/1683317 / tests: ./tests/bugs/glusterfs/bug-866459.t failing on s390x https://bugzilla.redhat.com/1676356 / write-behind: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' [...truncated 2 lines...] -------------- next part -------------- A non-text attachment was scrubbed... Name: build.log Type: application/octet-stream Size: 2794 bytes Desc: not available URL: From ykaul at redhat.com Mon Mar 4 07:50:25 2019 From: ykaul at redhat.com (Yaniv Kaul) Date: Mon, 4 Mar 2019 09:50:25 +0200 Subject: [Gluster-devel] What happened to 'gluster-ansible' RPM? Message-ID: I've used[1] it to deploy Gluster (on CentOS 7) and now it seems to be missing. I'm not seeing this meta package @ [2] - am I supposed to install the specific packages? TIA, Y. [1] https://github.com/mykaul/vg/blob/d36ad9948a1be49be5b7f7d95a2007aa8c540d95/ansible/machine_config.yml#L75 [2] https://copr.fedorainfracloud.org/coprs/sac/gluster-ansible/packages/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From surs at redhat.com Mon Mar 4 08:19:45 2019 From: surs at redhat.com (Sachidananda URS) Date: Mon, 4 Mar 2019 13:49:45 +0530 Subject: [Gluster-devel] What happened to 'gluster-ansible' RPM? In-Reply-To: References: Message-ID: Hi Yaniv, On Mon, Mar 4, 2019 at 1:21 PM Yaniv Kaul wrote: > I've used[1] it to deploy Gluster (on CentOS 7) and now it seems to be > missing. > I'm not seeing this meta package @ [2] - am I supposed to install the > specific packages? > > We still have the meta-package. It is called gluster-ansible-roles[1][2], I'm sorry about this renaming. The reason behind this change is the discrepancy I had created in upstream and downstream naming. The downstream package was called gluster-ansible-roles (we followed the same convention as ovirt roles). But by then I had done a few builds in upstream. So, recently we synced the naming. Future Fedora packages will follow this naming convention. -sac [1] https://copr.fedorainfracloud.org/coprs/sac/gluster-ansible/build/863675/ [2] https://copr.fedorainfracloud.org/coprs/sac/gluster-ansible/builds/ > TIA, > Y. > [1] > https://github.com/mykaul/vg/blob/d36ad9948a1be49be5b7f7d95a2007aa8c540d95/ansible/machine_config.yml#L75 > [2] https://copr.fedorainfracloud.org/coprs/sac/gluster-ansible/packages/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From spisla80 at gmail.com Mon Mar 4 09:43:43 2019 From: spisla80 at gmail.com (David Spisla) Date: Mon, 4 Mar 2019 10:43:43 +0100 Subject: [Gluster-devel] Bitrot: Time of signing depending on the file size??? In-Reply-To: References: Message-ID: Hello Kotresh, Yes, the fd was still open for larger files. I could verify this with a 500MiB file and some smaller files. After a specific time only the fd for the 500MiB was up and the file still had no signature, for the smaller files there were no fds and they already had a signature. I don't know the reason for this. Maybe the client still keep th fd open? I opened a bug for this: https://bugzilla.redhat.com/show_bug.cgi?id=1685023 Regards David Am Fr., 1. M?rz 2019 um 18:29 Uhr schrieb Kotresh Hiremath Ravishankar < khiremat at redhat.com>: > Interesting observation! But as discussed in the thread bitrot signing > processes depends 2 min timeout (by default) after last fd closes. It > doesn't have any co-relation with the size of the file. > Did you happen to verify that the fd was still open for large files for > some reason? > > > > On Fri, Mar 1, 2019 at 1:19 PM David Spisla wrote: > >> Hello folks, >> >> I did some observations concerning the bitrot daemon. It seems to be that >> the bitrot signer is signing files depending on file size. I copied files >> with different sizes into a volume and I was wonderung because the files >> get their signature not the same time (I keep the expiry time default with >> 120). Here are some examples: >> >> 300 KB file ~2-3 m >> 70 MB file ~ 40 m >> 115 MB file ~ 1 Sh >> 800 MB file ~ 4,5 h >> >> What is the expected behaviour here? >> Why does it take so long to sign a 800MB file? >> What about 500GB or 1TB? >> Is there a way to speed up the sign process? >> >> My ambition is to understand this observation >> >> Regards >> David Spisla >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-devel > > > > -- > Thanks and Regards, > Kotresh H R > -------------- next part -------------- An HTML attachment was scrubbed... URL: From atumball at redhat.com Mon Mar 4 15:01:38 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Mon, 4 Mar 2019 20:31:38 +0530 Subject: [Gluster-devel] GlusterFS - 6.0RC - Test days (27th, 28th Feb) In-Reply-To: References: Message-ID: Thanks to those who participated. Update at present: We found 3 blocker bugs in upgrade scenarios, and hence have marked release as pending upon them. We will keep these lists updated about progress. -Amar On Mon, Feb 25, 2019 at 11:41 PM Amar Tumballi Suryanarayan < atumball at redhat.com> wrote: > Hi all, > > We are calling out our users, and developers to contribute in validating > ?glusterfs-6.0rc? build in their usecase. Specially for the cases of > upgrade, stability, and performance. > > Some of the key highlights of the release are listed in release-notes > draft > . > Please note that there are some of the features which are being dropped out > of this release, and hence making sure your setup is not going to have an > issue is critical. Also the default lru-limit option in fuse mount for > Inodes should help to control the memory usage of client processes. All the > good reason to give it a shot in your test setup. > > If you are developer using gfapi interface to integrate with other > projects, you also have some signature changes, so please make sure your > project would work with latest release. Or even if you are using a project > which depends on gfapi, report the error with new RPMs (if any). We will > help fix it. > > As part of test days, we want to focus on testing the latest upcoming > release i.e. GlusterFS-6, and one or the other gluster volunteers would be > there on #gluster channel on freenode to assist the people. Some of the key > things we are looking as bug reports are: > > - > > See if upgrade from your current version to 6.0rc is smooth, and works > as documented. > - Report bugs in process, or in documentation if you find mismatch. > - > > Functionality is all as expected for your usecase. > - No issues with actual application you would run on production etc. > - > > Performance has not degraded in your usecase. > - While we have added some performance options to the code, not all of > them are turned on, as they have to be done based on usecases. > - Make sure the default setup is at least same as your current > version > - Try out few options mentioned in release notes (especially, > --auto-invalidation=no) and see if it helps performance. > - > > While doing all the above, check below: > - see if the log files are making sense, and not flooding with some > ?for developer only? type of messages. > - get ?profile info? output from old and now, and see if there is > anything which is out of normal expectation. Check with us on the numbers. > - get a ?statedump? when there are some issues. Try to make sense > of it, and raise a bug if you don?t understand it completely. > > > Process > expected on test days. > > - > > We have a tracker bug > [0] > - We will attach all the ?blocker? bugs to this bug. > - > > Use this link to report bugs, so that we have more metadata around > given bugzilla. > - Click Here > > [1] > - > > The test cases which are to be tested are listed here in this sheet > [2], > please add, update, and keep it up-to-date to reduce duplicate efforts. > > Lets together make this release a success. > > Also check if we covered some of the open issues from Weekly untriaged > bugs > > [3] > > For details on build and RPMs check this email > > [4] > > Finally, the dates :-) > > - Wednesday - Feb 27th, and > - Thursday - Feb 28th > > Note that our goal is to identify as many issues as possible in upgrade > and stability scenarios, and if any blockers are found, want to make sure > we release with the fix for same. So each of you, Gluster users, feel > comfortable to upgrade to 6.0 version. > > Regards, > Gluster Ants. > > -- > Amar Tumballi (amarts) > -- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: From amukherj at redhat.com Mon Mar 4 15:08:04 2019 From: amukherj at redhat.com (Atin Mukherjee) Date: Mon, 4 Mar 2019 20:38:04 +0530 Subject: [Gluster-devel] [Gluster-Maintainers] GlusterFS - 6.0RC - Test days (27th, 28th Feb) In-Reply-To: References: Message-ID: On Mon, 4 Mar 2019 at 20:33, Amar Tumballi Suryanarayan wrote: > Thanks to those who participated. > > Update at present: > > We found 3 blocker bugs in upgrade scenarios, and hence have marked release > as pending upon them. We will keep these lists updated about progress. I?d like to clarify that upgrade testing is blocked. So just fixing these test blocker(s) isn?t enough to call release-6 green. We need to continue and finish the rest of the upgrade tests once the respective bugs are fixed. > > -Amar > > On Mon, Feb 25, 2019 at 11:41 PM Amar Tumballi Suryanarayan < > atumball at redhat.com> wrote: > > > Hi all, > > > > We are calling out our users, and developers to contribute in validating > > ?glusterfs-6.0rc? build in their usecase. Specially for the cases of > > upgrade, stability, and performance. > > > > Some of the key highlights of the release are listed in release-notes > > draft > > < > https://github.com/gluster/glusterfs/blob/release-6/doc/release-notes/6.0.md > >. > > Please note that there are some of the features which are being dropped > out > > of this release, and hence making sure your setup is not going to have an > > issue is critical. Also the default lru-limit option in fuse mount for > > Inodes should help to control the memory usage of client processes. All > the > > good reason to give it a shot in your test setup. > > > > If you are developer using gfapi interface to integrate with other > > projects, you also have some signature changes, so please make sure your > > project would work with latest release. Or even if you are using a > project > > which depends on gfapi, report the error with new RPMs (if any). We will > > help fix it. > > > > As part of test days, we want to focus on testing the latest upcoming > > release i.e. GlusterFS-6, and one or the other gluster volunteers would > be > > there on #gluster channel on freenode to assist the people. Some of the > key > > things we are looking as bug reports are: > > > > - > > > > See if upgrade from your current version to 6.0rc is smooth, and works > > as documented. > > - Report bugs in process, or in documentation if you find mismatch. > > - > > > > Functionality is all as expected for your usecase. > > - No issues with actual application you would run on production etc. > > - > > > > Performance has not degraded in your usecase. > > - While we have added some performance options to the code, not all of > > them are turned on, as they have to be done based on usecases. > > - Make sure the default setup is at least same as your current > > version > > - Try out few options mentioned in release notes (especially, > > --auto-invalidation=no) and see if it helps performance. > > - > > > > While doing all the above, check below: > > - see if the log files are making sense, and not flooding with some > > ?for developer only? type of messages. > > - get ?profile info? output from old and now, and see if there is > > anything which is out of normal expectation. Check with us on the > numbers. > > - get a ?statedump? when there are some issues. Try to make sense > > of it, and raise a bug if you don?t understand it completely. > > > > > > < > https://hackmd.io/YB60uRCMQRC90xhNt4r6gA?both#Process-expected-on-test-days > >Process > > expected on test days. > > > > - > > > > We have a tracker bug > > [0] > > - We will attach all the ?blocker? bugs to this bug. > > - > > > > Use this link to report bugs, so that we have more metadata around > > given bugzilla. > > - Click Here > > < > https://bugzilla.redhat.com/enter_bug.cgi?blocked=1672818&bug_severity=high&component=core&priority=high&product=GlusterFS&status_whiteboard=gluster-test-day&version=6 > > > > [1] > > - > > > > The test cases which are to be tested are listed here in this sheet > > < > https://docs.google.com/spreadsheets/d/1AS-tDiJmAr9skK535MbLJGe_RfqDQ3j1abX1wtjwpL4/edit?usp=sharing > >[2], > > please add, update, and keep it up-to-date to reduce duplicate efforts -- - Atin (atinm) -------------- next part -------------- An HTML attachment was scrubbed... URL: From srangana at redhat.com Mon Mar 4 17:33:49 2019 From: srangana at redhat.com (Shyam Ranganathan) Date: Mon, 4 Mar 2019 12:33:49 -0500 Subject: [Gluster-devel] [Gluster-Maintainers] GlusterFS - 6.0RC - Test days (27th, 28th Feb) In-Reply-To: References: Message-ID: <5f27393d-cefb-aa47-12f8-1597677ffb50@redhat.com> On 3/4/19 10:08 AM, Atin Mukherjee wrote: > > > On Mon, 4 Mar 2019 at 20:33, Amar Tumballi Suryanarayan > > wrote: > > Thanks to those who participated. > > Update at present: > > We found 3 blocker bugs in upgrade scenarios, and hence have marked > release > as pending upon them. We will keep these lists updated about progress. > > > I?d like to clarify that upgrade testing is blocked. So just fixing > these test blocker(s) isn?t enough to call release-6 green. We need to > continue and finish the rest of the upgrade tests once the respective > bugs are fixed. Based on fixes expected by tomorrow for the upgrade fixes, we will build an RC1 candidate on Wednesday (6-Mar) (tagging early Wed. Eastern TZ). This RC can be used for further testing. > > > > -Amar > > On Mon, Feb 25, 2019 at 11:41 PM Amar Tumballi Suryanarayan < > atumball at redhat.com > wrote: > > > Hi all, > > > > We are calling out our users, and developers to contribute in > validating > > ?glusterfs-6.0rc? build in their usecase. Specially for the cases of > > upgrade, stability, and performance. > > > > Some of the key highlights of the release are listed in release-notes > > draft > > > . > > Please note that there are some of the features which are being > dropped out > > of this release, and hence making sure your setup is not going to > have an > > issue is critical. Also the default lru-limit option in fuse mount for > > Inodes should help to control the memory usage of client > processes. All the > > good reason to give it a shot in your test setup. > > > > If you are developer using gfapi interface to integrate with other > > projects, you also have some signature changes, so please make > sure your > > project would work with latest release. Or even if you are using a > project > > which depends on gfapi, report the error with new RPMs (if any). > We will > > help fix it. > > > > As part of test days, we want to focus on testing the latest upcoming > > release i.e. GlusterFS-6, and one or the other gluster volunteers > would be > > there on #gluster channel on freenode to assist the people. Some > of the key > > things we are looking as bug reports are: > > > >? ? - > > > >? ? See if upgrade from your current version to 6.0rc is smooth, > and works > >? ? as documented. > >? ? - Report bugs in process, or in documentation if you find mismatch. > >? ? - > > > >? ? Functionality is all as expected for your usecase. > >? ? - No issues with actual application you would run on production > etc. > >? ? - > > > >? ? Performance has not degraded in your usecase. > >? ? - While we have added some performance options to the code, not > all of > >? ? ? ?them are turned on, as they have to be done based on usecases. > >? ? ? ?- Make sure the default setup is at least same as your current > >? ? ? ?version > >? ? ? ?- Try out few options mentioned in release notes (especially, > >? ? ? ?--auto-invalidation=no) and see if it helps performance. > >? ? - > > > >? ? While doing all the above, check below: > >? ? - see if the log files are making sense, and not flooding with some > >? ? ? ??for developer only? type of messages. > >? ? ? ?- get ?profile info? output from old and now, and see if > there is > >? ? ? ?anything which is out of normal expectation. Check with us > on the numbers. > >? ? ? ?- get a ?statedump? when there are some issues. Try to make > sense > >? ? ? ?of it, and raise a bug if you don?t understand it completely. > > > > > > > Process > > expected on test days. > > > >? ? - > > > >? ? We have a tracker bug > >? ? [0] > >? ? - We will attach all the ?blocker? bugs to this bug. > >? ? - > > > >? ? Use this link to report bugs, so that we have more metadata around > >? ? given bugzilla. > >? ? - Click Here > >? ? ? > ? > >? ? ? ?[1] > >? ? - > > > >? ? The test cases which are to be tested are listed here in this sheet > >? ? > [2], > >? ? please add, update, and keep it up-to-date to reduce duplicate > efforts > > -- > - Atin (atinm) > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > From rgowdapp at redhat.com Mon Mar 4 17:45:59 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Mon, 4 Mar 2019 23:15:59 +0530 Subject: [Gluster-devel] [Gluster-users] "rpc_clnt_ping_timer_expired" errors In-Reply-To: References: <96B07283-D8AB-4F06-909D-E00424625528@cmcc.it> <42758A0E-8BE9-497D-BDE3-55D7DC633BC7@cmcc.it> <6A8CF4A4-98EA-48C3-A059-D60D1B2721C7@cmcc.it> <2CF49168-9C1B-4931-8C34-8157262A137A@cmcc.it> <7A151CC9-A0AE-4A45-B450-A4063D216D9E@cmcc.it> <32D53ECE-3F49-4415-A6EE-241B351AC2BA@cmcc.it> <4685A75B-5978-4338-9C9F-4A02FB40B9BC@cmcc.it> <4D2E6B04-C2E8-4FD5-B72D-E7C05931C6F9@cmcc.it> <4E332A56-B318-4BC1-9F44-84AB4392A5DE@cmcc.it> <832FD362-3B14-40D8-8530-604419300476@cmcc.it> <8D926643-1093-48ED-823F-D8F117F702CF@cmcc.it> <9D0BE438-DF11-4D0A-AB85-B44357032F29@cmcc.it> <5F0AC378-8170-4342-8473-9C17159CAC1D@cmcc.it> <7A50E86D-9E27-4EA7-883B-45E9F973991A@cmcc.it> <58B5DB7F-DCB4-4FBF-B1F7-681315B1613A@cmcc.it> <6327B44F-4E7E-46BB-A74C-70F4457DD1EB@cmcc.it> <0167DF4A-8329-4A1A-B439-857DFA6F78BB@cmcc.it> <763F334E-35B8-4729-B8E1-D30866754EEE@cmcc.it> <91DFC9AC-4805-41EB-AC6F-5722E018DD6E@cmcc.it> <8A9752B8-B231-4570-8FF4-8D3D781E7D42@cmcc.it> <47A24804-F975-4EE6-9FA5-67FCDA18D707@cmcc.it> <637FF5D2-D1F4-4686-9D48-646A96F67B96@cmcc.it> <4A87495F-3755-48F7-8507-085869069C64@cmcc.it> <3854BBF6-5B98-4AB3-A67E-E7DE59E69A63@cmcc.it> <313DA021-9173-4899-96B0-831B10B00B61@cmcc.it> <17996AFD-DFC8-40F3-9D09-DB6DDAD5B7D6@cmcc.it> <7074B5D8-955A-4802-A9F3-606C99734417@cmcc.it> <83B84BF9-8334-4230-B6F8-0BC4BFBEFF15@cmcc.it> <133B9AE4-9792-4F72-AD91-D36E7B9EC711@cmcc.it> <6611C4B0-57FD-4390-88B5-BD373555D4C5@cmcc.it> Message-ID: +Gluster Devel , +Gluster-users I would like to point out another issue. Even if what I suggested prevents disconnects, part of the solution would be only symptomatic treatment and doesn't address the root cause of the problem. In most of the ping-timer-expiry issues, the root cause is the increased load on bricks and the inability of bricks to be responsive under high load. So, the actual solution would be doing any or both of the following: * identify the source of increased load and if possible throttle it. Internal heal processes like self-heal, rebalance, quota heal are known to pump traffic into bricks without much throttling (io-threads _might_ do some throttling, but my understanding is its not sufficient). * identify the reason for bricks to become unresponsive during load. This may be fixable issues like not enough event-threads to read from network or difficult to fix issues like fsync on backend fs freezing the process or semi fixable issues (in code) like lock contention. So any genuine effort to fix ping-timer-issues (to be honest most of the times they are not issues related to rpc/network) would involve performance characterization of various subsystems on bricks and clients. Various subsystems can include (but not necessarily limited to), underlying OS/filesystem, glusterfs processes, CPU consumption etc regards, Raghavendra On Mon, Mar 4, 2019 at 9:31 PM Mauro Tridici wrote: > Thank you, let?s try! > I will inform you about the effects of the change. > > Regards, > Mauro > > On 4 Mar 2019, at 16:55, Raghavendra Gowdappa wrote: > > > > On Mon, Mar 4, 2019 at 8:54 PM Mauro Tridici > wrote: > >> Hi Raghavendra, >> >> thank you for your reply. >> Yes, you are right. It is a problem that seems to happen randomly. >> At this moment, server.event-threads value is 4. I will try to increase >> this value to 8. Do you think that it could be a valid value ? >> > > Yes. We can try with that. You should see at least frequency of ping-timer > related disconnects reduce with this value (even if it doesn't eliminate > the problem completely). > > >> Regards, >> Mauro >> >> >> On 4 Mar 2019, at 15:36, Raghavendra Gowdappa >> wrote: >> >> >> >> On Mon, Mar 4, 2019 at 8:01 PM Nithya Balachandran >> wrote: >> >>> Hi Mauro, >>> >>> It looks like some problem on s06. Are all your other nodes ok? Can you >>> send us the gluster logs from this node? >>> >>> @Raghavendra G , do you have any idea as to >>> how this can be debugged? Maybe running top ? Or debug brick logs? >>> >> >> If we can reproduce the problem, collecting tcpdump on both ends of >> connection will help. But, one common problem is these bugs are >> inconsistently reproducible and hence we may not be able to capture tcpdump >> at correct intervals. Other than that, we can try to collect some evidence >> that poller threads were busy (waiting on locks). But, not sure what debug >> data provides that information. >> >> From what I know, its difficult to collect evidence for this issue and we >> could only reason about it. >> >> We can try a workaround though - try increasing server.event-threads and >> see whether ping-timer expiry issues go away with an optimal value. If >> that's the case, it kind of provides proof for our hypothesis. >> >> >>> >>> Regards, >>> Nithya >>> >>> On Mon, 4 Mar 2019 at 15:25, Mauro Tridici >>> wrote: >>> >>>> Hi All, >>>> >>>> some minutes ago I received this message from NAGIOS server >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> ****** Nagios *****Notification Type: PROBLEMService: Brick - >>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: CRITICALDate/Time: Mon >>>> Mar 4 10:25:33 CET 2019Additional Info:CHECK_NRPE STATE CRITICAL: Socket >>>> timeout after 10 seconds.* >>>> >>>> I checked the network, RAM and CPUs usage on s06 node and everything >>>> seems to be ok. >>>> No bricks are in error state. In /var/log/messages, I detected again a >>>> crash of ?check_vol_utili? that I think it is a module used by NRPE >>>> executable (that is the NAGIOS client). >>>> >>>> Mar 4 10:15:29 s06 kernel: traps: check_vol_utili[161224] general >>>> protection ip:7facffa0a66d sp:7ffe9f4e6fc0 error:0 in >>>> libglusterfs.so.0.0.1[7facff9b7000+f7000] >>>> Mar 4 10:15:29 s06 abrt-hook-ccpp: Process 161224 (python2.7) of user >>>> 0 killed by SIGSEGV - dumping core >>>> Mar 4 10:15:29 s06 abrt-server: Generating core_backtrace >>>> Mar 4 10:15:29 s06 abrt-server: Error: Unable to open './coredump': No >>>> such file or directory >>>> Mar 4 10:16:01 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:16:01 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:16:01 s06 systemd: Started Session 201010 of user root. >>>> Mar 4 10:16:01 s06 systemd: Starting Session 201010 of user root. >>>> Mar 4 10:16:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 4 10:16:01 s06 systemd: Stopping User Slice of root. >>>> Mar 4 10:16:24 s06 abrt-server: Duplicate: UUID >>>> Mar 4 10:16:24 s06 abrt-server: DUP_OF_DIR: >>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>> Mar 4 10:16:24 s06 abrt-server: Deleting problem directory >>>> ccpp-2019-03-04-10:15:29-161224 (dup of ccpp-2018-09-25-12:27:42-13041) >>>> Mar 4 10:16:24 s06 abrt-server: Generating core_backtrace >>>> Mar 4 10:16:24 s06 abrt-server: Error: Unable to open './coredump': No >>>> such file or directory >>>> Mar 4 10:16:24 s06 abrt-server: Cannot notify >>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>> 'report_uReport' exited with 1 >>>> Mar 4 10:16:24 s06 abrt-hook-ccpp: Process 161391 (python2.7) of user >>>> 0 killed by SIGABRT - dumping core >>>> Mar 4 10:16:25 s06 abrt-server: Generating core_backtrace >>>> Mar 4 10:16:25 s06 abrt-server: Error: Unable to open './coredump': No >>>> such file or directory >>>> Mar 4 10:17:01 s06 systemd: Created slice User Slice of root. >>>> >>>> Also, I noticed the following errors that I think are very critical: >>>> >>>> Mar 4 10:21:12 s06 glustershd[20355]: [2019-03-04 09:21:12.954798] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server >>>> 192.168.0.55:49158 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:22:01 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:22:01 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:22:01 s06 systemd: Started Session 201017 of user root. >>>> Mar 4 10:22:01 s06 systemd: Starting Session 201017 of user root. >>>> Mar 4 10:22:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 4 10:22:01 s06 systemd: Stopping User Slice of root. >>>> Mar 4 10:22:03 s06 glustershd[20355]: [2019-03-04 09:22:03.964120] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server >>>> 192.168.0.54:49165 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:23:01 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:23:01 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:23:01 s06 systemd: Started Session 201018 of user root. >>>> Mar 4 10:23:01 s06 systemd: Starting Session 201018 of user root. >>>> Mar 4 10:23:02 s06 systemd: Removed slice User Slice of root. >>>> Mar 4 10:23:02 s06 systemd: Stopping User Slice of root. >>>> Mar 4 10:24:01 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:24:01 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:24:01 s06 systemd: Started Session 201019 of user root. >>>> Mar 4 10:24:01 s06 systemd: Starting Session 201019 of user root. >>>> Mar 4 10:24:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 4 10:24:01 s06 systemd: Stopping User Slice of root. >>>> Mar 4 10:24:03 s06 glustershd[20355]: [2019-03-04 09:24:03.982502] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-16: server >>>> 192.168.0.52:49158 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746109] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-3: server >>>> 192.168.0.51:49153 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746215] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server >>>> 192.168.0.52:49156 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746260] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-21: server >>>> 192.168.0.51:49159 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746296] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server >>>> 192.168.0.52:49161 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746413] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server >>>> 192.168.0.54:49165 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:24:07 s06 glustershd[20355]: [2019-03-04 09:24:07.982952] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-45: server >>>> 192.168.0.54:49155 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:24:18 s06 glustershd[20355]: [2019-03-04 09:24:18.990929] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server >>>> 192.168.0.52:49161 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:24:31 s06 glustershd[20355]: [2019-03-04 09:24:31.995781] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-20: server >>>> 192.168.0.53:49159 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:25:01 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:25:01 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:25:01 s06 systemd: Started Session 201020 of user root. >>>> Mar 4 10:25:01 s06 systemd: Starting Session 201020 of user root. >>>> Mar 4 10:25:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 4 10:25:01 s06 systemd: Stopping User Slice of root. >>>> Mar 4 10:25:57 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:25:57 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:25:57 s06 systemd-logind: New session 201021 of user root. >>>> Mar 4 10:25:57 s06 systemd: Started Session 201021 of user root. >>>> Mar 4 10:25:57 s06 systemd: Starting Session 201021 of user root. >>>> Mar 4 10:26:01 s06 systemd: Started Session 201022 of user root. >>>> Mar 4 10:26:01 s06 systemd: Starting Session 201022 of user root. >>>> Mar 4 10:26:21 s06 nrpe[162388]: Error: Could not complete SSL >>>> handshake with 192.168.1.56: 5 >>>> Mar 4 10:27:01 s06 systemd: Started Session 201023 of user root. >>>> Mar 4 10:27:01 s06 systemd: Starting Session 201023 of user root. >>>> Mar 4 10:28:01 s06 systemd: Started Session 201024 of user root. >>>> Mar 4 10:28:01 s06 systemd: Starting Session 201024 of user root. >>>> Mar 4 10:29:01 s06 systemd: Started Session 201025 of user root. >>>> Mar 4 10:29:01 s06 systemd: Starting Session 201025 of user root. >>>> >>>> But, unfortunately, I don?t understand why it is happening. >>>> Now, NAGIOS server shows that s06 status is ok: >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> ****** Nagios *****Notification Type: RECOVERYService: Brick - >>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: OKDate/Time: Mon Mar 4 >>>> 10:35:23 CET 2019Additional Info:OK: Brick /gluster/mnt2/brick is up* >>>> >>>> Nothing is changed from RAM, CPUs, and NETWORK point of view. >>>> /var/log/message file has been updated: >>>> >>>> Mar 4 10:32:01 s06 systemd: Starting Session 201029 of user root. >>>> Mar 4 10:32:30 s06 glustershd[20355]: [2019-03-04 09:32:30.069082] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server >>>> 192.168.0.52:49156 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:32:55 s06 glustershd[20355]: [2019-03-04 09:32:55.074689] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-66: server >>>> 192.168.0.54:49167 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 4 10:33:01 s06 systemd: Started Session 201030 of user root. >>>> Mar 4 10:33:01 s06 systemd: Starting Session 201030 of user root. >>>> Mar 4 10:34:01 s06 systemd: Started Session 201031 of user root. >>>> Mar 4 10:34:01 s06 systemd: Starting Session 201031 of user root. >>>> Mar 4 10:35:01 s06 nrpe[162562]: Could not read request from client >>>> 192.168.1.56, bailing out... >>>> Mar 4 10:35:01 s06 nrpe[162562]: INFO: SSL Socket Shutdown. >>>> Mar 4 10:35:01 s06 systemd: Started Session 201032 of user root. >>>> Mar 4 10:35:01 s06 systemd: Starting Session 201032 of user root. >>>> >>>> Could you please help me to understand what it?s happening ? >>>> Thank you in advance. >>>> >>>> Rergards, >>>> Mauro >>>> >>>> >>>> On 1 Mar 2019, at 12:17, Mauro Tridici wrote: >>>> >>>> >>>> Thank you, Milind. >>>> I executed the instructions you suggested: >>>> >>>> - grep ?blocked for? /var/log/messages on s06 returns no output (no >>>> ?blocked? word is detected in messages file); >>>> - in /var/log/messages file I can see this kind of error repeated for a >>>> lot of times: >>>> >>>> Mar 1 08:43:01 s06 systemd: Starting Session 196071 of user root. >>>> Mar 1 08:43:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 1 08:43:01 s06 systemd: Stopping User Slice of root. >>>> Mar 1 08:43:02 s06 kernel: traps: check_vol_utili[57091] general >>>> protection ip:7f88e76ee66d sp:7ffe5a5bcc30 error:0 in >>>> libglusterfs.so.0.0.1[7f88e769b000+f7000] >>>> Mar 1 08:43:02 s06 abrt-hook-ccpp: Process 57091 (python2.7) of user 0 >>>> killed by SIGSEGV - dumping core >>>> Mar 1 08:43:02 s06 abrt-server: Generating core_backtrace >>>> Mar 1 08:43:02 s06 abrt-server: Error: Unable to open './coredump': No >>>> such file or directory >>>> Mar 1 08:43:58 s06 abrt-server: Duplicate: UUID >>>> Mar 1 08:43:58 s06 abrt-server: DUP_OF_DIR: >>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>> Mar 1 08:43:58 s06 abrt-server: Deleting problem directory >>>> ccpp-2019-03-01-08:43:02-57091 (dup of ccpp-2018-09-25-12:27:42-13041) >>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Activating service >>>> name='org.freedesktop.problems' (using servicehelper) >>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Successfully activated service >>>> 'org.freedesktop.problems' >>>> Mar 1 08:43:58 s06 abrt-server: Generating core_backtrace >>>> Mar 1 08:43:58 s06 abrt-server: Error: Unable to open './coredump': No >>>> such file or directory >>>> Mar 1 08:43:58 s06 abrt-server: Cannot notify >>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>> 'report_uReport' exited with 1 >>>> Mar 1 08:44:01 s06 systemd: Created slice User Slice of root. >>>> Mar 1 08:44:01 s06 systemd: Starting User Slice of root. >>>> Mar 1 08:44:01 s06 systemd: Started Session 196072 of user root. >>>> Mar 1 08:44:01 s06 systemd: Starting Session 196072 of user root. >>>> Mar 1 08:44:01 s06 systemd: Removed slice User Slice of root. >>>> >>>> - in /var/log/messages file I can see also 4 errors related to other >>>> cluster servers: >>>> >>>> Mar 1 11:05:01 s06 systemd: Starting User Slice of root. >>>> Mar 1 11:05:01 s06 systemd: Started Session 196230 of user root. >>>> Mar 1 11:05:01 s06 systemd: Starting Session 196230 of user root. >>>> Mar 1 11:05:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 1 11:05:01 s06 systemd: Stopping User Slice of root. >>>> Mar 1 11:05:59 s06 glustershd[70117]: [2019-03-01 10:05:59.347094] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-33: server >>>> 192.168.0.51:49163 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 1 11:06:01 s06 systemd: Created slice User Slice of root. >>>> Mar 1 11:06:01 s06 systemd: Starting User Slice of root. >>>> Mar 1 11:06:01 s06 systemd: Started Session 196231 of user root. >>>> Mar 1 11:06:01 s06 systemd: Starting Session 196231 of user root. >>>> Mar 1 11:06:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 1 11:06:01 s06 systemd: Stopping User Slice of root. >>>> Mar 1 11:06:12 s06 glustershd[70117]: [2019-03-01 10:06:12.351319] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-1: server >>>> 192.168.0.52:49153 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 1 11:06:38 s06 glustershd[70117]: [2019-03-01 10:06:38.356920] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-7: server >>>> 192.168.0.52:49155 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 1 11:07:01 s06 systemd: Created slice User Slice of root. >>>> Mar 1 11:07:01 s06 systemd: Starting User Slice of root. >>>> Mar 1 11:07:01 s06 systemd: Started Session 196232 of user root. >>>> Mar 1 11:07:01 s06 systemd: Starting Session 196232 of user root. >>>> Mar 1 11:07:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 1 11:07:01 s06 systemd: Stopping User Slice of root. >>>> Mar 1 11:07:36 s06 glustershd[70117]: [2019-03-01 10:07:36.366259] C >>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-0: server >>>> 192.168.0.51:49152 has not responded in the last 42 seconds, >>>> disconnecting. >>>> Mar 1 11:08:01 s06 systemd: Created slice User Slice of root. >>>> >>>> No ?blocked? word is in /var/log/messages files on other cluster >>>> servers. >>>> In attachment, the /var/log/messages file from s06 server. >>>> >>>> Thank you in advance, >>>> Mauro >>>> >>>> >>>> >>>> >>>> On 1 Mar 2019, at 11:47, Milind Changire wrote: >>>> >>>> The traces of very high disk activity on the servers are often found in >>>> /var/log/messages >>>> You might want to grep for "blocked for" in /var/log/messages on s06 >>>> and correlate the timestamps to confirm the unresponsiveness as reported in >>>> gluster client logs. >>>> In cases of high disk activity, although the operating system continues >>>> to respond to ICMP pings, the processes writing to disks often get blocked >>>> to a large flush to the disk which could span beyond 42 seconds and hence >>>> result in ping-timer-expiry logs. >>>> >>>> As a side note: >>>> If you indeed find gluster processes being blocked in >>>> /var/log/messages, you might want to tweak sysctl tunables called >>>> vm.dirty_background_ratio or vm.dirty_background_bytes to a smaller value >>>> than the existing. Please read up more on those tunables before touching >>>> the settings. >>>> >>>> >>>> On Fri, Mar 1, 2019 at 4:06 PM Mauro Tridici >>>> wrote: >>>> >>>>> >>>>> Hi all, >>>>> >>>>> in attachment the client log captured after changing >>>>> network.ping-timeout option. >>>>> I noticed this error involving server 192.168.0.56 (s06) >>>>> >>>>> [2019-03-01 09:23:36.077287] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >>>>> 0-tier2-client-71: changing ping timeout to 42 (from 0) >>>>> [2019-03-01 09:23:36.078213] I >>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>> volfile,continuing >>>>> [2019-03-01 09:23:36.078432] I >>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>> volfile,continuing >>>>> [2019-03-01 09:23:36.092357] I >>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>> volfile,continuing >>>>> [2019-03-01 09:23:36.094146] I >>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>> volfile,continuing >>>>> [2019-03-01 10:06:24.708082] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-50: server >>>>> 192.168.0.56:49156 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> >>>>> I don?t know why it happens, s06 server seems to be reachable. >>>>> >>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>> Trying 192.168.0.56... >>>>> Connected to 192.168.0.56. >>>>> Escape character is '^]'. >>>>> ^CConnection closed by foreign host. >>>>> [athena_login2][/users/home/sysm02/]> ping 192.168.0.56 >>>>> PING 192.168.0.56 (192.168.0.56) 56(84) bytes of data. >>>>> 64 bytes from 192.168.0.56: icmp_seq=1 ttl=64 time=0.116 ms >>>>> 64 bytes from 192.168.0.56: icmp_seq=2 ttl=64 time=0.101 ms >>>>> >>>>> --- 192.168.0.56 ping statistics --- >>>>> 2 packets transmitted, 2 received, 0% packet loss, time 1528ms >>>>> rtt min/avg/max/mdev = 0.101/0.108/0.116/0.012 ms >>>>> >>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>> Trying 192.168.0.56... >>>>> Connected to 192.168.0.56. >>>>> Escape character is '^]'. >>>>> >>>>> Thank you for your help, >>>>> Mauro >>>>> >>>>> >>>>> >>>>> On 1 Mar 2019, at 10:29, Mauro Tridici wrote: >>>>> >>>>> Hi all, >>>>> >>>>> thank you for the explanation. >>>>> I just changed network.ping-timeout option to default value >>>>> (network.ping-timeout=42). >>>>> >>>>> I will check the logs to see if the errors will appear again. >>>>> >>>>> Regards, >>>>> Mauro >>>>> >>>>> On 1 Mar 2019, at 04:43, Milind Changire wrote: >>>>> >>>>> network.ping-timeout should not be set to zero for non-glusterd >>>>> clients. >>>>> glusterd is a special case for which ping-timeout is set to zero via >>>>> /etc/glusterfs/glusterd.vol >>>>> >>>>> Setting network.ping-timeout to zero disables arming of the ping timer >>>>> for connections. This disables testing the connection for responsiveness >>>>> and hence avoids proactive fail-over. >>>>> >>>>> Please reset network.ping-timeout to a non-zero positive value, eg. 42 >>>>> >>>>> >>>>> On Thu, Feb 28, 2019 at 5:07 PM Nithya Balachandran < >>>>> nbalacha at redhat.com> wrote: >>>>> >>>>>> Adding Raghavendra and Milind to comment on this. >>>>>> >>>>>> What is the effect of setting network.ping-timeout to 0 and should it >>>>>> be set back to 42? >>>>>> Regards, >>>>>> Nithya >>>>>> >>>>>> On Thu, 28 Feb 2019 at 16:01, Mauro Tridici >>>>>> wrote: >>>>>> >>>>>>> Hi Nithya, >>>>>>> >>>>>>> sorry for the late. >>>>>>> network.ping-timeout has been set to 0 in order to try to solve some >>>>>>> timeout problems, but it didn?t help. >>>>>>> I can set it to the default value. >>>>>>> >>>>>>> Can I proceed with the change? >>>>>>> >>>>>>> Thank you, >>>>>>> Mauro >>>>>>> >>>>>>> >>>>>>> On 28 Feb 2019, at 04:41, Nithya Balachandran >>>>>>> wrote: >>>>>>> >>>>>>> Hi Mauro, >>>>>>> >>>>>>> Is network.ping-timeout still set to 0. The default value is 42. Is >>>>>>> there a particular reason why this was changed? >>>>>>> >>>>>>> Regards, >>>>>>> Nithya >>>>>>> >>>>>>> >>>>>>> On Wed, 27 Feb 2019 at 21:32, Mauro Tridici >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> Hi Xavi, >>>>>>>> >>>>>>>> thank you for the detailed explanation and suggestions. >>>>>>>> Yes, transport.listen-backlog option is still set to 1024. >>>>>>>> >>>>>>>> I will check the network and connectivity status using ?ping? and >>>>>>>> ?telnet? as soon as the errors will come back again. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Mauro >>>>>>>> >>>>>>>> Il giorno 27 feb 2019, alle ore 16:42, Xavi Hernandez < >>>>>>>> jahernan at redhat.com> ha scritto: >>>>>>>> >>>>>>>> Hi Mauro, >>>>>>>> >>>>>>>> those errors say that the mount point is not connected to some of >>>>>>>> the bricks while executing operations. I see references to 3rd and 6th >>>>>>>> bricks of several disperse sets, which seem to map to server s06. For some >>>>>>>> reason, gluster is having troubles connecting from the client machine to >>>>>>>> that particular server. At the end of the log I see that after long time a >>>>>>>> reconnect is done to both of them. However little after, other bricks from >>>>>>>> the s05 get disconnected and a reconnect times out. >>>>>>>> >>>>>>>> That's really odd. It seems like if server/communication is cut to >>>>>>>> s06 for some time, then restored, and then the same happens to the next >>>>>>>> server. >>>>>>>> >>>>>>>> If the servers are really online and it's only a communication >>>>>>>> issue, it explains why server memory and network has increased: if the >>>>>>>> problem only exists between the client and servers, any write made by the >>>>>>>> client will automatically mark the file as damaged, since some of the >>>>>>>> servers have not been updated. Since self-heal runs from the server nodes, >>>>>>>> they will probably be correctly connected to all bricks, which allows them >>>>>>>> to heal the just damaged file, which increases memory and network usage. >>>>>>>> >>>>>>>> I guess you still have transport.listen-backlog set to 1024, right ? >>>>>>>> >>>>>>>> Just to try to identify if the problem really comes from network, >>>>>>>> can you check if you lose some pings from the client to all of the servers >>>>>>>> while you are seeing those errors in the log file ? >>>>>>>> >>>>>>>> You can also check if during those errors, you can telnet to the >>>>>>>> port of the brick from the client. >>>>>>>> >>>>>>>> Xavi >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Feb 26, 2019 at 10:17 AM Mauro Tridici < >>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>> >>>>>>>>> Hi Nithya, >>>>>>>>> >>>>>>>>> ?df -h? operation is not still slow, but no users are using the >>>>>>>>> volume, RAM and NETWORK usage is ok on the client node. >>>>>>>>> >>>>>>>>> I was worried about this kind of warnings/errors: >>>>>>>>> >>>>>>>>> [2019-02-25 10:59:00.664323] W [MSGID: 122035] >>>>>>>>> [ec-common.c:571:ec_child_select] 0-tier2-disperse-6: Executing operation >>>>>>>>> with some subvolumes unavailable (20) >>>>>>>>> >>>>>>>>> [2019-02-26 03:11:35.212603] E >>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>> called at 2019-02-26 03:10:56.549903 (xid=0x106f1c5) >>>>>>>>> >>>>>>>>> [2019-02-26 03:13:03.313831] E >>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-50: connection to >>>>>>>>> 192.168.0.56:49156 failed (Timeout della connessione); >>>>>>>>> disconnecting socket >>>>>>>>> >>>>>>>>> It seems that some subvolumes are not available and 192.168.0.56 >>>>>>>>> server (s06) is not reachable. >>>>>>>>> But gluster servers are up&running and bricks are ok. >>>>>>>>> >>>>>>>>> In attachment the updated tier2.log file. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> Regards, >>>>>>>>> Mauro >>>>>>>>> >>>>>>>>> Il giorno 26 feb 2019, alle ore 04:03, Nithya Balachandran < >>>>>>>>> nbalacha at redhat.com> ha scritto: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I see a lot of EC messages in the log but they don't seem very >>>>>>>>> serious. Xavi, can you take a look? >>>>>>>>> >>>>>>>>> The only errors I see are: >>>>>>>>> [2019-02-25 10:58:45.519871] E >>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>> called at 2019-02-25 10:57:47.429969 (xid=0xd26fe7) >>>>>>>>> [2019-02-25 10:58:51.461493] E >>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>> 0-tier2-client-41: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>> called at 2019-02-25 10:57:47.499174 (xid=0xf47d6a) >>>>>>>>> [2019-02-25 11:07:57.152874] E >>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-70: connection to >>>>>>>>> 192.168.0.55:49163 failed (Timeout della connessione); >>>>>>>>> disconnecting socket >>>>>>>>> >>>>>>>>> >>>>>>>>> Is the df -h operation still slow? If yes, can you take a tcpdump >>>>>>>>> of the client while running df -h and send that across? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Nithya >>>>>>>>> >>>>>>>>> On Mon, 25 Feb 2019 at 17:27, Mauro Tridici >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Sorry, some minutes after my last mail message, I noticed that >>>>>>>>>> ?df -h? command hanged for a while before returns the prompt. >>>>>>>>>> Yesterday, everything was ok in the gluster client log, but, >>>>>>>>>> today, I see a lot of errors (please, take a look to the attached file). >>>>>>>>>> >>>>>>>>>> On the client node, I detected an important RAM e NETWORK usage. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Do you think that the errors have been caused by the client >>>>>>>>>> resources usage? >>>>>>>>>> >>>>>>>>>> Thank you in advance, >>>>>>>>>> Mauro >>>>>>>>>> >>>>>>>>>> >>>> >> > ------------------------- > Mauro Tridici > > Fondazione CMCC > CMCC Supercomputing Center > presso Complesso Ecotekne - Universit? del Salento - > Strada Prov.le Lecce - Monteroni sn > 73100 Lecce IT > http://www.cmcc.it > > mobile: (+39) 327 5630841 > email: mauro.tridici at cmcc.it > https://it.linkedin.com/in/mauro-tridici-5977238b > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aspandey at redhat.com Tue Mar 5 08:51:43 2019 From: aspandey at redhat.com (Ashish Pandey) Date: Tue, 5 Mar 2019 03:51:43 -0500 (EST) Subject: [Gluster-devel] Gluster : Improvements on "heal info" command In-Reply-To: <1555599631.5547169.1551771451185.JavaMail.zimbra@redhat.com> Message-ID: <1136179364.5559263.1551775903014.JavaMail.zimbra@redhat.com> Hi All, We have observed and heard from gluster users about the long time "heal info" command takes. Even when we all want to know if a gluster volume is healthy or not, it takes time to list down all the files from all the bricks after which we can be sure if the volume is healthy or not. Here, we have come up with some options for "heal info" command which provide report quickly and reliably. gluster v heal vol info --subvol=[number of the subvol] --brick=[one,all] -------- Problem: "gluster v heal info" command picks each subvolume and checks the .glusterfs/indices/xattrop folder of every brick of that subvolume to find out if there is any entry which needs to be healed. It picks the entry and takes a lock on that entry to check xattrs to find out if that entry actually needs heal or not. This LOCK->CHECK-XATTR->UNLOCK cycle takes lot of time for each file. Let's consider two most often seen cases for which we use "heal info" and try to understand the improvements. Case -1 : Consider 4+2 EC volume and all the bricks on 6 different nodes. A brick of the volume is down and client has written 10000 files on one of the mount point of this volume. Entries for these 10K files will be created on ".glusterfs/indices/xattrop" on all the rest of 5 bricks. Now, brick is UP and when we use "heal info" command for this volume, it goes to all the bricks and picks these 10K file entries and goes through LOCK->CHECK-XATTR->UNLOCK cycle for all the files. This happens for all the bricks, that means, we check 50K files and perform the LOCK->CHECK-XATTR->UNLOCK cycle 50K times, while only 10K entries were sufficient to check. It is a very time consuming operation. If IO"s are happening one some of the new files, we check these files also which will add the time. Here, all we wanted to know if our volume has been healed and healthy. Solution : Whenever a brick goes down and comes up and when we use "heal info" command, our *main intention* is to find out if the volume is *healthy* or *unhealthy*. A volume is unhealthy even if one file is not healthy. So, we should scan bricks one by one and as soon as we find that one brick is having some entries which require to be healed, we can come out and list the files and say the volume is not healthy. No need to scan rest of the bricks. That's where "--brick=[one,all]" option has been introduced. "gluster v heal vol info --brick=[one,all]" "one" - It will scan the brick sequentially and as soon as it will find any unhealthy entries, it will list it out and stop scanning other bricks. "all" - It will act just like current behavior and provide all the files from all the bricks. If we do not provide this option, default (current) behavior will be applicable. Case -2 : Consider 24 X (4+2) EC volume. Let's say one brick from *only one* of the sub volume has been replaced and a heal has been triggered. To know if the volume is in healthy state, we go to each brick of *each and every sub volume* and check if there are any entries in ".glusterfs/indices/xattrop" folder which need heal or not. If we know which sub volume participated in brick replacement, we just need to check health of that sub volume and not query/check other sub volumes. If several clients are writing number of files on this volume, an entry for each of these files will be created in .glusterfs/indices/xattrop and "heal info' command will go through LOCK->CHECK-XATTR->UNLOCK cycle to find out if these entries need heal or not which takes lot of time. In addition to this a client will also see performance drop as it will have to release and take lock again. Solution: Provide an option to mention number of sub volume for which we want to check heal info. "gluster v heal vol info --subvol= " Here, --subvol will be given number of the subvolume we want to check. Example: "gluster v heal vol info --subvol=1 " =================================== Performance Data - A quick performance test done on standalone system. Type: Distributed-Disperse Volume ID: ea40eb13-d42c-431c-9c89-0153e834e67e Status: Started Snapshot Count: 0 Number of Bricks: 2 x (4 + 2) = 12 Transport-type: tcp Bricks: Brick1: apandey:/home/apandey/bricks/gluster/vol-1 Brick2: apandey:/home/apandey/bricks/gluster/vol-2 Brick3: apandey:/home/apandey/bricks/gluster/vol-3 Brick4: apandey:/home/apandey/bricks/gluster/vol-4 Brick5: apandey:/home/apandey/bricks/gluster/vol-5 Brick6: apandey:/home/apandey/bricks/gluster/vol-6 Brick7: apandey:/home/apandey/bricks/gluster/new-1 Brick8: apandey:/home/apandey/bricks/gluster/new-2 Brick9: apandey:/home/apandey/bricks/gluster/new-3 Brick10: apandey:/home/apandey/bricks/gluster/new-4 Brick11: apandey:/home/apandey/bricks/gluster/new-5 Brick12: apandey:/home/apandey/bricks/gluster/new-6 Just disabled the shd to get the data - Killed one brick each from two subvolumes and wrote 2000 files on mount point. [root at apandey vol]# for i in {1..2000};do echo abc >> file-$i; done Start the volume using force option and get the heal info. Following is the data - [root at apandey glusterfs]# time gluster v heal vol info --brick=one >> /dev/null <<<<<<<< This will scan brick one by one and come out as soon as we find volume is unhealthy. real 0m8.316s user 0m2.241s sys 0m1.278s [root at apandey glusterfs]# [root at apandey glusterfs]# time gluster v heal vol info >> /dev/null <<<<<<<< This is current behavior. real 0m26.097s user 0m10.868s sys 0m6.198s [root at apandey glusterfs]# =================================== I would like your comments/suggestions on this improvements. Specially, would like to hear on the new syntax of the command - gluster v heal vol info --subvol=[number of the subvol] --brick=[one,all] Note that if we do not provide new options, command will behave just like it does right now. Also, this improvement is valid for AFR and EC. --- Ashish -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Tue Mar 5 09:13:43 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Tue, 5 Mar 2019 14:43:43 +0530 Subject: [Gluster-devel] Experiences with FUSE in real world - Presentation at Vault 2019 Message-ID: All, Recently me, Manoj and Csaba presented on positives and negatives of implementing File systems in userspace using FUSE [1]. We had based the talk on our experiences with Glusterfs having FUSE as the native interface. The slides can also be found at [1]. [1] https://www.usenix.org/conference/vault19/presentation/pillai regards, Raghavendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From khiremat at redhat.com Tue Mar 5 15:13:23 2019 From: khiremat at redhat.com (Kotresh Hiremath Ravishankar) Date: Tue, 5 Mar 2019 20:43:23 +0530 Subject: [Gluster-devel] Bitrot: Time of signing depending on the file size??? In-Reply-To: References: Message-ID: Hi David, Thanks for raising the bug. But from the above validation, it's clear that bitrot is not directly involved. Bitrot waits for last fd to be closed. We will have to investigate the reason for fd not being closed for large files. Thanks, Kotresh HR On Mon, Mar 4, 2019 at 3:13 PM David Spisla wrote: > Hello Kotresh, > > Yes, the fd was still open for larger files. I could verify this with a > 500MiB file and some smaller files. After a specific time only the fd for > the 500MiB was up and the file still had no signature, for the smaller > files there were no fds and they already had a signature. I don't know the > reason for this. Maybe the client still keep th fd open? I opened a bug for > this: > https://bugzilla.redhat.com/show_bug.cgi?id=1685023 > > Regards > David > > Am Fr., 1. M?rz 2019 um 18:29 Uhr schrieb Kotresh Hiremath Ravishankar < > khiremat at redhat.com>: > >> Interesting observation! But as discussed in the thread bitrot signing >> processes depends 2 min timeout (by default) after last fd closes. It >> doesn't have any co-relation with the size of the file. >> Did you happen to verify that the fd was still open for large files for >> some reason? >> >> >> >> On Fri, Mar 1, 2019 at 1:19 PM David Spisla wrote: >> >>> Hello folks, >>> >>> I did some observations concerning the bitrot daemon. It seems to be >>> that the bitrot signer is signing files depending on file size. I copied >>> files with different sizes into a volume and I was wonderung because the >>> files get their signature not the same time (I keep the expiry time default >>> with 120). Here are some examples: >>> >>> 300 KB file ~2-3 m >>> 70 MB file ~ 40 m >>> 115 MB file ~ 1 Sh >>> 800 MB file ~ 4,5 h >>> >>> What is the expected behaviour here? >>> Why does it take so long to sign a 800MB file? >>> What about 500GB or 1TB? >>> Is there a way to speed up the sign process? >>> >>> My ambition is to understand this observation >>> >>> Regards >>> David Spisla >>> _______________________________________________ >>> Gluster-devel mailing list >>> Gluster-devel at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-devel >> >> >> >> -- >> Thanks and Regards, >> Kotresh H R >> > -- Thanks and Regards, Kotresh H R -------------- next part -------------- An HTML attachment was scrubbed... URL: From spisla80 at gmail.com Tue Mar 5 15:30:18 2019 From: spisla80 at gmail.com (David Spisla) Date: Tue, 5 Mar 2019 16:30:18 +0100 Subject: [Gluster-devel] Bitrot: Time of signing depending on the file size??? In-Reply-To: References: Message-ID: Hello Kotresh, allrigt, I have updated the "Component" field in the bug to 'core' . I am looking forward fixing this bug Regards David Spisla Am Di., 5. M?rz 2019 um 16:13 Uhr schrieb Kotresh Hiremath Ravishankar < khiremat at redhat.com>: > Hi David, > > Thanks for raising the bug. But from the above validation, it's clear that > bitrot is not directly involved. Bitrot waits for last fd to be closed. We > will have to investigate the reason for fd not being closed for large files. > > Thanks, > Kotresh HR > > On Mon, Mar 4, 2019 at 3:13 PM David Spisla wrote: > >> Hello Kotresh, >> >> Yes, the fd was still open for larger files. I could verify this with a >> 500MiB file and some smaller files. After a specific time only the fd for >> the 500MiB was up and the file still had no signature, for the smaller >> files there were no fds and they already had a signature. I don't know the >> reason for this. Maybe the client still keep th fd open? I opened a bug for >> this: >> https://bugzilla.redhat.com/show_bug.cgi?id=1685023 >> >> Regards >> David >> >> Am Fr., 1. M?rz 2019 um 18:29 Uhr schrieb Kotresh Hiremath Ravishankar < >> khiremat at redhat.com>: >> >>> Interesting observation! But as discussed in the thread bitrot signing >>> processes depends 2 min timeout (by default) after last fd closes. It >>> doesn't have any co-relation with the size of the file. >>> Did you happen to verify that the fd was still open for large files for >>> some reason? >>> >>> >>> >>> On Fri, Mar 1, 2019 at 1:19 PM David Spisla wrote: >>> >>>> Hello folks, >>>> >>>> I did some observations concerning the bitrot daemon. It seems to be >>>> that the bitrot signer is signing files depending on file size. I copied >>>> files with different sizes into a volume and I was wonderung because the >>>> files get their signature not the same time (I keep the expiry time default >>>> with 120). Here are some examples: >>>> >>>> 300 KB file ~2-3 m >>>> 70 MB file ~ 40 m >>>> 115 MB file ~ 1 Sh >>>> 800 MB file ~ 4,5 h >>>> >>>> What is the expected behaviour here? >>>> Why does it take so long to sign a 800MB file? >>>> What about 500GB or 1TB? >>>> Is there a way to speed up the sign process? >>>> >>>> My ambition is to understand this observation >>>> >>>> Regards >>>> David Spisla >>>> _______________________________________________ >>>> Gluster-devel mailing list >>>> Gluster-devel at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-devel >>> >>> >>> >>> -- >>> Thanks and Regards, >>> Kotresh H R >>> >> > > -- > Thanks and Regards, > Kotresh H R > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srangana at redhat.com Tue Mar 5 18:21:11 2019 From: srangana at redhat.com (Shyam Ranganathan) Date: Tue, 5 Mar 2019 13:21:11 -0500 Subject: [Gluster-devel] [Gluster-Maintainers] GlusterFS - 6.0RC - Test days (27th, 28th Feb) In-Reply-To: <5f27393d-cefb-aa47-12f8-1597677ffb50@redhat.com> References: <5f27393d-cefb-aa47-12f8-1597677ffb50@redhat.com> Message-ID: <2257667e-751a-2c0e-6c99-b7aedbab379d@redhat.com> On 3/4/19 12:33 PM, Shyam Ranganathan wrote: > On 3/4/19 10:08 AM, Atin Mukherjee wrote: >> >> >> On Mon, 4 Mar 2019 at 20:33, Amar Tumballi Suryanarayan >> > wrote: >> >> Thanks to those who participated. >> >> Update at present: >> >> We found 3 blocker bugs in upgrade scenarios, and hence have marked >> release >> as pending upon them. We will keep these lists updated about progress. >> >> >> I?d like to clarify that upgrade testing is blocked. So just fixing >> these test blocker(s) isn?t enough to call release-6 green. We need to >> continue and finish the rest of the upgrade tests once the respective >> bugs are fixed. > > Based on fixes expected by tomorrow for the upgrade fixes, we will build > an RC1 candidate on Wednesday (6-Mar) (tagging early Wed. Eastern TZ). > This RC can be used for further testing. There have been no backports for the upgrade failures, request folks working on the same to post a list of bugs that need to be fixed, to enable tracking the same. (also, ensure they are marked against the release-6 tracker [1]) Also, we need to start writing out the upgrade guide for release-6, any volunteers for the same? Thanks, Shyam [1] Release-6 tracker bug: https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-6.0 From abhishpaliwal at gmail.com Wed Mar 6 04:35:44 2019 From: abhishpaliwal at gmail.com (ABHISHEK PALIWAL) Date: Wed, 6 Mar 2019 10:05:44 +0530 Subject: [Gluster-devel] Not able to start glusterd Message-ID: Hi Team, I am facing the issue where at the time of starting the glusterd segmentation fault is reported. Below are the logs root at 128:/usr/sbin# ./glusterd --debug [1970-01-01 15:19:43.940386] I [MSGID: 100030] [glusterfsd.c:2691:main] 0-./glusterd: Started running ./glusterd version 5.0 (args: ./glusterd --debug) [1970-01-01 15:19:43.940855] D [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5 [1970-01-01 15:19:43.941736] D [MSGID: 0] [glusterfsd.c:747:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol [1970-01-01 15:19:43.945796] D [MSGID: 101097] [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on /usr/lib64/glusterfs/5.0/xlator/mgmt/glusterd.so: undefined symbol: xlator_api. Fall back to old symbols [1970-01-01 15:19:43.946279] I [MSGID: 106478] [glusterd.c:1435:init] 0-management: Maximum allowed open file descriptors set to 65536 [1970-01-01 15:19:43.946419] I [MSGID: 106479] [glusterd.c:1491:init] 0-management: Using /var/lib/glusterd as working directory [1970-01-01 15:19:43.946515] I [MSGID: 106479] [glusterd.c:1497:init] 0-management: Using /var/run/gluster as pid file working directory [1970-01-01 15:19:43.946968] D [MSGID: 0] [glusterd.c:458:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog value: 10 [1970-01-01 15:19:43.947139] D [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: RPC service inited. [1970-01-01 15:19:43.947241] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [1970-01-01 15:19:43.947379] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.0/rpc-transport/socket.so [1970-01-01 15:19:43.955198] D [socket.c:4464:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [1970-01-01 15:19:43.955316] D [socket.c:4482:socket_init] 0-socket.management: Reconfigued transport.keepalivecnt=9 [1970-01-01 15:19:43.955415] D [socket.c:4167:ssl_setup_connection_params] 0-socket.management: SSL support on the I/O path is NOT enabled [1970-01-01 15:19:43.955504] D [socket.c:4170:ssl_setup_connection_params] 0-socket.management: SSL support for glusterd is NOT enabled [1970-01-01 15:19:43.955612] D [name.c:572:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet6 [1970-01-01 15:19:43.955928] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so [1970-01-01 15:19:43.956079] E [rpc-transport.c:273:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [1970-01-01 15:19:43.956177] W [rpc-transport.c:277:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [1970-01-01 15:19:43.956270] W [rpcsvc.c:1789:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [1970-01-01 15:19:43.956362] E [MSGID: 106244] [glusterd.c:1798:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [1970-01-01 15:19:43.956459] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 [1970-01-01 15:19:43.956561] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 [1970-01-01 15:19:43.956666] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 [1970-01-01 15:19:43.956758] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0 [1970-01-01 15:19:43.956853] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 [1970-01-01 15:19:43.956946] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 [1970-01-01 15:19:43.957062] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 [1970-01-01 15:19:43.957205] D [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: RPC service inited. [1970-01-01 15:19:43.957303] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [1970-01-01 15:19:43.957408] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.0/rpc-transport/socket.so [1970-01-01 15:19:43.957563] D [socket.c:4424:socket_init] 0-socket.management: disabling nodelay [1970-01-01 15:19:43.957650] D [socket.c:4464:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0 [1970-01-01 15:19:43.957738] D [socket.c:4482:socket_init] 0-socket.management: Reconfigued transport.keepalivecnt=9 [1970-01-01 15:19:43.957830] D [socket.c:4167:ssl_setup_connection_params] 0-socket.management: SSL support on the I/O path is NOT enabled [1970-01-01 15:19:43.957922] D [socket.c:4170:ssl_setup_connection_params] 0-socket.management: SSL support for glusterd is NOT enabled [1970-01-01 15:19:43.958186] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [1970-01-01 15:19:43.958280] D [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 [1970-01-01 15:19:43.958461] D [MSGID: 0] [glusterd-utils.c:7878:glusterd_sm_tr_log_init] 0-glusterd: returning 0 [1970-01-01 15:19:43.958557] D [MSGID: 0] [glusterd.c:1875:init] 0-management: cannot get run-with-valgrind value [1970-01-01 15:19:43.960895] E [MSGID: 101032] [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [1970-01-01 15:19:43.961016] D [MSGID: 0] [store.c:452:gf_store_handle_retrieve] 0-: Returning -1 [1970-01-01 15:19:43.961108] D [MSGID: 0] [glusterd-store.c:2169:glusterd_retrieve_op_version] 0-management: Unable to get store handle! [1970-01-01 15:19:43.961216] E [MSGID: 101032] [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory] [1970-01-01 15:19:43.961325] D [MSGID: 0] [store.c:452:gf_store_handle_retrieve] 0-: Returning -1 [1970-01-01 15:19:43.961428] D [MSGID: 0] [glusterd-store.c:2345:glusterd_retrieve_uuid] 0-management: Unable to get storehandle! [1970-01-01 15:19:43.961523] D [MSGID: 0] [glusterd-store.c:2366:glusterd_retrieve_uuid] 0-management: Returning -1 [1970-01-01 15:19:43.961617] I [MSGID: 106514] [glusterd-store.c:2304:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 50000 [1970-01-01 15:19:43.962658] D [MSGID: 0] [store.c:432:gf_store_handle_new] 0-: Returning 0 [1970-01-01 15:19:43.962750] D [MSGID: 0] [store.c:452:gf_store_handle_retrieve] 0-: Returning 0 [1970-01-01 15:19:43.963047] D [MSGID: 0] [store.c:515:gf_store_iter_new] 0-: Returning with 0 [1970-01-01 15:19:43.963194] D [MSGID: 0] [store.c:632:gf_store_iter_get_next] 0-: Returning with 0 [1970-01-01 15:19:43.963318] D [MSGID: 0] [store.c:632:gf_store_iter_get_next] 0-: Returning with -1 [1970-01-01 15:19:43.963455] D [MSGID: 0] [store.c:473:gf_store_handle_destroy] 0-: Returning 0 [1970-01-01 15:19:43.963757] D [MSGID: 0] [glusterd-store.c:3546:glusterd_store_retrieve_volumes] 0-management: Returning with 0 [1970-01-01 15:19:43.964159] D [MSGID: 0] [glusterd-store.c:4662:glusterd_store_retrieve_peers] 0-management: Returning with 0 [1970-01-01 15:19:43.964471] I [MSGID: 106194] [glusterd-store.c:3983:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [1970-01-01 15:19:43.964580] D [MSGID: 0] [glusterd-store.c:4104:glusterd_store_retrieve_snaps] 0-management: Returning with 0 [1970-01-01 15:19:43.964680] D [MSGID: 0] [glusterd-store.c:4894:glusterd_restore] 0-management: Returning 0 [1970-01-01 15:19:43.965060] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-management: option event-threads using set value 1 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.listen-backlog 10 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [1970-01-01 15:19:43.966808] I [MSGID: 101190] [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [1970-01-01 15:19:44.840454] E [rpcsvc.c:513:rpcsvc_request_create] 0-rpc-service: RPC version not supported (XID: 0x0, Ver: 0, Program: 0, ProgVers: 0, Proc: 2) from trans (socket.management) [1970-01-01 15:19:44.840884] D [rpcsvc.c:1416:rpcsvc_error_reply] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xffbac)[0x3fffa12acbe4] (--> /usr/lib64/libgfrpc.so.0(+0xc5f4)[0x3fffa12525f4] (--> /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00] (--> /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224] (--> /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c] ))))) 0-: sending a RPC error reply [1970-01-01 15:19:44.841055] D [logging.c:1805:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 5 extra log messages [1970-01-01 15:19:44.841156] D [logging.c:1808:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 5 extra log messages pending frames: patchset: git://git.gluster.org/glusterfs.git signal received: 11 time of crash: 1970-01-01 15:19:44 configuration details: argp 1 backtrace 1 dlfcn 1 libpthread 1 llistxattr 1 setfsid 1 spinlock 1 epoll.h 1 xattr.h 1 st_atim.tv_nsec 1 package-string: glusterfs 5.0 /usr/lib64/libglusterfs.so.0(+0x422a4)[0x3fffa12ab2a4] /usr/lib64/libglusterfs.so.0(gf_print_trace-0xf5080)[0x3fffa12b82e0] ./glusterd(glusterfsd_print_trace-0x22fa4)[0x100067ec] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fffa13f0478] /lib64/libc.so.6(xdr_accepted_reply-0x72d3c)[0x3fffa11375cc] /lib64/libc.so.6(xdr_accepted_reply-0x72d9c)[0x3fffa113756c] /lib64/libc.so.6(xdr_union-0x63a94)[0x3fffa1147dd4] /lib64/libc.so.6(xdr_replymsg-0x72c58)[0x3fffa11376e0] /lib64/libc.so.6(xdr_sizeof-0x62a78)[0x3fffa1149120] /usr/lib64/libgfrpc.so.0(+0x9b0c)[0x3fffa124fb0c] /usr/lib64/libgfrpc.so.0(rpcsvc_submit_generic-0x149f4)[0x3fffa125228c] /usr/lib64/libgfrpc.so.0(+0xc614)[0x3fffa1252614] /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00] /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224] /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c] /usr/lib64/libgfrpc.so.0(rpc_transport_notify-0x10eec)[0x3fffa125610c] /usr/lib64/glusterfs/5.0/rpc-transport/socket.so(+0xc09c)[0x3fff9d51709c] /usr/lib64/libglusterfs.so.0(+0xb84bc)[0x3fffa13214bc] /lib64/libpthread.so.0(+0xbb30)[0x3fffa11bdb30] /lib64/libc.so.6(clone-0x9e964)[0x3fffa110817c] --------- Segmentation fault (core dumped) Could you please help me, what actually the problem? -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: From srakonde at redhat.com Wed Mar 6 07:28:08 2019 From: srakonde at redhat.com (Sanju Rakonde) Date: Wed, 6 Mar 2019 12:58:08 +0530 Subject: [Gluster-devel] [Gluster-users] Not able to start glusterd In-Reply-To: References: Message-ID: Abhishek, We need below information on investigate this issue. 1. gluster --version 2. Please run glusterd in gdb, so that we can capture the backtrace. I see some rpc errors in log, but backtrace will be more helpful. To run glusterd in gdb, you need start glusterd in gdb (i.e. gdb glusterd, and then give the command "run -N"). when you see a segmentation fault, please capture the backtrace and paste it here. On Wed, Mar 6, 2019 at 10:07 AM ABHISHEK PALIWAL wrote: > Hi Team, > > I am facing the issue where at the time of starting the glusterd > segmentation fault is reported. > > Below are the logs > > root at 128:/usr/sbin# ./glusterd --debug > [1970-01-01 15:19:43.940386] I [MSGID: 100030] [glusterfsd.c:2691:main] > 0-./glusterd: Started running ./glusterd version 5.0 (args: ./glusterd > --debug) > [1970-01-01 15:19:43.940855] D > [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting > timer now. Timeout = 120, current buf size = 5 > [1970-01-01 15:19:43.941736] D [MSGID: 0] [glusterfsd.c:747:get_volfp] > 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol > [1970-01-01 15:19:43.945796] D [MSGID: 101097] > [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on > /usr/lib64/glusterfs/5.0/xlator/mgmt/glusterd.so: undefined symbol: > xlator_api. Fall back to old symbols > [1970-01-01 15:19:43.946279] I [MSGID: 106478] [glusterd.c:1435:init] > 0-management: Maximum allowed open file descriptors set to 65536 > [1970-01-01 15:19:43.946419] I [MSGID: 106479] [glusterd.c:1491:init] > 0-management: Using /var/lib/glusterd as working directory > [1970-01-01 15:19:43.946515] I [MSGID: 106479] [glusterd.c:1497:init] > 0-management: Using /var/run/gluster as pid file working directory > [1970-01-01 15:19:43.946968] D [MSGID: 0] > [glusterd.c:458:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog > value: 10 > [1970-01-01 15:19:43.947139] D [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: > RPC service inited. > [1970-01-01 15:19:43.947241] D [rpcsvc.c:2146:rpcsvc_program_register] > 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, > Port: 0 > [1970-01-01 15:19:43.947379] D [rpc-transport.c:269:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib64/glusterfs/5.0/rpc-transport/socket.so > [1970-01-01 15:19:43.955198] D [socket.c:4464:socket_init] > 0-socket.management: Configued transport.tcp-user-timeout=0 > [1970-01-01 15:19:43.955316] D [socket.c:4482:socket_init] > 0-socket.management: Reconfigued transport.keepalivecnt=9 > [1970-01-01 15:19:43.955415] D [socket.c:4167:ssl_setup_connection_params] > 0-socket.management: SSL support on the I/O path is NOT enabled > [1970-01-01 15:19:43.955504] D [socket.c:4170:ssl_setup_connection_params] > 0-socket.management: SSL support for glusterd is NOT enabled > [1970-01-01 15:19:43.955612] D [name.c:572:server_fill_address_family] > 0-socket.management: option address-family not specified, defaulting to > inet6 > [1970-01-01 15:19:43.955928] D [rpc-transport.c:269:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so > [1970-01-01 15:19:43.956079] E [rpc-transport.c:273:rpc_transport_load] > 0-rpc-transport: /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so: cannot > open shared object file: No such file or directory > [1970-01-01 15:19:43.956177] W [rpc-transport.c:277:rpc_transport_load] > 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not > valid or not found on this machine > [1970-01-01 15:19:43.956270] W [rpcsvc.c:1789:rpcsvc_create_listener] > 0-rpc-service: cannot create listener, initing the transport failed > [1970-01-01 15:19:43.956362] E [MSGID: 106244] [glusterd.c:1798:init] > 0-management: creation of 1 listeners failed, continuing with succeeded > transport > [1970-01-01 15:19:43.956459] D [rpcsvc.c:2146:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, > Ver: 2, Port: 0 > [1970-01-01 15:19:43.956561] D [rpcsvc.c:2146:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: > 1238463, Ver: 2, Port: 0 > [1970-01-01 15:19:43.956666] D [rpcsvc.c:2146:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, > Ver: 2, Port: 0 > [1970-01-01 15:19:43.956758] D [rpcsvc.c:2146:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, > Ver: 3, Port: 0 > [1970-01-01 15:19:43.956853] D [rpcsvc.c:2146:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: > 1, Port: 0 > [1970-01-01 15:19:43.956946] D [rpcsvc.c:2146:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, > Ver: 2, Port: 0 > [1970-01-01 15:19:43.957062] D [rpcsvc.c:2146:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: > 1239873, Ver: 1, Port: 0 > [1970-01-01 15:19:43.957205] D [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: > RPC service inited. > [1970-01-01 15:19:43.957303] D [rpcsvc.c:2146:rpcsvc_program_register] > 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, > Port: 0 > [1970-01-01 15:19:43.957408] D [rpc-transport.c:269:rpc_transport_load] > 0-rpc-transport: attempt to load file > /usr/lib64/glusterfs/5.0/rpc-transport/socket.so > [1970-01-01 15:19:43.957563] D [socket.c:4424:socket_init] > 0-socket.management: disabling nodelay > [1970-01-01 15:19:43.957650] D [socket.c:4464:socket_init] > 0-socket.management: Configued transport.tcp-user-timeout=0 > [1970-01-01 15:19:43.957738] D [socket.c:4482:socket_init] > 0-socket.management: Reconfigued transport.keepalivecnt=9 > [1970-01-01 15:19:43.957830] D [socket.c:4167:ssl_setup_connection_params] > 0-socket.management: SSL support on the I/O path is NOT enabled > [1970-01-01 15:19:43.957922] D [socket.c:4170:ssl_setup_connection_params] > 0-socket.management: SSL support for glusterd is NOT enabled > [1970-01-01 15:19:43.958186] D [rpcsvc.c:2146:rpcsvc_program_register] > 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: > 2, Port: 0 > [1970-01-01 15:19:43.958280] D [rpcsvc.c:2146:rpcsvc_program_register] > 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), > Num: 14398633, Ver: 2, Port: 0 > [1970-01-01 15:19:43.958461] D [MSGID: 0] > [glusterd-utils.c:7878:glusterd_sm_tr_log_init] 0-glusterd: returning 0 > [1970-01-01 15:19:43.958557] D [MSGID: 0] [glusterd.c:1875:init] > 0-management: cannot get run-with-valgrind value > [1970-01-01 15:19:43.960895] E [MSGID: 101032] > [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding to > /var/lib/glusterd/glusterd.info. [No such file or directory] > [1970-01-01 15:19:43.961016] D [MSGID: 0] > [store.c:452:gf_store_handle_retrieve] 0-: Returning -1 > [1970-01-01 15:19:43.961108] D [MSGID: 0] > [glusterd-store.c:2169:glusterd_retrieve_op_version] 0-management: Unable > to get store handle! > [1970-01-01 15:19:43.961216] E [MSGID: 101032] > [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding to > /var/lib/glusterd/glusterd.info. [No such file or directory] > [1970-01-01 15:19:43.961325] D [MSGID: 0] > [store.c:452:gf_store_handle_retrieve] 0-: Returning -1 > [1970-01-01 15:19:43.961428] D [MSGID: 0] > [glusterd-store.c:2345:glusterd_retrieve_uuid] 0-management: Unable to get > storehandle! > [1970-01-01 15:19:43.961523] D [MSGID: 0] > [glusterd-store.c:2366:glusterd_retrieve_uuid] 0-management: Returning -1 > [1970-01-01 15:19:43.961617] I [MSGID: 106514] > [glusterd-store.c:2304:glusterd_restore_op_version] 0-management: Detected > new install. Setting op-version to maximum : 50000 > [1970-01-01 15:19:43.962658] D [MSGID: 0] > [store.c:432:gf_store_handle_new] 0-: Returning 0 > [1970-01-01 15:19:43.962750] D [MSGID: 0] > [store.c:452:gf_store_handle_retrieve] 0-: Returning 0 > [1970-01-01 15:19:43.963047] D [MSGID: 0] [store.c:515:gf_store_iter_new] > 0-: Returning with 0 > [1970-01-01 15:19:43.963194] D [MSGID: 0] > [store.c:632:gf_store_iter_get_next] 0-: Returning with 0 > [1970-01-01 15:19:43.963318] D [MSGID: 0] > [store.c:632:gf_store_iter_get_next] 0-: Returning with -1 > [1970-01-01 15:19:43.963455] D [MSGID: 0] > [store.c:473:gf_store_handle_destroy] 0-: Returning 0 > [1970-01-01 15:19:43.963757] D [MSGID: 0] > [glusterd-store.c:3546:glusterd_store_retrieve_volumes] 0-management: > Returning with 0 > [1970-01-01 15:19:43.964159] D [MSGID: 0] > [glusterd-store.c:4662:glusterd_store_retrieve_peers] 0-management: > Returning with 0 > [1970-01-01 15:19:43.964471] I [MSGID: 106194] > [glusterd-store.c:3983:glusterd_store_retrieve_missed_snaps_list] > 0-management: No missed snaps list. > [1970-01-01 15:19:43.964580] D [MSGID: 0] > [glusterd-store.c:4104:glusterd_store_retrieve_snaps] 0-management: > Returning with 0 > [1970-01-01 15:19:43.964680] D [MSGID: 0] > [glusterd-store.c:4894:glusterd_restore] 0-management: Returning 0 > [1970-01-01 15:19:43.965060] D [MSGID: 0] > [options.c:1225:xlator_option_init_int32] 0-management: option > event-threads using set value 1 > Final graph: > > +------------------------------------------------------------------------------+ > 1: volume management > 2: type mgmt/glusterd > 3: option rpc-auth.auth-glusterfs on > 4: option rpc-auth.auth-unix on > 5: option rpc-auth.auth-null on > 6: option rpc-auth-allow-insecure on > 7: option transport.listen-backlog 10 > 8: option event-threads 1 > 9: option ping-timeout 0 > 10: option transport.socket.read-fail-log off > 11: option transport.socket.keepalive-interval 2 > 12: option transport.socket.keepalive-time 10 > 13: option transport-type rdma > 14: option working-directory /var/lib/glusterd > 15: end-volume > 16: > > +------------------------------------------------------------------------------+ > [1970-01-01 15:19:43.966808] I [MSGID: 101190] > [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 1 > [1970-01-01 15:19:44.840454] E [rpcsvc.c:513:rpcsvc_request_create] > 0-rpc-service: RPC version not supported (XID: 0x0, Ver: 0, Program: 0, > ProgVers: 0, Proc: 2) from trans (socket.management) > [1970-01-01 15:19:44.840884] D [rpcsvc.c:1416:rpcsvc_error_reply] (--> > /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xffbac)[0x3fffa12acbe4] > (--> /usr/lib64/libgfrpc.so.0(+0xc5f4)[0x3fffa12525f4] (--> > /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00] (--> > /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224] (--> > /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c] ))))) 0-: sending a RPC > error reply > [1970-01-01 15:19:44.841055] D [logging.c:1805:gf_log_flush_extra_msgs] > 0-logging-infra: Log buffer size reduced. About to flush 5 extra log > messages > [1970-01-01 15:19:44.841156] D [logging.c:1808:gf_log_flush_extra_msgs] > 0-logging-infra: Just flushed 5 extra log messages > pending frames: > patchset: git://git.gluster.org/glusterfs.git > signal received: 11 > time of crash: > 1970-01-01 15:19:44 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string: glusterfs 5.0 > /usr/lib64/libglusterfs.so.0(+0x422a4)[0x3fffa12ab2a4] > /usr/lib64/libglusterfs.so.0(gf_print_trace-0xf5080)[0x3fffa12b82e0] > ./glusterd(glusterfsd_print_trace-0x22fa4)[0x100067ec] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fffa13f0478] > /lib64/libc.so.6(xdr_accepted_reply-0x72d3c)[0x3fffa11375cc] > /lib64/libc.so.6(xdr_accepted_reply-0x72d9c)[0x3fffa113756c] > /lib64/libc.so.6(xdr_union-0x63a94)[0x3fffa1147dd4] > /lib64/libc.so.6(xdr_replymsg-0x72c58)[0x3fffa11376e0] > /lib64/libc.so.6(xdr_sizeof-0x62a78)[0x3fffa1149120] > /usr/lib64/libgfrpc.so.0(+0x9b0c)[0x3fffa124fb0c] > /usr/lib64/libgfrpc.so.0(rpcsvc_submit_generic-0x149f4)[0x3fffa125228c] > /usr/lib64/libgfrpc.so.0(+0xc614)[0x3fffa1252614] > /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00] > /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224] > /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c] > /usr/lib64/libgfrpc.so.0(rpc_transport_notify-0x10eec)[0x3fffa125610c] > /usr/lib64/glusterfs/5.0/rpc-transport/socket.so(+0xc09c)[0x3fff9d51709c] > /usr/lib64/libglusterfs.so.0(+0xb84bc)[0x3fffa13214bc] > /lib64/libpthread.so.0(+0xbb30)[0x3fffa11bdb30] > /lib64/libc.so.6(clone-0x9e964)[0x3fffa110817c] > --------- > Segmentation fault (core dumped) > > Could you please help me, what actually the problem? > > > -- > > > > > Regards > Abhishek Paliwal > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -- Thanks, Sanju -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhishpaliwal at gmail.com Wed Mar 6 08:33:09 2019 From: abhishpaliwal at gmail.com (ABHISHEK PALIWAL) Date: Wed, 6 Mar 2019 14:03:09 +0530 Subject: [Gluster-devel] [Gluster-users] Not able to start glusterd In-Reply-To: References: Message-ID: Hi Sanju, Thanks for the response. I have resolved the issue, actually I have updated from 3.7.6 to 5.0, in new version RPC is coming from libtirpb , but I forgot to enable "--with-libtirpc" in configuration. After enabling able to start glusterd. Regards, Abhishek On Wed, Mar 6, 2019 at 12:58 PM Sanju Rakonde wrote: > Abhishek, > > We need below information on investigate this issue. > 1. gluster --version > 2. Please run glusterd in gdb, so that we can capture the backtrace. I see > some rpc errors in log, but backtrace will be more helpful. > To run glusterd in gdb, you need start glusterd in gdb (i.e. gdb > glusterd, and then give the command "run -N"). when you see a segmentation > fault, please capture the backtrace and paste it here. > > On Wed, Mar 6, 2019 at 10:07 AM ABHISHEK PALIWAL > wrote: > >> Hi Team, >> >> I am facing the issue where at the time of starting the glusterd >> segmentation fault is reported. >> >> Below are the logs >> >> root at 128:/usr/sbin# ./glusterd --debug >> [1970-01-01 15:19:43.940386] I [MSGID: 100030] [glusterfsd.c:2691:main] >> 0-./glusterd: Started running ./glusterd version 5.0 (args: ./glusterd >> --debug) >> [1970-01-01 15:19:43.940855] D >> [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting >> timer now. Timeout = 120, current buf size = 5 >> [1970-01-01 15:19:43.941736] D [MSGID: 0] [glusterfsd.c:747:get_volfp] >> 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol >> [1970-01-01 15:19:43.945796] D [MSGID: 101097] >> [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on >> /usr/lib64/glusterfs/5.0/xlator/mgmt/glusterd.so: undefined symbol: >> xlator_api. Fall back to old symbols >> [1970-01-01 15:19:43.946279] I [MSGID: 106478] [glusterd.c:1435:init] >> 0-management: Maximum allowed open file descriptors set to 65536 >> [1970-01-01 15:19:43.946419] I [MSGID: 106479] [glusterd.c:1491:init] >> 0-management: Using /var/lib/glusterd as working directory >> [1970-01-01 15:19:43.946515] I [MSGID: 106479] [glusterd.c:1497:init] >> 0-management: Using /var/run/gluster as pid file working directory >> [1970-01-01 15:19:43.946968] D [MSGID: 0] >> [glusterd.c:458:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog >> value: 10 >> [1970-01-01 15:19:43.947139] D [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: >> RPC service inited. >> [1970-01-01 15:19:43.947241] D [rpcsvc.c:2146:rpcsvc_program_register] >> 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, >> Port: 0 >> [1970-01-01 15:19:43.947379] D [rpc-transport.c:269:rpc_transport_load] >> 0-rpc-transport: attempt to load file >> /usr/lib64/glusterfs/5.0/rpc-transport/socket.so >> [1970-01-01 15:19:43.955198] D [socket.c:4464:socket_init] >> 0-socket.management: Configued transport.tcp-user-timeout=0 >> [1970-01-01 15:19:43.955316] D [socket.c:4482:socket_init] >> 0-socket.management: Reconfigued transport.keepalivecnt=9 >> [1970-01-01 15:19:43.955415] D >> [socket.c:4167:ssl_setup_connection_params] 0-socket.management: SSL >> support on the I/O path is NOT enabled >> [1970-01-01 15:19:43.955504] D >> [socket.c:4170:ssl_setup_connection_params] 0-socket.management: SSL >> support for glusterd is NOT enabled >> [1970-01-01 15:19:43.955612] D [name.c:572:server_fill_address_family] >> 0-socket.management: option address-family not specified, defaulting to >> inet6 >> [1970-01-01 15:19:43.955928] D [rpc-transport.c:269:rpc_transport_load] >> 0-rpc-transport: attempt to load file >> /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so >> [1970-01-01 15:19:43.956079] E [rpc-transport.c:273:rpc_transport_load] >> 0-rpc-transport: /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so: cannot >> open shared object file: No such file or directory >> [1970-01-01 15:19:43.956177] W [rpc-transport.c:277:rpc_transport_load] >> 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not >> valid or not found on this machine >> [1970-01-01 15:19:43.956270] W [rpcsvc.c:1789:rpcsvc_create_listener] >> 0-rpc-service: cannot create listener, initing the transport failed >> [1970-01-01 15:19:43.956362] E [MSGID: 106244] [glusterd.c:1798:init] >> 0-management: creation of 1 listeners failed, continuing with succeeded >> transport >> [1970-01-01 15:19:43.956459] D [rpcsvc.c:2146:rpcsvc_program_register] >> 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, >> Ver: 2, Port: 0 >> [1970-01-01 15:19:43.956561] D [rpcsvc.c:2146:rpcsvc_program_register] >> 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: >> 1238463, Ver: 2, Port: 0 >> [1970-01-01 15:19:43.956666] D [rpcsvc.c:2146:rpcsvc_program_register] >> 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, >> Ver: 2, Port: 0 >> [1970-01-01 15:19:43.956758] D [rpcsvc.c:2146:rpcsvc_program_register] >> 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, >> Ver: 3, Port: 0 >> [1970-01-01 15:19:43.956853] D [rpcsvc.c:2146:rpcsvc_program_register] >> 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: >> 1, Port: 0 >> [1970-01-01 15:19:43.956946] D [rpcsvc.c:2146:rpcsvc_program_register] >> 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, >> Ver: 2, Port: 0 >> [1970-01-01 15:19:43.957062] D [rpcsvc.c:2146:rpcsvc_program_register] >> 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: >> 1239873, Ver: 1, Port: 0 >> [1970-01-01 15:19:43.957205] D [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: >> RPC service inited. >> [1970-01-01 15:19:43.957303] D [rpcsvc.c:2146:rpcsvc_program_register] >> 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, >> Port: 0 >> [1970-01-01 15:19:43.957408] D [rpc-transport.c:269:rpc_transport_load] >> 0-rpc-transport: attempt to load file >> /usr/lib64/glusterfs/5.0/rpc-transport/socket.so >> [1970-01-01 15:19:43.957563] D [socket.c:4424:socket_init] >> 0-socket.management: disabling nodelay >> [1970-01-01 15:19:43.957650] D [socket.c:4464:socket_init] >> 0-socket.management: Configued transport.tcp-user-timeout=0 >> [1970-01-01 15:19:43.957738] D [socket.c:4482:socket_init] >> 0-socket.management: Reconfigued transport.keepalivecnt=9 >> [1970-01-01 15:19:43.957830] D >> [socket.c:4167:ssl_setup_connection_params] 0-socket.management: SSL >> support on the I/O path is NOT enabled >> [1970-01-01 15:19:43.957922] D >> [socket.c:4170:ssl_setup_connection_params] 0-socket.management: SSL >> support for glusterd is NOT enabled >> [1970-01-01 15:19:43.958186] D [rpcsvc.c:2146:rpcsvc_program_register] >> 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: >> 2, Port: 0 >> [1970-01-01 15:19:43.958280] D [rpcsvc.c:2146:rpcsvc_program_register] >> 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), >> Num: 14398633, Ver: 2, Port: 0 >> [1970-01-01 15:19:43.958461] D [MSGID: 0] >> [glusterd-utils.c:7878:glusterd_sm_tr_log_init] 0-glusterd: returning 0 >> [1970-01-01 15:19:43.958557] D [MSGID: 0] [glusterd.c:1875:init] >> 0-management: cannot get run-with-valgrind value >> [1970-01-01 15:19:43.960895] E [MSGID: 101032] >> [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding to >> /var/lib/glusterd/glusterd.info. [No such file or directory] >> [1970-01-01 15:19:43.961016] D [MSGID: 0] >> [store.c:452:gf_store_handle_retrieve] 0-: Returning -1 >> [1970-01-01 15:19:43.961108] D [MSGID: 0] >> [glusterd-store.c:2169:glusterd_retrieve_op_version] 0-management: Unable >> to get store handle! >> [1970-01-01 15:19:43.961216] E [MSGID: 101032] >> [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding to >> /var/lib/glusterd/glusterd.info. [No such file or directory] >> [1970-01-01 15:19:43.961325] D [MSGID: 0] >> [store.c:452:gf_store_handle_retrieve] 0-: Returning -1 >> [1970-01-01 15:19:43.961428] D [MSGID: 0] >> [glusterd-store.c:2345:glusterd_retrieve_uuid] 0-management: Unable to get >> storehandle! >> [1970-01-01 15:19:43.961523] D [MSGID: 0] >> [glusterd-store.c:2366:glusterd_retrieve_uuid] 0-management: Returning -1 >> [1970-01-01 15:19:43.961617] I [MSGID: 106514] >> [glusterd-store.c:2304:glusterd_restore_op_version] 0-management: Detected >> new install. Setting op-version to maximum : 50000 >> [1970-01-01 15:19:43.962658] D [MSGID: 0] >> [store.c:432:gf_store_handle_new] 0-: Returning 0 >> [1970-01-01 15:19:43.962750] D [MSGID: 0] >> [store.c:452:gf_store_handle_retrieve] 0-: Returning 0 >> [1970-01-01 15:19:43.963047] D [MSGID: 0] [store.c:515:gf_store_iter_new] >> 0-: Returning with 0 >> [1970-01-01 15:19:43.963194] D [MSGID: 0] >> [store.c:632:gf_store_iter_get_next] 0-: Returning with 0 >> [1970-01-01 15:19:43.963318] D [MSGID: 0] >> [store.c:632:gf_store_iter_get_next] 0-: Returning with -1 >> [1970-01-01 15:19:43.963455] D [MSGID: 0] >> [store.c:473:gf_store_handle_destroy] 0-: Returning 0 >> [1970-01-01 15:19:43.963757] D [MSGID: 0] >> [glusterd-store.c:3546:glusterd_store_retrieve_volumes] 0-management: >> Returning with 0 >> [1970-01-01 15:19:43.964159] D [MSGID: 0] >> [glusterd-store.c:4662:glusterd_store_retrieve_peers] 0-management: >> Returning with 0 >> [1970-01-01 15:19:43.964471] I [MSGID: 106194] >> [glusterd-store.c:3983:glusterd_store_retrieve_missed_snaps_list] >> 0-management: No missed snaps list. >> [1970-01-01 15:19:43.964580] D [MSGID: 0] >> [glusterd-store.c:4104:glusterd_store_retrieve_snaps] 0-management: >> Returning with 0 >> [1970-01-01 15:19:43.964680] D [MSGID: 0] >> [glusterd-store.c:4894:glusterd_restore] 0-management: Returning 0 >> [1970-01-01 15:19:43.965060] D [MSGID: 0] >> [options.c:1225:xlator_option_init_int32] 0-management: option >> event-threads using set value 1 >> Final graph: >> >> +------------------------------------------------------------------------------+ >> 1: volume management >> 2: type mgmt/glusterd >> 3: option rpc-auth.auth-glusterfs on >> 4: option rpc-auth.auth-unix on >> 5: option rpc-auth.auth-null on >> 6: option rpc-auth-allow-insecure on >> 7: option transport.listen-backlog 10 >> 8: option event-threads 1 >> 9: option ping-timeout 0 >> 10: option transport.socket.read-fail-log off >> 11: option transport.socket.keepalive-interval 2 >> 12: option transport.socket.keepalive-time 10 >> 13: option transport-type rdma >> 14: option working-directory /var/lib/glusterd >> 15: end-volume >> 16: >> >> +------------------------------------------------------------------------------+ >> [1970-01-01 15:19:43.966808] I [MSGID: 101190] >> [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread >> with index 1 >> [1970-01-01 15:19:44.840454] E [rpcsvc.c:513:rpcsvc_request_create] >> 0-rpc-service: RPC version not supported (XID: 0x0, Ver: 0, Program: 0, >> ProgVers: 0, Proc: 2) from trans (socket.management) >> [1970-01-01 15:19:44.840884] D [rpcsvc.c:1416:rpcsvc_error_reply] (--> >> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xffbac)[0x3fffa12acbe4] >> (--> /usr/lib64/libgfrpc.so.0(+0xc5f4)[0x3fffa12525f4] (--> >> /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00] (--> >> /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224] (--> >> /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c] ))))) 0-: sending a RPC >> error reply >> [1970-01-01 15:19:44.841055] D [logging.c:1805:gf_log_flush_extra_msgs] >> 0-logging-infra: Log buffer size reduced. About to flush 5 extra log >> messages >> [1970-01-01 15:19:44.841156] D [logging.c:1808:gf_log_flush_extra_msgs] >> 0-logging-infra: Just flushed 5 extra log messages >> pending frames: >> patchset: git://git.gluster.org/glusterfs.git >> signal received: 11 >> time of crash: >> 1970-01-01 15:19:44 >> configuration details: >> argp 1 >> backtrace 1 >> dlfcn 1 >> libpthread 1 >> llistxattr 1 >> setfsid 1 >> spinlock 1 >> epoll.h 1 >> xattr.h 1 >> st_atim.tv_nsec 1 >> package-string: glusterfs 5.0 >> /usr/lib64/libglusterfs.so.0(+0x422a4)[0x3fffa12ab2a4] >> /usr/lib64/libglusterfs.so.0(gf_print_trace-0xf5080)[0x3fffa12b82e0] >> ./glusterd(glusterfsd_print_trace-0x22fa4)[0x100067ec] >> linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fffa13f0478] >> /lib64/libc.so.6(xdr_accepted_reply-0x72d3c)[0x3fffa11375cc] >> /lib64/libc.so.6(xdr_accepted_reply-0x72d9c)[0x3fffa113756c] >> /lib64/libc.so.6(xdr_union-0x63a94)[0x3fffa1147dd4] >> /lib64/libc.so.6(xdr_replymsg-0x72c58)[0x3fffa11376e0] >> /lib64/libc.so.6(xdr_sizeof-0x62a78)[0x3fffa1149120] >> /usr/lib64/libgfrpc.so.0(+0x9b0c)[0x3fffa124fb0c] >> /usr/lib64/libgfrpc.so.0(rpcsvc_submit_generic-0x149f4)[0x3fffa125228c] >> /usr/lib64/libgfrpc.so.0(+0xc614)[0x3fffa1252614] >> /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00] >> /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224] >> /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c] >> /usr/lib64/libgfrpc.so.0(rpc_transport_notify-0x10eec)[0x3fffa125610c] >> /usr/lib64/glusterfs/5.0/rpc-transport/socket.so(+0xc09c)[0x3fff9d51709c] >> /usr/lib64/libglusterfs.so.0(+0xb84bc)[0x3fffa13214bc] >> /lib64/libpthread.so.0(+0xbb30)[0x3fffa11bdb30] >> /lib64/libc.so.6(clone-0x9e964)[0x3fffa110817c] >> --------- >> Segmentation fault (core dumped) >> >> Could you please help me, what actually the problem? >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > Thanks, > Sanju > -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkavunga at redhat.com Wed Mar 6 08:57:44 2019 From: rkavunga at redhat.com (RAFI KC) Date: Wed, 6 Mar 2019 14:27:44 +0530 Subject: [Gluster-devel] [Gluster-users] Not able to start glusterd In-Reply-To: References: Message-ID: Hi Abhishek, Good to know that you have resolved your problem. Do you think any more information is required to add in the upgrade doc for a smooth upgrade flow. It would be great to see a PR to the repo https://github.com/gluster/glusterdocs/tree/master/docs/Upgrade-Guide for updating the doc with the information. Regards Rafi KC On 3/6/19 2:03 PM, ABHISHEK PALIWAL wrote: > Hi Sanju, > > Thanks for the response. > > I have resolved the issue, actually I have updated from 3.7.6 to 5.0, > in new version RPC is coming from libtirpb , but I forgot to enable > "--with-libtirpc" in configuration. > > After enabling able to start glusterd. > > Regards, > Abhishek > > On Wed, Mar 6, 2019 at 12:58 PM Sanju Rakonde > wrote: > > Abhishek, > > We need below information on investigate this issue. > 1. gluster --version > 2. Please run glusterd in gdb, so that we can capture the > backtrace. I see some rpc errors in log, but backtrace will be > more helpful. > ? ? To run glusterd in gdb, you need start glusterd in gdb (i.e. > gdb glusterd, and then give the command "run -N"). when you see a > segmentation? ? ? ? ? ?fault, please capture the backtrace and > paste it here. > > On Wed, Mar 6, 2019 at 10:07 AM ABHISHEK PALIWAL > > wrote: > > Hi Team, > > I am facing the issue where at the time of starting the > glusterd segmentation fault is reported. > > Below are the logs > > root at 128:/usr/sbin# ./glusterd? --debug > [1970-01-01 15:19:43.940386] I [MSGID: 100030] > [glusterfsd.c:2691:main] 0-./glusterd: Started running > ./glusterd version 5.0 (args: ./glusterd --debug) > [1970-01-01 15:19:43.940855] D > [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: > Starting timer now. Timeout = 120, current buf size = 5 > [1970-01-01 15:19:43.941736] D [MSGID: 0] > [glusterfsd.c:747:get_volfp] 0-glusterfsd: loading volume file > /etc/glusterfs/glusterd.vol > [1970-01-01 15:19:43.945796] D [MSGID: 101097] > [xlator.c:341:xlator_dynload_newway] 0-xlator: > dlsym(xlator_api) on > /usr/lib64/glusterfs/5.0/xlator/mgmt/glusterd.so: undefined > symbol: xlator_api. Fall back to old symbols > [1970-01-01 15:19:43.946279] I [MSGID: 106478] > [glusterd.c:1435:init] 0-management: Maximum allowed open file > descriptors set to 65536 > [1970-01-01 15:19:43.946419] I [MSGID: 106479] > [glusterd.c:1491:init] 0-management: Using /var/lib/glusterd > as working directory > [1970-01-01 15:19:43.946515] I [MSGID: 106479] > [glusterd.c:1497:init] 0-management: Using /var/run/gluster as > pid file working directory > [1970-01-01 15:19:43.946968] D [MSGID: 0] > [glusterd.c:458:glusterd_rpcsvc_options_build] 0-glusterd: > listen-backlog value: 10 > [1970-01-01 15:19:43.947139] D [rpcsvc.c:2607:rpcsvc_init] > 0-rpc-service: RPC service inited. > [1970-01-01 15:19:43.947241] D > [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New > program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 > [1970-01-01 15:19:43.947379] D > [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: > attempt to load file > /usr/lib64/glusterfs/5.0/rpc-transport/socket.so > [1970-01-01 15:19:43.955198] D [socket.c:4464:socket_init] > 0-socket.management: Configued transport.tcp-user-timeout=0 > [1970-01-01 15:19:43.955316] D [socket.c:4482:socket_init] > 0-socket.management: Reconfigued transport.keepalivecnt=9 > [1970-01-01 15:19:43.955415] D > [socket.c:4167:ssl_setup_connection_params] > 0-socket.management: SSL support on the I/O path is NOT enabled > [1970-01-01 15:19:43.955504] D > [socket.c:4170:ssl_setup_connection_params] > 0-socket.management: SSL support for glusterd is NOT enabled > [1970-01-01 15:19:43.955612] D > [name.c:572:server_fill_address_family] 0-socket.management: > option address-family not specified, defaulting to inet6 > [1970-01-01 15:19:43.955928] D > [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: > attempt to load file > /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so > [1970-01-01 15:19:43.956079] E > [rpc-transport.c:273:rpc_transport_load] 0-rpc-transport: > /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so: cannot open > shared object file: No such file or directory > [1970-01-01 15:19:43.956177] W > [rpc-transport.c:277:rpc_transport_load] 0-rpc-transport: > volume 'rdma.management': transport-type 'rdma' is not valid > or not found on this machine > [1970-01-01 15:19:43.956270] W > [rpcsvc.c:1789:rpcsvc_create_listener] 0-rpc-service: cannot > create listener, initing the transport failed > [1970-01-01 15:19:43.956362] E [MSGID: 106244] > [glusterd.c:1798:init] 0-management: creation of 1 listeners > failed, continuing with succeeded transport > [1970-01-01 15:19:43.956459] D > [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New > program registered: GlusterD svc peer, Num: 1238437, Ver: 2, > Port: 0 > [1970-01-01 15:19:43.956561] D > [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New > program registered: GlusterD svc cli read-only, Num: 1238463, > Ver: 2, Port: 0 > [1970-01-01 15:19:43.956666] D > [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New > program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, > Port: 0 > [1970-01-01 15:19:43.956758] D > [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New > program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: > 3, Port: 0 > [1970-01-01 15:19:43.956853] D > [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New > program registered: Gluster Portmap, Num: 34123456, Ver: 1, > Port: 0 > [1970-01-01 15:19:43.956946] D > [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New > program registered: Gluster Handshake, Num: 14398633, Ver: 2, > Port: 0 > [1970-01-01 15:19:43.957062] D > [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New > program registered: Gluster MGMT Handshake, Num: 1239873, Ver: > 1, Port: 0 > [1970-01-01 15:19:43.957205] D [rpcsvc.c:2607:rpcsvc_init] > 0-rpc-service: RPC service inited. > [1970-01-01 15:19:43.957303] D > [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New > program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 > [1970-01-01 15:19:43.957408] D > [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: > attempt to load file > /usr/lib64/glusterfs/5.0/rpc-transport/socket.so > [1970-01-01 15:19:43.957563] D [socket.c:4424:socket_init] > 0-socket.management: disabling nodelay > [1970-01-01 15:19:43.957650] D [socket.c:4464:socket_init] > 0-socket.management: Configued transport.tcp-user-timeout=0 > [1970-01-01 15:19:43.957738] D [socket.c:4482:socket_init] > 0-socket.management: Reconfigued transport.keepalivecnt=9 > [1970-01-01 15:19:43.957830] D > [socket.c:4167:ssl_setup_connection_params] > 0-socket.management: SSL support on the I/O path is NOT enabled > [1970-01-01 15:19:43.957922] D > [socket.c:4170:ssl_setup_connection_params] > 0-socket.management: SSL support for glusterd is NOT enabled > [1970-01-01 15:19:43.958186] D > [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New > program registered: GlusterD svc cli, Num: 1238463, Ver: 2, > Port: 0 > [1970-01-01 15:19:43.958280] D > [rpcsvc.c:2146:rpcsvc_program_register] 0-rpc-service: New > program registered: Gluster Handshake (CLI Getspec), Num: > 14398633, Ver: 2, Port: 0 > [1970-01-01 15:19:43.958461] D [MSGID: 0] > [glusterd-utils.c:7878:glusterd_sm_tr_log_init] 0-glusterd: > returning 0 > [1970-01-01 15:19:43.958557] D [MSGID: 0] > [glusterd.c:1875:init] 0-management: cannot get > run-with-valgrind value > [1970-01-01 15:19:43.960895] E [MSGID: 101032] > [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding > to /var/lib/glusterd/glusterd.info . [No > such file or directory] > [1970-01-01 15:19:43.961016] D [MSGID: 0] > [store.c:452:gf_store_handle_retrieve] 0-: Returning -1 > [1970-01-01 15:19:43.961108] D [MSGID: 0] > [glusterd-store.c:2169:glusterd_retrieve_op_version] > 0-management: Unable to get store handle! > [1970-01-01 15:19:43.961216] E [MSGID: 101032] > [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding > to /var/lib/glusterd/glusterd.info . [No > such file or directory] > [1970-01-01 15:19:43.961325] D [MSGID: 0] > [store.c:452:gf_store_handle_retrieve] 0-: Returning -1 > [1970-01-01 15:19:43.961428] D [MSGID: 0] > [glusterd-store.c:2345:glusterd_retrieve_uuid] 0-management: > Unable to get storehandle! > [1970-01-01 15:19:43.961523] D [MSGID: 0] > [glusterd-store.c:2366:glusterd_retrieve_uuid] 0-management: > Returning -1 > [1970-01-01 15:19:43.961617] I [MSGID: 106514] > [glusterd-store.c:2304:glusterd_restore_op_version] > 0-management: Detected new install. Setting op-version to > maximum : 50000 > [1970-01-01 15:19:43.962658] D [MSGID: 0] > [store.c:432:gf_store_handle_new] 0-: Returning 0 > [1970-01-01 15:19:43.962750] D [MSGID: 0] > [store.c:452:gf_store_handle_retrieve] 0-: Returning 0 > [1970-01-01 15:19:43.963047] D [MSGID: 0] > [store.c:515:gf_store_iter_new] 0-: Returning with 0 > [1970-01-01 15:19:43.963194] D [MSGID: 0] > [store.c:632:gf_store_iter_get_next] 0-: Returning with 0 > [1970-01-01 15:19:43.963318] D [MSGID: 0] > [store.c:632:gf_store_iter_get_next] 0-: Returning with -1 > [1970-01-01 15:19:43.963455] D [MSGID: 0] > [store.c:473:gf_store_handle_destroy] 0-: Returning 0 > [1970-01-01 15:19:43.963757] D [MSGID: 0] > [glusterd-store.c:3546:glusterd_store_retrieve_volumes] > 0-management: Returning with 0 > [1970-01-01 15:19:43.964159] D [MSGID: 0] > [glusterd-store.c:4662:glusterd_store_retrieve_peers] > 0-management: Returning with 0 > [1970-01-01 15:19:43.964471] I [MSGID: 106194] > [glusterd-store.c:3983:glusterd_store_retrieve_missed_snaps_list] > 0-management: No missed snaps list. > [1970-01-01 15:19:43.964580] D [MSGID: 0] > [glusterd-store.c:4104:glusterd_store_retrieve_snaps] > 0-management: Returning with 0 > [1970-01-01 15:19:43.964680] D [MSGID: 0] > [glusterd-store.c:4894:glusterd_restore] 0-management: Returning 0 > [1970-01-01 15:19:43.965060] D [MSGID: 0] > [options.c:1225:xlator_option_init_int32] 0-management: option > event-threads using set value 1 > Final graph: > +------------------------------------------------------------------------------+ > ? 1: volume management > ? 2:? ? ?type mgmt/glusterd > ? 3:? ? ?option rpc-auth.auth-glusterfs on > ? 4:? ? ?option rpc-auth.auth-unix on > ? 5:? ? ?option rpc-auth.auth-null on > ? 6:? ? ?option rpc-auth-allow-insecure on > ? 7:? ? ?option transport.listen-backlog 10 > ? 8:? ? ?option event-threads 1 > ? 9:? ? ?option ping-timeout 0 > ?10:? ? ?option transport.socket.read-fail-log off > ?11:? ? ?option transport.socket.keepalive-interval 2 > ?12:? ? ?option transport.socket.keepalive-time 10 > ?13:? ? ?option transport-type rdma > ?14:? ? ?option working-directory /var/lib/glusterd > ?15: end-volume > ?16: > +------------------------------------------------------------------------------+ > [1970-01-01 15:19:43.966808] I [MSGID: 101190] > [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: > Started thread with index 1 > [1970-01-01 15:19:44.840454] E > [rpcsvc.c:513:rpcsvc_request_create] 0-rpc-service: RPC > version not supported (XID: 0x0, Ver: 0, Program: 0, ProgVers: > 0, Proc: 2) from trans (socket.management) > [1970-01-01 15:19:44.840884] D > [rpcsvc.c:1416:rpcsvc_error_reply] (--> > /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xffbac)[0x3fffa12acbe4] > (--> /usr/lib64/libgfrpc.so.0(+0xc5f4)[0x3fffa12525f4] (--> > /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00] (--> > /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224] (--> > /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c] ))))) 0-: > sending a RPC error reply > [1970-01-01 15:19:44.841055] D > [logging.c:1805:gf_log_flush_extra_msgs] 0-logging-infra: Log > buffer size reduced. About to flush 5 extra log messages > [1970-01-01 15:19:44.841156] D > [logging.c:1808:gf_log_flush_extra_msgs] 0-logging-infra: Just > flushed 5 extra log messages > pending frames: > patchset: git://git.gluster.org/glusterfs.git > > signal received: 11 > time of crash: > 1970-01-01 15:19:44 > configuration details: > argp 1 > backtrace 1 > dlfcn 1 > libpthread 1 > llistxattr 1 > setfsid 1 > spinlock 1 > epoll.h 1 > xattr.h 1 > st_atim.tv_nsec 1 > package-string: glusterfs 5.0 > /usr/lib64/libglusterfs.so.0(+0x422a4)[0x3fffa12ab2a4] > /usr/lib64/libglusterfs.so.0(gf_print_trace-0xf5080)[0x3fffa12b82e0] > ./glusterd(glusterfsd_print_trace-0x22fa4)[0x100067ec] > linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fffa13f0478] > /lib64/libc.so.6(xdr_accepted_reply-0x72d3c)[0x3fffa11375cc] > /lib64/libc.so.6(xdr_accepted_reply-0x72d9c)[0x3fffa113756c] > /lib64/libc.so.6(xdr_union-0x63a94)[0x3fffa1147dd4] > /lib64/libc.so.6(xdr_replymsg-0x72c58)[0x3fffa11376e0] > /lib64/libc.so.6(xdr_sizeof-0x62a78)[0x3fffa1149120] > /usr/lib64/libgfrpc.so.0(+0x9b0c)[0x3fffa124fb0c] > /usr/lib64/libgfrpc.so.0(rpcsvc_submit_generic-0x149f4)[0x3fffa125228c] > /usr/lib64/libgfrpc.so.0(+0xc614)[0x3fffa1252614] > /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00] > /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224] > /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c] > /usr/lib64/libgfrpc.so.0(rpc_transport_notify-0x10eec)[0x3fffa125610c] > /usr/lib64/glusterfs/5.0/rpc-transport/socket.so(+0xc09c)[0x3fff9d51709c] > /usr/lib64/libglusterfs.so.0(+0xb84bc)[0x3fffa13214bc] > /lib64/libpthread.so.0(+0xbb30)[0x3fffa11bdb30] > /lib64/libc.so.6(clone-0x9e964)[0x3fffa110817c] > --------- > Segmentation fault (core dumped) > > Could you? please help me, what actually the problem? > > > -- > > > > > Regards > Abhishek Paliwal > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > Thanks, > Sanju > > > > -- > > > > > Regards > Abhishek Paliwal > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From dkhandel at redhat.com Wed Mar 6 12:07:53 2019 From: dkhandel at redhat.com (Deepshikha Khandelwal) Date: Wed, 6 Mar 2019 17:37:53 +0530 Subject: [Gluster-devel] [Gluster-infra] 8/10 AWS jenkins builders disconnected Message-ID: Hello, Today while debugging the centos7-regression failed builds I saw most of the builders did not pass the instance status check on AWS and were unreachable. Misc investigated this and came to know about the patch[1] which seems to break the builder one after the other. They all ran the regression test for this specific change before going offline. We suspect that this change do result in infinite loop of processes as we did not see any trace of error in the system logs. We did reboot all those builders and they all seem to be running fine now. Please let us know if you see any such issues again. [1] https://review.gluster.org/#/c/glusterfs/+/22290/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Wed Mar 6 12:23:20 2019 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Wed, 6 Mar 2019 17:53:20 +0530 Subject: [Gluster-devel] [Gluster-infra] 8/10 AWS jenkins builders disconnected In-Reply-To: References: Message-ID: On Wed, Mar 6, 2019 at 5:38 PM Deepshikha Khandelwal wrote: > > Hello, > > Today while debugging the centos7-regression failed builds I saw most of the builders did not pass the instance status check on AWS and were unreachable. > > Misc investigated this and came to know about the patch[1] which seems to break the builder one after the other. They all ran the regression test for this specific change before going offline. > We suspect that this change do result in infinite loop of processes as we did not see any trace of error in the system logs. > > We did reboot all those builders and they all seem to be running fine now. > The question though is - what to do about the patch, if the patch itself is the root cause? Is this assigned to anyone to look into? > Please let us know if you see any such issues again. > > [1] https://review.gluster.org/#/c/glusterfs/+/22290/ -- sankarshan mukhopadhyay From dkhandel at redhat.com Wed Mar 6 12:31:39 2019 From: dkhandel at redhat.com (Deepshikha Khandelwal) Date: Wed, 6 Mar 2019 18:01:39 +0530 Subject: [Gluster-devel] [Gluster-infra] 8/10 AWS jenkins builders disconnected In-Reply-To: References: Message-ID: Yes, Mohit is looking into it. There's some issue in the patch itself. I forgot to link the bug filed for this: https://bugzilla.redhat.com/show_bug.cgi?id=1685813 On Wed, Mar 6, 2019 at 5:54 PM Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On Wed, Mar 6, 2019 at 5:38 PM Deepshikha Khandelwal > wrote: > > > > Hello, > > > > Today while debugging the centos7-regression failed builds I saw most of > the builders did not pass the instance status check on AWS and were > unreachable. > > > > Misc investigated this and came to know about the patch[1] which seems > to break the builder one after the other. They all ran the regression test > for this specific change before going offline. > > We suspect that this change do result in infinite loop of processes as > we did not see any trace of error in the system logs. > > > > We did reboot all those builders and they all seem to be running fine > now. > > > > The question though is - what to do about the patch, if the patch > itself is the root cause? Is this assigned to anyone to look into? > > > Please let us know if you see any such issues again. > > > > [1] https://review.gluster.org/#/c/glusterfs/+/22290/ > > > -- > sankarshan mukhopadhyay > > _______________________________________________ > Gluster-infra mailing list > Gluster-infra at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-infra > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mscherer at redhat.com Wed Mar 6 15:17:30 2019 From: mscherer at redhat.com (Michael Scherer) Date: Wed, 06 Mar 2019 16:17:30 +0100 Subject: [Gluster-devel] [Gluster-infra] 8/10 AWS jenkins builders disconnected In-Reply-To: References: Message-ID: Le mercredi 06 mars 2019 ? 17:53 +0530, Sankarshan Mukhopadhyay a ?crit : > On Wed, Mar 6, 2019 at 5:38 PM Deepshikha Khandelwal > wrote: > > > > Hello, > > > > Today while debugging the centos7-regression failed builds I saw > > most of the builders did not pass the instance status check on AWS > > and were unreachable. > > > > Misc investigated this and came to know about the patch[1] which > > seems to break the builder one after the other. They all ran the > > regression test for this specific change before going offline. > > We suspect that this change do result in infinite loop of processes > > as we did not see any trace of error in the system logs. > > > > We did reboot all those builders and they all seem to be running > > fine now. > > > > The question though is - what to do about the patch, if the patch > itself is the root cause? Is this assigned to anyone to look into? We also pondered on wether we should protect the builder from that kind of issue. But since: - we are not sure that the hypothesis is right - any protection based on "limit the number of process" would surely sooner or later block legitimate tests, and requires adjustement (and likely investigation) we didn't choose to follow that road for now. > > Please let us know if you see any such issues again. > > > > [1] https://review.gluster.org/#/c/glusterfs/+/22290/ > > -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part URL: From aspandey at redhat.com Wed Mar 6 17:03:17 2019 From: aspandey at redhat.com (Ashish Pandey) Date: Wed, 6 Mar 2019 12:03:17 -0500 (EST) Subject: [Gluster-devel] [Gluster-users] Gluster : Improvements on "heal info" command In-Reply-To: <1arqcq3vmp99pobus3sb2kaw.1551891086158@email.android.com> References: <1arqcq3vmp99pobus3sb2kaw.1551891086158@email.android.com> Message-ID: <1454335424.6314606.1551891797145.JavaMail.zimbra@redhat.com> No, it is not necessary that the first brick would be local one. I really don't think starting from local node will make a difference. The major time spent is not in getting list of entries from .gluster/indices/xattrop folder. The LOCK->XATTR_CHECK->UNLOCK is the cycle which takes most of the time which is not going to change even if it is from local brick. --- Ashish ----- Original Message ----- From: "Strahil" To: "Ashish" , "Gluster" , "Gluster" Sent: Wednesday, March 6, 2019 10:21:26 PM Subject: Re: [Gluster-users] Gluster : Improvements on "heal info" command Hi , This sounds nice. I would like to ask if the order is starting from the local node's bricks first ? (I am talking about --brick=one) Best Regards, Strahil Nikolov On Mar 5, 2019 10:51, Ashish Pandey wrote: Hi All, We have observed and heard from gluster users about the long time "heal info" command takes. Even when we all want to know if a gluster volume is healthy or not, it takes time to list down all the files from all the bricks after which we can be sure if the volume is healthy or not. Here, we have come up with some options for "heal info" command which provide report quickly and reliably. gluster v heal vol info --subvol=[number of the subvol] --brick=[one,all] -------- Problem: "gluster v heal info" command picks each subvolume and checks the .glusterfs/indices/xattrop folder of every brick of that subvolume to find out if there is any entry which needs to be healed. It picks the entry and takes a lock on that entry to check xattrs to find out if that entry actually needs heal or not. This LOCK->CHECK-XATTR->UNLOCK cycle takes lot of time for each file. Let's consider two most often seen cases for which we use "heal info" and try to understand the improvements. Case -1 : Consider 4+2 EC volume and all the bricks on 6 different nodes. A brick of the volume is down and client has written 10000 files on one of the mount point of this volume. Entries for these 10K files will be created on ".glusterfs/indices/xattrop" on all the rest of 5 bricks. Now, brick is UP and when we use "heal info" command for this volume, it goes to all the bricks and picks these 10K file entries and goes through LOCK->CHECK-XATTR->UNLOCK cycle for all the files. This happens for all the bricks, that means, we check 50K files and perform the LOCK->CHECK-XATTR->UNLOCK cycle 50K times, while only 10K entries were sufficient to check. It is a very time consuming operation. If IO"s are happening one some of the new files, we check these files also which will add the time. Here, all we wanted to know if our volume has been healed and healthy. Solution : Whenever a brick goes down and comes up and when we use "heal info" command, our *main intention* is to find out if the volume is *healthy* or *unhealthy*. A volume is unhealthy even if one file is not healthy. So, we should scan bricks one by one and as soon as we find that one brick is having some entries which require to be healed, we can come out and list the files and say the volume is not healthy. No need to scan rest of the bricks. That's where "--brick=[one,all]" option has been introduced. "gluster v heal vol info --brick=[one,all]" "one" - It will scan the brick sequentially and as soon as it will find any unhealthy entries, it will list it out and stop scanning other bricks. "all" - It will act just like current behavior and provide all the files from all the bricks. If we do not provide this option, default (current) behavior will be applicable. Case -2 : Consider 24 X (4+2) EC volume. Let's say one brick from *only one* of the sub volume has been replaced and a heal has been triggered. To know if the volume is in healthy state, we go to each brick of *each and every sub volume* and check if there are any entries in ".glusterfs/indices/xattrop" folder which need heal or not. If we know which sub volume participated in brick replacement, we just need to check health of that sub volume and not query/check other sub volumes. If several clients are writing number of files on this volume, an entry for each of these files will be created in .glusterfs/indices/xattrop and "heal info' command will go through LOCK->CHECK-XATTR->UNLOCK cycle to find out if these entries need heal or not which takes lot of time. In addition to this a client will also see performance drop as it will have to release and take lock again. Solution: Provide an option to mention number of sub volume for which we want to check heal info. "gluster v heal vol info --subvol= " Here, --subvol will be given number of the subvolume we want to check. Example: "gluster v heal vol info --subvol=1 " =================================== Performance Data - A quick performance test done on standalone system. Type: Distributed-Disperse Volume ID: ea40eb13-d42c-431c-9c89-0153e834e67e Status: Started Snapshot Count: 0 Number of Bricks: 2 x (4 + 2) = 12 Transport-type: tcp Bricks: Brick1: apandey:/home/apandey/bricks/gluster/vol-1 Brick2: apandey:/home/apandey/bricks/gluster/vol-2 Brick3: apandey:/home/apandey/bricks/gluster/vol-3 Brick4: apandey:/home/apandey/bricks/gluster/vol-4 Brick5: apandey:/home/apandey/bricks/gluster/vol-5 Brick6: apandey:/home/apandey/bricks/gluster/vol-6 Brick7: apandey:/home/apandey/bricks/gluster/new-1 Brick8: apandey:/home/apandey/bricks/gluster/new-2 Brick9: apandey:/home/apandey/bricks/gluster/new-3 Brick10: apandey:/home/apandey/bricks/gluster/new-4 Brick11: apandey:/home/apandey/bricks/gluster/new-5 Brick12: apandey:/home/apandey/bricks/gluster/new-6 Just disabled the shd to get the data - Killed one brick each from two subvolumes and wrote 2000 files on mount point. [root at apandey vol]# for i in {1..2000};do echo abc >> file-$i; done Start the volume using force option and get the heal info. Following is the data - [root at apandey glusterfs]# time gluster v heal vol info --brick=one >> /dev/null <<<<<<<< This will scan brick one by one and come out as soon as we find volume is unhealthy. real 0m8.316s user 0m2.241s sys 0m1.278s [root at apandey glusterfs]# [root at apandey glusterfs]# time gluster v heal vol info >> /dev/null <<<<<<<< This is current behavior. real 0m26.097s user 0m10.868s sys 0m6.198s [root at apandey glusterfs]# =================================== I would like your comments/suggestions on this improvements. Specially, would like to hear on the new syntax of the command - gluster v heal vol info --subvol=[number of the subvol] --brick=[one,all] Note that if we do not provide new options, command will behave just like it does right now. Also, this improvement is valid for AFR and EC. --- Ashish _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Thu Mar 7 06:54:01 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Thu, 7 Mar 2019 12:24:01 +0530 Subject: [Gluster-devel] [Gluster-users] Experiences with FUSE in real world - Presentationat Vault 2019 In-Reply-To: References: Message-ID: Unfortunately, there is no recording. However, we are willing to discuss our findings if you've specific questions. We can do that in this thread. On Thu, Mar 7, 2019 at 10:33 AM Strahil wrote: > Thanks a lot. > Is there a recording of that ? > > Best Regards, > Strahil Nikolov > On Mar 5, 2019 11:13, Raghavendra Gowdappa wrote: > > All, > > Recently me, Manoj and Csaba presented on positives and negatives of > implementing File systems in userspace using FUSE [1]. We had based the > talk on our experiences with Glusterfs having FUSE as the native interface. > The slides can also be found at [1]. > > [1] https://www.usenix.org/conference/vault19/presentation/pillai > > regards, > Raghavendra > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dkhandel at redhat.com Thu Mar 7 09:46:07 2019 From: dkhandel at redhat.com (Deepshikha Khandelwal) Date: Thu, 7 Mar 2019 15:16:07 +0530 Subject: [Gluster-devel] [Gluster-infra] Upgrading build.gluster.org Message-ID: Hello, I?ve planned to do an upgrade of build.gluster.org tomorrow morning so as to install and pull in the latest security upgrade of the Jenkins plugins. I?ll stop all the running jobs and re-trigger them once I'm done with upgradation. The downtime window will be from : UTC: 0330 to 0400 IST: 0900 to 0930 The outage is for 30 minutes. Please bear with us as we continue to ensure the latest plugins and fixes for build.gluster.org Thanks, Deepshikha -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Thu Mar 7 13:43:32 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Thu, 7 Mar 2019 19:13:32 +0530 Subject: [Gluster-devel] [Gluster-users] Experiences with FUSE in real world - Presentationat Vault 2019 In-Reply-To: <0qhas4o39y8l9my09wxfritk.1551957758236@email.android.com> References: <0qhas4o39y8l9my09wxfritk.1551957758236@email.android.com> Message-ID: On Thu, Mar 7, 2019 at 4:51 PM Strahil wrote: > Thanks, > > I have nothing in mind - but I know from experience that live sessions are > much more interesting and going in deep. > I'll schedule a Bluejeans session on this. Will update the thread with a date and time. Best Regards, > Strahil Nikolov > On Mar 7, 2019 08:54, Raghavendra Gowdappa wrote: > > Unfortunately, there is no recording. However, we are willing to discuss > our findings if you've specific questions. We can do that in this thread. > > On Thu, Mar 7, 2019 at 10:33 AM Strahil wrote: > > Thanks a lot. > Is there a recording of that ? > > Best Regards, > Strahil Nikolov > On Mar 5, 2019 11:13, Raghavendra Gowdappa wrote: > > All, > > Recently me, Manoj and Csaba presented on positives and negatives of > implementing File systems in userspace using FUSE [1]. We had based the > talk on our experiences with Glusterfs having FUSE as the native interface. > The slides can also be found at [1]. > > [1] https://www.usenix.org/conference/vault19/presentation/pillai > > regards, > Raghavendra > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amukherj at redhat.com Thu Mar 7 18:20:12 2019 From: amukherj at redhat.com (Atin Mukherjee) Date: Thu, 7 Mar 2019 23:50:12 +0530 Subject: [Gluster-devel] [Gluster-Maintainers] GlusterFS - 6.0RC - Test days (27th, 28th Feb) In-Reply-To: <2257667e-751a-2c0e-6c99-b7aedbab379d@redhat.com> References: <5f27393d-cefb-aa47-12f8-1597677ffb50@redhat.com> <2257667e-751a-2c0e-6c99-b7aedbab379d@redhat.com> Message-ID: I am not sure how BZ 1683815 can be a blocker at RC. We have a fix ready, but to me it doesn't look like a blocker. Vijay - any objections? Also the bugzilla dependency of all bugs attached to the release-6 is sort of messed up. I see most of the times a mainline bug along with its clones are attached to the tracker which is unnecessary. This has happened because of default clone but I request every bugzilla assignees to spend few additional seconds to establish the right dependency. I have tried to correct few of them and will do the rest by next Monday. That?d help us to filter out the unnecessary ones and get to know how many actual blockers we have. On Tue, Mar 5, 2019 at 11:51 PM Shyam Ranganathan wrote: > On 3/4/19 12:33 PM, Shyam Ranganathan wrote: > > On 3/4/19 10:08 AM, Atin Mukherjee wrote: > >> > >> > >> On Mon, 4 Mar 2019 at 20:33, Amar Tumballi Suryanarayan > >> > wrote: > >> > >> Thanks to those who participated. > >> > >> Update at present: > >> > >> We found 3 blocker bugs in upgrade scenarios, and hence have marked > >> release > >> as pending upon them. We will keep these lists updated about > progress. > >> > >> > >> I?d like to clarify that upgrade testing is blocked. So just fixing > >> these test blocker(s) isn?t enough to call release-6 green. We need to > >> continue and finish the rest of the upgrade tests once the respective > >> bugs are fixed. > > > > Based on fixes expected by tomorrow for the upgrade fixes, we will build > > an RC1 candidate on Wednesday (6-Mar) (tagging early Wed. Eastern TZ). > > This RC can be used for further testing. > > There have been no backports for the upgrade failures, request folks > working on the same to post a list of bugs that need to be fixed, to > enable tracking the same. (also, ensure they are marked against the > release-6 tracker [1]) > > Also, we need to start writing out the upgrade guide for release-6, any > volunteers for the same? > > Thanks, > Shyam > > [1] Release-6 tracker bug: > https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-6.0 > -- - Atin (atinm) -------------- next part -------------- An HTML attachment was scrubbed... URL: From vbellur at redhat.com Thu Mar 7 19:25:12 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Thu, 7 Mar 2019 11:25:12 -0800 Subject: [Gluster-devel] [Gluster-Maintainers] GlusterFS - 6.0RC - Test days (27th, 28th Feb) In-Reply-To: References: <5f27393d-cefb-aa47-12f8-1597677ffb50@redhat.com> <2257667e-751a-2c0e-6c99-b7aedbab379d@redhat.com> Message-ID: On Thu, Mar 7, 2019 at 10:20 AM Atin Mukherjee wrote: > I am not sure how BZ 1683815 > can be a blocker at > RC. We have a fix ready, but to me it doesn't look like a blocker. Vijay - > any objections? > None from me as it is an existing minor problem across multiple releases now. I guess it got added as a blocker for RC as I used the URL in the testing day announcement email. We can remove it from the blocker list for RC. Thanks, Vijay > Also the bugzilla dependency of all bugs attached to the release-6 is sort > of messed up. I see most of the times a mainline bug along with its clones > are attached to the tracker which is unnecessary. This has happened because > of default clone but I request every bugzilla assignees to spend few > additional seconds to establish the right dependency. > > I have tried to correct few of them and will do the rest by next Monday. > That?d help us to filter out the unnecessary ones and get to know how many > actual blockers we have. > > On Tue, Mar 5, 2019 at 11:51 PM Shyam Ranganathan > wrote: > >> On 3/4/19 12:33 PM, Shyam Ranganathan wrote: >> > On 3/4/19 10:08 AM, Atin Mukherjee wrote: >> >> >> >> >> >> On Mon, 4 Mar 2019 at 20:33, Amar Tumballi Suryanarayan >> >> > wrote: >> >> >> >> Thanks to those who participated. >> >> >> >> Update at present: >> >> >> >> We found 3 blocker bugs in upgrade scenarios, and hence have marked >> >> release >> >> as pending upon them. We will keep these lists updated about >> progress. >> >> >> >> >> >> I?d like to clarify that upgrade testing is blocked. So just fixing >> >> these test blocker(s) isn?t enough to call release-6 green. We need to >> >> continue and finish the rest of the upgrade tests once the respective >> >> bugs are fixed. >> > >> > Based on fixes expected by tomorrow for the upgrade fixes, we will build >> > an RC1 candidate on Wednesday (6-Mar) (tagging early Wed. Eastern TZ). >> > This RC can be used for further testing. >> >> There have been no backports for the upgrade failures, request folks >> working on the same to post a list of bugs that need to be fixed, to >> enable tracking the same. (also, ensure they are marked against the >> release-6 tracker [1]) >> >> Also, we need to start writing out the upgrade guide for release-6, any >> volunteers for the same? >> >> Thanks, >> Shyam >> >> [1] Release-6 tracker bug: >> https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-6.0 >> > -- > - Atin (atinm) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhishpaliwal at gmail.com Fri Mar 8 08:49:40 2019 From: abhishpaliwal at gmail.com (ABHISHEK PALIWAL) Date: Fri, 8 Mar 2019 14:19:40 +0530 Subject: [Gluster-devel] Glusterfsd crashed with SIGSEGV Message-ID: Hi Team, I am using Glusterfs 5.4, where after setting the gluster mount point when trying to access it, glusterfsd is getting crashed and mount point through the "Transport endpoint is not connected error. Here I are the gdb log for the core file warning: Could not load shared library symbols for linux-vdso64.so.1. Do you need "set solib-search-path" or "set sysroot"? [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 --volfile-id gv0.128.224.95.140.tmp-bric'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, bytes=bytes at entry=36) at malloc.c:3327 3327 { [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))] (gdb) (gdb) (gdb) bt #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, bytes=bytes at entry=36) at malloc.c:3327 #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at malloc.c:2921 #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, len=) at xdr_sizeof.c:89 #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0 #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 #5 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c132020, size=, proc=) at xdr_ref.c:84 #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c132020, obj_size=, xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 #8 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131ea0, size=, proc=) at xdr_ref.c:84 #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131ea0, obj_size=, xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 #11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131d20, size=, proc=) at xdr_ref.c:84 #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131d20, obj_size=, xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 #14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131ba0, size=, proc=) at xdr_ref.c:84 #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131ba0, obj_size=, xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 #17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131a20, size=, proc=) at xdr_ref.c:84 #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131a20, obj_size=, xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 #20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c1318a0, size=, proc=) at xdr_ref.c:84 #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c1318a0, obj_size=, xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 #23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131720, size=, proc=) at xdr_ref.c:84 #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131720, obj_size=, xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 #26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c1315a0, size=, proc=) at xdr_ref.c:84 #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c1315a0, obj_size=, xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 #29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c131420, size=, proc=) at xdr_ref.c:84 #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c131420, obj_size=, xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 #32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, pp=0x3fff7c1312a0, size=, proc=) at xdr_ref.c:84 #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, objpp=0x3fff7c1312a0, obj_size=, xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 Frames are getting repeated, could any one please me. -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: From nigel at nigelb.me Sun Mar 10 18:01:00 2019 From: nigel at nigelb.me (Nigel Babu) Date: Sun, 10 Mar 2019 14:01:00 -0400 Subject: [Gluster-devel] Removing myself as maintainer Message-ID: Hello folks, This change has gone through, but I wanted to let folks here know as well. I'm removing myself as maintainer from everything to reflect that I will no longer be the primary point of contact for any of the components I used to own. However, I will still be around and contributing as I get time and energy. -- nigelb From jenkins at build.gluster.org Mon Mar 11 01:45:02 2019 From: jenkins at build.gluster.org (jenkins at build.gluster.org) Date: Mon, 11 Mar 2019 01:45:02 +0000 (UTC) Subject: [Gluster-devel] Weekly Untriaged Bugs Message-ID: <1047651482.8.1552268703068.JavaMail.jenkins@jenkins-el7.rht.gluster.org> [...truncated 6 lines...] https://bugzilla.redhat.com/1679904 / core: client log flooding with intentional socket shutdown message when a brick is down https://bugzilla.redhat.com/1685023 / core: FD processes for larger files are not closed soon after FOP finished https://bugzilla.redhat.com/1677555 / core: Glusterfs brick is crashed due to segfault caused by broken gfid symlink https://bugzilla.redhat.com/1686396 / core: ls and rm run on contents of same directory from a single mount point results in ENOENT errors https://bugzilla.redhat.com/1683815 / core: Memory leak when peer detach fails https://bugzilla.redhat.com/1679744 / core: Minio gateway nas does not work with 2 + 1 dispersed volumes https://bugzilla.redhat.com/1678640 / core: Running 'control-cpu-load.sh' prevents CTDB starting https://bugzilla.redhat.com/1679892 / glusterd: assertion failure log in glusterd.log file when a volume start is triggered https://bugzilla.redhat.com/1683880 / glusterd: Multiple shd processes are running on brick_mux environmet https://bugzilla.redhat.com/1683526 / glusterd: rebalance start command doesn't throw up error message if the command fails https://bugzilla.redhat.com/1686353 / libgfapi: flooding of "dict is NULL" logging https://bugzilla.redhat.com/1687063 / locks: glusterd :symbol lookup error: undefined symbol :use_spinlocks https://bugzilla.redhat.com/1679169 / md-cache: Integer Overflow possible in md-cache.c due to data type inconsistency https://bugzilla.redhat.com/1679170 / md-cache: Integer Overflow possible in md-cache.c due to data type inconsistency https://bugzilla.redhat.com/1677557 / nfs: gNFS crashed when processing "gluster v profile [vol] info nfs" https://bugzilla.redhat.com/1685337 / packaging: Updating Fedora 28 fail with "Package glusterfs-5.4-1.fc28.x86_64.rpm is not signed" https://bugzilla.redhat.com/1677804 / posix-acl: POSIX ACLs are absent on FUSE-mounted volume using tmpfs bricks (posix-acl-autoload usually returns -1) https://bugzilla.redhat.com/1678378 / project-infrastructure: Add a nightly build verification job in Jenkins for release-6 https://bugzilla.redhat.com/1685576 / project-infrastructure: DNS delegation record for rhhi-dev.gluster.org https://bugzilla.redhat.com/1685813 / project-infrastructure: Not able to run centos-regression getting exception error https://bugzilla.redhat.com/1686754 / project-infrastructure: Requesting merge rights for Cloudsync https://bugzilla.redhat.com/1686461 / quota: Quotad.log filled with 0-dict is not sent on wire [Invalid argument] messages https://bugzilla.redhat.com/1682925 / replicate: Gluster volumes never heal during oVirt 4.2->4.3 upgrade https://bugzilla.redhat.com/1683317 / tests: ./tests/bugs/glusterfs/bug-866459.t failing on s390x https://bugzilla.redhat.com/1676356 / write-behind: glusterfs FUSE client crashing every few days with 'Failed to dispatch handler' [...truncated 2 lines...] -------------- next part -------------- A non-text attachment was scrubbed... Name: build.log Type: application/octet-stream Size: 3240 bytes Desc: not available URL: From abhishpaliwal at gmail.com Mon Mar 11 10:09:34 2019 From: abhishpaliwal at gmail.com (ABHISHEK PALIWAL) Date: Mon, 11 Mar 2019 15:39:34 +0530 Subject: [Gluster-devel] Glusterfsd crashed with SIGSEGV In-Reply-To: References: Message-ID: Hi Team, COuld you please provide some pointer to debug it further. Regards, Abhishek On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL wrote: > Hi Team, > > I am using Glusterfs 5.4, where after setting the gluster mount point when > trying to access it, glusterfsd is getting crashed and mount point through > the "Transport endpoint is not connected error. > > Here I are the gdb log for the core file > > warning: Could not load shared library symbols for linux-vdso64.so.1. > Do you need "set solib-search-path" or "set sysroot"? > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 --volfile-id > gv0.128.224.95.140.tmp-bric'. > Program terminated with signal SIGSEGV, Segmentation fault. > #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, > bytes=bytes at entry=36) at malloc.c:3327 > 3327 { > [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))] > (gdb) > (gdb) > (gdb) bt > #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, > bytes=bytes at entry=36) at malloc.c:3327 > #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at malloc.c:2921 > #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, len= out>) at xdr_sizeof.c:89 > #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0 > #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > #5 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, > pp=0x3fff7c132020, size=, proc=) at > xdr_ref.c:84 > #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, > objpp=0x3fff7c132020, obj_size=, > xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > #8 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, > pp=0x3fff7c131ea0, size=, proc=) at > xdr_ref.c:84 > #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, > objpp=0x3fff7c131ea0, obj_size=, > xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > #11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, > pp=0x3fff7c131d20, size=, proc=) at > xdr_ref.c:84 > #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, > objpp=0x3fff7c131d20, obj_size=, > xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > #14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, > pp=0x3fff7c131ba0, size=, proc=) at > xdr_ref.c:84 > #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, > objpp=0x3fff7c131ba0, obj_size=, > xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > #17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, > pp=0x3fff7c131a20, size=, proc=) at > xdr_ref.c:84 > #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, > objpp=0x3fff7c131a20, obj_size=, > xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > #20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, > pp=0x3fff7c1318a0, size=, proc=) at > xdr_ref.c:84 > #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, > objpp=0x3fff7c1318a0, obj_size=, > xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > #23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, > pp=0x3fff7c131720, size=, proc=) at > xdr_ref.c:84 > #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, > objpp=0x3fff7c131720, obj_size=, > xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > #26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, > pp=0x3fff7c1315a0, size=, proc=) at > xdr_ref.c:84 > #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, > objpp=0x3fff7c1315a0, obj_size=, > xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > #29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, > pp=0x3fff7c131420, size=, proc=) at > xdr_ref.c:84 > #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, > objpp=0x3fff7c131420, obj_size=, > xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > #32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, > pp=0x3fff7c1312a0, size=, proc=) at > xdr_ref.c:84 > #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, > objpp=0x3fff7c1312a0, obj_size=, > xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > > Frames are getting repeated, could any one please me. > -- > Regards > Abhishek Paliwal > -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: From atumball at redhat.com Mon Mar 11 13:38:36 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Mon, 11 Mar 2019 19:08:36 +0530 Subject: [Gluster-devel] [Gluster-users] Glusterfsd crashed with SIGSEGV In-Reply-To: References: Message-ID: Hi Abhishek, Can you check and get back to us? ``` bash# ldd /usr/lib64/libglusterfs.so bash# ldd /usr/lib64/libgfrpc.so ``` Also considering you have the core, can you do `(gdb) thr apply all bt full` and pass it on? Thanks & Regards, Amar On Mon, Mar 11, 2019 at 3:41 PM ABHISHEK PALIWAL wrote: > Hi Team, > > COuld you please provide some pointer to debug it further. > > Regards, > Abhishek > > On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL > wrote: > >> Hi Team, >> >> I am using Glusterfs 5.4, where after setting the gluster mount point >> when trying to access it, glusterfsd is getting crashed and mount point >> through the "Transport endpoint is not connected error. >> >> Here I are the gdb log for the core file >> >> warning: Could not load shared library symbols for linux-vdso64.so.1. >> Do you need "set solib-search-path" or "set sysroot"? >> [Thread debugging using libthread_db enabled] >> Using host libthread_db library "/lib64/libthread_db.so.1". >> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >> --volfile-id gv0.128.224.95.140.tmp-bric'. >> Program terminated with signal SIGSEGV, Segmentation fault. >> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >> bytes=bytes at entry=36) at malloc.c:3327 >> 3327 { >> [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))] >> (gdb) >> (gdb) >> (gdb) bt >> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >> bytes=bytes at entry=36) at malloc.c:3327 >> #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at malloc.c:2921 >> #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, len=> out>) at xdr_sizeof.c:89 >> #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0 >> #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> #5 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >> pp=0x3fff7c132020, size=, proc=) at >> xdr_ref.c:84 >> #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >> objpp=0x3fff7c132020, obj_size=, >> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> #8 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >> pp=0x3fff7c131ea0, size=, proc=) at >> xdr_ref.c:84 >> #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >> objpp=0x3fff7c131ea0, obj_size=, >> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> #11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >> pp=0x3fff7c131d20, size=, proc=) at >> xdr_ref.c:84 >> #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >> objpp=0x3fff7c131d20, obj_size=, >> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> #14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >> pp=0x3fff7c131ba0, size=, proc=) at >> xdr_ref.c:84 >> #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >> objpp=0x3fff7c131ba0, obj_size=, >> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> #17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >> pp=0x3fff7c131a20, size=, proc=) at >> xdr_ref.c:84 >> #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >> objpp=0x3fff7c131a20, obj_size=, >> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> #20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >> pp=0x3fff7c1318a0, size=, proc=) at >> xdr_ref.c:84 >> #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >> objpp=0x3fff7c1318a0, obj_size=, >> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> #23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >> pp=0x3fff7c131720, size=, proc=) at >> xdr_ref.c:84 >> #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >> objpp=0x3fff7c131720, obj_size=, >> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> #26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >> pp=0x3fff7c1315a0, size=, proc=) at >> xdr_ref.c:84 >> #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >> objpp=0x3fff7c1315a0, obj_size=, >> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> #29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >> pp=0x3fff7c131420, size=, proc=) at >> xdr_ref.c:84 >> #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >> objpp=0x3fff7c131420, obj_size=, >> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> #32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >> pp=0x3fff7c1312a0, size=, proc=) at >> xdr_ref.c:84 >> #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >> objpp=0x3fff7c1312a0, obj_size=, >> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> >> Frames are getting repeated, could any one please me. >> -- >> Regards >> Abhishek Paliwal >> > > > -- > > > > > Regards > Abhishek Paliwal > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Tue Mar 12 02:46:19 2019 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Tue, 12 Mar 2019 08:16:19 +0530 Subject: [Gluster-devel] Github#268 Compatibility with Alpine Linux Message-ID: Saw some recent activity on - is there a plan to address this or, should the interested users be informed about other plans? /s From rgowdapp at redhat.com Tue Mar 12 04:17:27 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Tue, 12 Mar 2019 09:47:27 +0530 Subject: [Gluster-devel] [Gluster-users] "rpc_clnt_ping_timer_expired" errors In-Reply-To: References: <96B07283-D8AB-4F06-909D-E00424625528@cmcc.it> <42758A0E-8BE9-497D-BDE3-55D7DC633BC7@cmcc.it> <6A8CF4A4-98EA-48C3-A059-D60D1B2721C7@cmcc.it> <2CF49168-9C1B-4931-8C34-8157262A137A@cmcc.it> <7A151CC9-A0AE-4A45-B450-A4063D216D9E@cmcc.it> <32D53ECE-3F49-4415-A6EE-241B351AC2BA@cmcc.it> <4685A75B-5978-4338-9C9F-4A02FB40B9BC@cmcc.it> <4D2E6B04-C2E8-4FD5-B72D-E7C05931C6F9@cmcc.it> <4E332A56-B318-4BC1-9F44-84AB4392A5DE@cmcc.it> <832FD362-3B14-40D8-8530-604419300476@cmcc.it> <8D926643-1093-48ED-823F-D8F117F702CF@cmcc.it> <9D0BE438-DF11-4D0A-AB85-B44357032F29@cmcc.it> <5F0AC378-8170-4342-8473-9C17159CAC1D@cmcc.it> <7A50E86D-9E27-4EA7-883B-45E9F973991A@cmcc.it> <58B5DB7F-DCB4-4FBF-B1F7-681315B1613A@cmcc.it> <6327B44F-4E7E-46BB-A74C-70F4457DD1EB@cmcc.it> <0167DF4A-8329-4A1A-B439-857DFA6F78BB@cmcc.it> <763F334E-35B8-4729-B8E1-D30866754EEE@cmcc.it> <91DFC9AC-4805-41EB-AC6F-5722E018DD6E@cmcc.it> <8A9752B8-B231-4570-8FF4-8D3D781E7D42@cmcc.it> <47A24804-F975-4EE6-9FA5-67FCDA18D707@cmcc.it> <637FF5D2-D1F4-4686-9D48-646A96F67B96@cmcc.it> <4A87495F-3755-48F7-8507-085869069C64@cmcc.it> <3854BBF6-5B98-4AB3-A67E-E7DE59E69A63@cmcc.it> <313DA021-9173-4899-96B0-831B10B00B61@cmcc.it> <17996AFD-DFC8-40F3-9D09-DB6DDAD5B7D6@cmcc.it> <7074B5D8-955A-4802-A9F3-606C99734417@cmcc.it> <83B84BF9-8334-4230-B6F8-0BC4BFBEFF15@cmcc.it> <133B9AE4-9792-4F72-AD91-D36E7B9EC711@cmcc.it> <6611C4B0-57FD-4390-88B5-BD373555D4C5@cmcc.it> Message-ID: Was the suggestion to increase server.event-thread values tried? If yes, what were the results? On Mon, Mar 11, 2019 at 2:40 PM Mauro Tridici wrote: > Dear All, > > do you have any suggestions about the right way to "debug" this issue? > In attachment, the updated logs of ?s06" gluster server. > > I noticed a lot of intermittent warning and error messages. > > Thank you in advance, > Mauro > > > > On 4 Mar 2019, at 18:45, Raghavendra Gowdappa wrote: > > > +Gluster Devel , +Gluster-users > > > I would like to point out another issue. Even if what I suggested prevents > disconnects, part of the solution would be only symptomatic treatment and > doesn't address the root cause of the problem. In most of the > ping-timer-expiry issues, the root cause is the increased load on bricks > and the inability of bricks to be responsive under high load. So, the > actual solution would be doing any or both of the following: > * identify the source of increased load and if possible throttle it. > Internal heal processes like self-heal, rebalance, quota heal are known to > pump traffic into bricks without much throttling (io-threads _might_ do > some throttling, but my understanding is its not sufficient). > * identify the reason for bricks to become unresponsive during load. This > may be fixable issues like not enough event-threads to read from network or > difficult to fix issues like fsync on backend fs freezing the process or > semi fixable issues (in code) like lock contention. > > So any genuine effort to fix ping-timer-issues (to be honest most of the > times they are not issues related to rpc/network) would involve performance > characterization of various subsystems on bricks and clients. Various > subsystems can include (but not necessarily limited to), underlying > OS/filesystem, glusterfs processes, CPU consumption etc > > regards, > Raghavendra > > On Mon, Mar 4, 2019 at 9:31 PM Mauro Tridici > wrote: > >> Thank you, let?s try! >> I will inform you about the effects of the change. >> >> Regards, >> Mauro >> >> On 4 Mar 2019, at 16:55, Raghavendra Gowdappa >> wrote: >> >> >> >> On Mon, Mar 4, 2019 at 8:54 PM Mauro Tridici >> wrote: >> >>> Hi Raghavendra, >>> >>> thank you for your reply. >>> Yes, you are right. It is a problem that seems to happen randomly. >>> At this moment, server.event-threads value is 4. I will try to increase >>> this value to 8. Do you think that it could be a valid value ? >>> >> >> Yes. We can try with that. You should see at least frequency of >> ping-timer related disconnects reduce with this value (even if it doesn't >> eliminate the problem completely). >> >> >>> Regards, >>> Mauro >>> >>> >>> On 4 Mar 2019, at 15:36, Raghavendra Gowdappa >>> wrote: >>> >>> >>> >>> On Mon, Mar 4, 2019 at 8:01 PM Nithya Balachandran >>> wrote: >>> >>>> Hi Mauro, >>>> >>>> It looks like some problem on s06. Are all your other nodes ok? Can you >>>> send us the gluster logs from this node? >>>> >>>> @Raghavendra G , do you have any idea as to >>>> how this can be debugged? Maybe running top ? Or debug brick logs? >>>> >>> >>> If we can reproduce the problem, collecting tcpdump on both ends of >>> connection will help. But, one common problem is these bugs are >>> inconsistently reproducible and hence we may not be able to capture tcpdump >>> at correct intervals. Other than that, we can try to collect some evidence >>> that poller threads were busy (waiting on locks). But, not sure what debug >>> data provides that information. >>> >>> From what I know, its difficult to collect evidence for this issue and >>> we could only reason about it. >>> >>> We can try a workaround though - try increasing server.event-threads and >>> see whether ping-timer expiry issues go away with an optimal value. If >>> that's the case, it kind of provides proof for our hypothesis. >>> >>> >>>> >>>> Regards, >>>> Nithya >>>> >>>> On Mon, 4 Mar 2019 at 15:25, Mauro Tridici >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> some minutes ago I received this message from NAGIOS server >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> ****** Nagios *****Notification Type: PROBLEMService: Brick - >>>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: CRITICALDate/Time: Mon >>>>> Mar 4 10:25:33 CET 2019Additional Info:CHECK_NRPE STATE CRITICAL: Socket >>>>> timeout after 10 seconds.* >>>>> >>>>> I checked the network, RAM and CPUs usage on s06 node and everything >>>>> seems to be ok. >>>>> No bricks are in error state. In /var/log/messages, I detected again a >>>>> crash of ?check_vol_utili? that I think it is a module used by NRPE >>>>> executable (that is the NAGIOS client). >>>>> >>>>> Mar 4 10:15:29 s06 kernel: traps: check_vol_utili[161224] general >>>>> protection ip:7facffa0a66d sp:7ffe9f4e6fc0 error:0 in >>>>> libglusterfs.so.0.0.1[7facff9b7000+f7000] >>>>> Mar 4 10:15:29 s06 abrt-hook-ccpp: Process 161224 (python2.7) of user >>>>> 0 killed by SIGSEGV - dumping core >>>>> Mar 4 10:15:29 s06 abrt-server: Generating core_backtrace >>>>> Mar 4 10:15:29 s06 abrt-server: Error: Unable to open './coredump': >>>>> No such file or directory >>>>> Mar 4 10:16:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:16:01 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:16:01 s06 systemd: Started Session 201010 of user root. >>>>> Mar 4 10:16:01 s06 systemd: Starting Session 201010 of user root. >>>>> Mar 4 10:16:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 4 10:16:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 4 10:16:24 s06 abrt-server: Duplicate: UUID >>>>> Mar 4 10:16:24 s06 abrt-server: DUP_OF_DIR: >>>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>> Mar 4 10:16:24 s06 abrt-server: Deleting problem directory >>>>> ccpp-2019-03-04-10:15:29-161224 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>> Mar 4 10:16:24 s06 abrt-server: Generating core_backtrace >>>>> Mar 4 10:16:24 s06 abrt-server: Error: Unable to open './coredump': >>>>> No such file or directory >>>>> Mar 4 10:16:24 s06 abrt-server: Cannot notify >>>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>>> 'report_uReport' exited with 1 >>>>> Mar 4 10:16:24 s06 abrt-hook-ccpp: Process 161391 (python2.7) of user >>>>> 0 killed by SIGABRT - dumping core >>>>> Mar 4 10:16:25 s06 abrt-server: Generating core_backtrace >>>>> Mar 4 10:16:25 s06 abrt-server: Error: Unable to open './coredump': >>>>> No such file or directory >>>>> Mar 4 10:17:01 s06 systemd: Created slice User Slice of root. >>>>> >>>>> Also, I noticed the following errors that I think are very critical: >>>>> >>>>> Mar 4 10:21:12 s06 glustershd[20355]: [2019-03-04 09:21:12.954798] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server >>>>> 192.168.0.55:49158 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:22:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:22:01 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:22:01 s06 systemd: Started Session 201017 of user root. >>>>> Mar 4 10:22:01 s06 systemd: Starting Session 201017 of user root. >>>>> Mar 4 10:22:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 4 10:22:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 4 10:22:03 s06 glustershd[20355]: [2019-03-04 09:22:03.964120] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server >>>>> 192.168.0.54:49165 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:23:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:23:01 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:23:01 s06 systemd: Started Session 201018 of user root. >>>>> Mar 4 10:23:01 s06 systemd: Starting Session 201018 of user root. >>>>> Mar 4 10:23:02 s06 systemd: Removed slice User Slice of root. >>>>> Mar 4 10:23:02 s06 systemd: Stopping User Slice of root. >>>>> Mar 4 10:24:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:24:01 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:24:01 s06 systemd: Started Session 201019 of user root. >>>>> Mar 4 10:24:01 s06 systemd: Starting Session 201019 of user root. >>>>> Mar 4 10:24:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 4 10:24:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 4 10:24:03 s06 glustershd[20355]: [2019-03-04 09:24:03.982502] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-16: server >>>>> 192.168.0.52:49158 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746109] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-3: server >>>>> 192.168.0.51:49153 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746215] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server >>>>> 192.168.0.52:49156 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746260] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-21: server >>>>> 192.168.0.51:49159 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746296] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server >>>>> 192.168.0.52:49161 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746413] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server >>>>> 192.168.0.54:49165 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:24:07 s06 glustershd[20355]: [2019-03-04 09:24:07.982952] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-45: server >>>>> 192.168.0.54:49155 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:24:18 s06 glustershd[20355]: [2019-03-04 09:24:18.990929] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server >>>>> 192.168.0.52:49161 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:24:31 s06 glustershd[20355]: [2019-03-04 09:24:31.995781] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-20: server >>>>> 192.168.0.53:49159 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:25:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:25:01 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:25:01 s06 systemd: Started Session 201020 of user root. >>>>> Mar 4 10:25:01 s06 systemd: Starting Session 201020 of user root. >>>>> Mar 4 10:25:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 4 10:25:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 4 10:25:57 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:25:57 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:25:57 s06 systemd-logind: New session 201021 of user root. >>>>> Mar 4 10:25:57 s06 systemd: Started Session 201021 of user root. >>>>> Mar 4 10:25:57 s06 systemd: Starting Session 201021 of user root. >>>>> Mar 4 10:26:01 s06 systemd: Started Session 201022 of user root. >>>>> Mar 4 10:26:01 s06 systemd: Starting Session 201022 of user root. >>>>> Mar 4 10:26:21 s06 nrpe[162388]: Error: Could not complete SSL >>>>> handshake with 192.168.1.56: 5 >>>>> Mar 4 10:27:01 s06 systemd: Started Session 201023 of user root. >>>>> Mar 4 10:27:01 s06 systemd: Starting Session 201023 of user root. >>>>> Mar 4 10:28:01 s06 systemd: Started Session 201024 of user root. >>>>> Mar 4 10:28:01 s06 systemd: Starting Session 201024 of user root. >>>>> Mar 4 10:29:01 s06 systemd: Started Session 201025 of user root. >>>>> Mar 4 10:29:01 s06 systemd: Starting Session 201025 of user root. >>>>> >>>>> But, unfortunately, I don?t understand why it is happening. >>>>> Now, NAGIOS server shows that s06 status is ok: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> ****** Nagios *****Notification Type: RECOVERYService: Brick - >>>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: OKDate/Time: Mon Mar 4 >>>>> 10:35:23 CET 2019Additional Info:OK: Brick /gluster/mnt2/brick is up* >>>>> >>>>> Nothing is changed from RAM, CPUs, and NETWORK point of view. >>>>> /var/log/message file has been updated: >>>>> >>>>> Mar 4 10:32:01 s06 systemd: Starting Session 201029 of user root. >>>>> Mar 4 10:32:30 s06 glustershd[20355]: [2019-03-04 09:32:30.069082] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server >>>>> 192.168.0.52:49156 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:32:55 s06 glustershd[20355]: [2019-03-04 09:32:55.074689] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-66: server >>>>> 192.168.0.54:49167 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 4 10:33:01 s06 systemd: Started Session 201030 of user root. >>>>> Mar 4 10:33:01 s06 systemd: Starting Session 201030 of user root. >>>>> Mar 4 10:34:01 s06 systemd: Started Session 201031 of user root. >>>>> Mar 4 10:34:01 s06 systemd: Starting Session 201031 of user root. >>>>> Mar 4 10:35:01 s06 nrpe[162562]: Could not read request from client >>>>> 192.168.1.56, bailing out... >>>>> Mar 4 10:35:01 s06 nrpe[162562]: INFO: SSL Socket Shutdown. >>>>> Mar 4 10:35:01 s06 systemd: Started Session 201032 of user root. >>>>> Mar 4 10:35:01 s06 systemd: Starting Session 201032 of user root. >>>>> >>>>> Could you please help me to understand what it?s happening ? >>>>> Thank you in advance. >>>>> >>>>> Rergards, >>>>> Mauro >>>>> >>>>> >>>>> On 1 Mar 2019, at 12:17, Mauro Tridici wrote: >>>>> >>>>> >>>>> Thank you, Milind. >>>>> I executed the instructions you suggested: >>>>> >>>>> - grep ?blocked for? /var/log/messages on s06 returns no output (no >>>>> ?blocked? word is detected in messages file); >>>>> - in /var/log/messages file I can see this kind of error repeated for >>>>> a lot of times: >>>>> >>>>> Mar 1 08:43:01 s06 systemd: Starting Session 196071 of user root. >>>>> Mar 1 08:43:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 1 08:43:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 1 08:43:02 s06 kernel: traps: check_vol_utili[57091] general >>>>> protection ip:7f88e76ee66d sp:7ffe5a5bcc30 error:0 in >>>>> libglusterfs.so.0.0.1[7f88e769b000+f7000] >>>>> Mar 1 08:43:02 s06 abrt-hook-ccpp: Process 57091 (python2.7) of user >>>>> 0 killed by SIGSEGV - dumping core >>>>> Mar 1 08:43:02 s06 abrt-server: Generating core_backtrace >>>>> Mar 1 08:43:02 s06 abrt-server: Error: Unable to open './coredump': >>>>> No such file or directory >>>>> Mar 1 08:43:58 s06 abrt-server: Duplicate: UUID >>>>> Mar 1 08:43:58 s06 abrt-server: DUP_OF_DIR: >>>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>> Mar 1 08:43:58 s06 abrt-server: Deleting problem directory >>>>> ccpp-2019-03-01-08:43:02-57091 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Activating service >>>>> name='org.freedesktop.problems' (using servicehelper) >>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Successfully activated >>>>> service 'org.freedesktop.problems' >>>>> Mar 1 08:43:58 s06 abrt-server: Generating core_backtrace >>>>> Mar 1 08:43:58 s06 abrt-server: Error: Unable to open './coredump': >>>>> No such file or directory >>>>> Mar 1 08:43:58 s06 abrt-server: Cannot notify >>>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>>> 'report_uReport' exited with 1 >>>>> Mar 1 08:44:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 1 08:44:01 s06 systemd: Starting User Slice of root. >>>>> Mar 1 08:44:01 s06 systemd: Started Session 196072 of user root. >>>>> Mar 1 08:44:01 s06 systemd: Starting Session 196072 of user root. >>>>> Mar 1 08:44:01 s06 systemd: Removed slice User Slice of root. >>>>> >>>>> - in /var/log/messages file I can see also 4 errors related to other >>>>> cluster servers: >>>>> >>>>> Mar 1 11:05:01 s06 systemd: Starting User Slice of root. >>>>> Mar 1 11:05:01 s06 systemd: Started Session 196230 of user root. >>>>> Mar 1 11:05:01 s06 systemd: Starting Session 196230 of user root. >>>>> Mar 1 11:05:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 1 11:05:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 1 11:05:59 s06 glustershd[70117]: [2019-03-01 10:05:59.347094] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-33: server >>>>> 192.168.0.51:49163 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 1 11:06:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 1 11:06:01 s06 systemd: Starting User Slice of root. >>>>> Mar 1 11:06:01 s06 systemd: Started Session 196231 of user root. >>>>> Mar 1 11:06:01 s06 systemd: Starting Session 196231 of user root. >>>>> Mar 1 11:06:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 1 11:06:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 1 11:06:12 s06 glustershd[70117]: [2019-03-01 10:06:12.351319] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-1: server >>>>> 192.168.0.52:49153 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 1 11:06:38 s06 glustershd[70117]: [2019-03-01 10:06:38.356920] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-7: server >>>>> 192.168.0.52:49155 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 1 11:07:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 1 11:07:01 s06 systemd: Starting User Slice of root. >>>>> Mar 1 11:07:01 s06 systemd: Started Session 196232 of user root. >>>>> Mar 1 11:07:01 s06 systemd: Starting Session 196232 of user root. >>>>> Mar 1 11:07:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 1 11:07:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 1 11:07:36 s06 glustershd[70117]: [2019-03-01 10:07:36.366259] C >>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-0: server >>>>> 192.168.0.51:49152 has not responded in the last 42 seconds, >>>>> disconnecting. >>>>> Mar 1 11:08:01 s06 systemd: Created slice User Slice of root. >>>>> >>>>> No ?blocked? word is in /var/log/messages files on other cluster >>>>> servers. >>>>> In attachment, the /var/log/messages file from s06 server. >>>>> >>>>> Thank you in advance, >>>>> Mauro >>>>> >>>>> >>>>> >>>>> >>>>> On 1 Mar 2019, at 11:47, Milind Changire wrote: >>>>> >>>>> The traces of very high disk activity on the servers are often found >>>>> in /var/log/messages >>>>> You might want to grep for "blocked for" in /var/log/messages on s06 >>>>> and correlate the timestamps to confirm the unresponsiveness as reported in >>>>> gluster client logs. >>>>> In cases of high disk activity, although the operating system >>>>> continues to respond to ICMP pings, the processes writing to disks often >>>>> get blocked to a large flush to the disk which could span beyond 42 seconds >>>>> and hence result in ping-timer-expiry logs. >>>>> >>>>> As a side note: >>>>> If you indeed find gluster processes being blocked in >>>>> /var/log/messages, you might want to tweak sysctl tunables called >>>>> vm.dirty_background_ratio or vm.dirty_background_bytes to a smaller value >>>>> than the existing. Please read up more on those tunables before touching >>>>> the settings. >>>>> >>>>> >>>>> On Fri, Mar 1, 2019 at 4:06 PM Mauro Tridici >>>>> wrote: >>>>> >>>>>> >>>>>> Hi all, >>>>>> >>>>>> in attachment the client log captured after changing >>>>>> network.ping-timeout option. >>>>>> I noticed this error involving server 192.168.0.56 (s06) >>>>>> >>>>>> [2019-03-01 09:23:36.077287] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >>>>>> 0-tier2-client-71: changing ping timeout to 42 (from 0) >>>>>> [2019-03-01 09:23:36.078213] I >>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>> volfile,continuing >>>>>> [2019-03-01 09:23:36.078432] I >>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>> volfile,continuing >>>>>> [2019-03-01 09:23:36.092357] I >>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>> volfile,continuing >>>>>> [2019-03-01 09:23:36.094146] I >>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>> volfile,continuing >>>>>> [2019-03-01 10:06:24.708082] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-50: server >>>>>> 192.168.0.56:49156 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> >>>>>> I don?t know why it happens, s06 server seems to be reachable. >>>>>> >>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>> Trying 192.168.0.56... >>>>>> Connected to 192.168.0.56. >>>>>> Escape character is '^]'. >>>>>> ^CConnection closed by foreign host. >>>>>> [athena_login2][/users/home/sysm02/]> ping 192.168.0.56 >>>>>> PING 192.168.0.56 (192.168.0.56) 56(84) bytes of data. >>>>>> 64 bytes from 192.168.0.56: icmp_seq=1 ttl=64 time=0.116 ms >>>>>> 64 bytes from 192.168.0.56: icmp_seq=2 ttl=64 time=0.101 ms >>>>>> >>>>>> --- 192.168.0.56 ping statistics --- >>>>>> 2 packets transmitted, 2 received, 0% packet loss, time 1528ms >>>>>> rtt min/avg/max/mdev = 0.101/0.108/0.116/0.012 ms >>>>>> >>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>> Trying 192.168.0.56... >>>>>> Connected to 192.168.0.56. >>>>>> Escape character is '^]'. >>>>>> >>>>>> Thank you for your help, >>>>>> Mauro >>>>>> >>>>>> >>>>>> >>>>>> On 1 Mar 2019, at 10:29, Mauro Tridici wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> thank you for the explanation. >>>>>> I just changed network.ping-timeout option to default value >>>>>> (network.ping-timeout=42). >>>>>> >>>>>> I will check the logs to see if the errors will appear again. >>>>>> >>>>>> Regards, >>>>>> Mauro >>>>>> >>>>>> On 1 Mar 2019, at 04:43, Milind Changire wrote: >>>>>> >>>>>> network.ping-timeout should not be set to zero for non-glusterd >>>>>> clients. >>>>>> glusterd is a special case for which ping-timeout is set to zero via >>>>>> /etc/glusterfs/glusterd.vol >>>>>> >>>>>> Setting network.ping-timeout to zero disables arming of the ping >>>>>> timer for connections. This disables testing the connection for >>>>>> responsiveness and hence avoids proactive fail-over. >>>>>> >>>>>> Please reset network.ping-timeout to a non-zero positive value, eg. 42 >>>>>> >>>>>> >>>>>> On Thu, Feb 28, 2019 at 5:07 PM Nithya Balachandran < >>>>>> nbalacha at redhat.com> wrote: >>>>>> >>>>>>> Adding Raghavendra and Milind to comment on this. >>>>>>> >>>>>>> What is the effect of setting network.ping-timeout to 0 and should >>>>>>> it be set back to 42? >>>>>>> Regards, >>>>>>> Nithya >>>>>>> >>>>>>> On Thu, 28 Feb 2019 at 16:01, Mauro Tridici >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Nithya, >>>>>>>> >>>>>>>> sorry for the late. >>>>>>>> network.ping-timeout has been set to 0 in order to try to solve >>>>>>>> some timeout problems, but it didn?t help. >>>>>>>> I can set it to the default value. >>>>>>>> >>>>>>>> Can I proceed with the change? >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Mauro >>>>>>>> >>>>>>>> >>>>>>>> On 28 Feb 2019, at 04:41, Nithya Balachandran >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Mauro, >>>>>>>> >>>>>>>> Is network.ping-timeout still set to 0. The default value is 42. Is >>>>>>>> there a particular reason why this was changed? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Nithya >>>>>>>> >>>>>>>> >>>>>>>> On Wed, 27 Feb 2019 at 21:32, Mauro Tridici >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Hi Xavi, >>>>>>>>> >>>>>>>>> thank you for the detailed explanation and suggestions. >>>>>>>>> Yes, transport.listen-backlog option is still set to 1024. >>>>>>>>> >>>>>>>>> I will check the network and connectivity status using ?ping? and >>>>>>>>> ?telnet? as soon as the errors will come back again. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Mauro >>>>>>>>> >>>>>>>>> Il giorno 27 feb 2019, alle ore 16:42, Xavi Hernandez < >>>>>>>>> jahernan at redhat.com> ha scritto: >>>>>>>>> >>>>>>>>> Hi Mauro, >>>>>>>>> >>>>>>>>> those errors say that the mount point is not connected to some of >>>>>>>>> the bricks while executing operations. I see references to 3rd and 6th >>>>>>>>> bricks of several disperse sets, which seem to map to server s06. For some >>>>>>>>> reason, gluster is having troubles connecting from the client machine to >>>>>>>>> that particular server. At the end of the log I see that after long time a >>>>>>>>> reconnect is done to both of them. However little after, other bricks from >>>>>>>>> the s05 get disconnected and a reconnect times out. >>>>>>>>> >>>>>>>>> That's really odd. It seems like if server/communication is cut to >>>>>>>>> s06 for some time, then restored, and then the same happens to the next >>>>>>>>> server. >>>>>>>>> >>>>>>>>> If the servers are really online and it's only a communication >>>>>>>>> issue, it explains why server memory and network has increased: if the >>>>>>>>> problem only exists between the client and servers, any write made by the >>>>>>>>> client will automatically mark the file as damaged, since some of the >>>>>>>>> servers have not been updated. Since self-heal runs from the server nodes, >>>>>>>>> they will probably be correctly connected to all bricks, which allows them >>>>>>>>> to heal the just damaged file, which increases memory and network usage. >>>>>>>>> >>>>>>>>> I guess you still have transport.listen-backlog set to 1024, right >>>>>>>>> ? >>>>>>>>> >>>>>>>>> Just to try to identify if the problem really comes from network, >>>>>>>>> can you check if you lose some pings from the client to all of the servers >>>>>>>>> while you are seeing those errors in the log file ? >>>>>>>>> >>>>>>>>> You can also check if during those errors, you can telnet to the >>>>>>>>> port of the brick from the client. >>>>>>>>> >>>>>>>>> Xavi >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Feb 26, 2019 at 10:17 AM Mauro Tridici < >>>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>>> >>>>>>>>>> Hi Nithya, >>>>>>>>>> >>>>>>>>>> ?df -h? operation is not still slow, but no users are using the >>>>>>>>>> volume, RAM and NETWORK usage is ok on the client node. >>>>>>>>>> >>>>>>>>>> I was worried about this kind of warnings/errors: >>>>>>>>>> >>>>>>>>>> [2019-02-25 10:59:00.664323] W [MSGID: 122035] >>>>>>>>>> [ec-common.c:571:ec_child_select] 0-tier2-disperse-6: Executing operation >>>>>>>>>> with some subvolumes unavailable (20) >>>>>>>>>> >>>>>>>>>> [2019-02-26 03:11:35.212603] E >>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>> called at 2019-02-26 03:10:56.549903 (xid=0x106f1c5) >>>>>>>>>> >>>>>>>>>> [2019-02-26 03:13:03.313831] E >>>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-50: connection to >>>>>>>>>> 192.168.0.56:49156 failed (Timeout della connessione); >>>>>>>>>> disconnecting socket >>>>>>>>>> >>>>>>>>>> It seems that some subvolumes are not available and 192.168.0.56 >>>>>>>>>> server (s06) is not reachable. >>>>>>>>>> But gluster servers are up&running and bricks are ok. >>>>>>>>>> >>>>>>>>>> In attachment the updated tier2.log file. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thank you. >>>>>>>>>> Regards, >>>>>>>>>> Mauro >>>>>>>>>> >>>>>>>>>> Il giorno 26 feb 2019, alle ore 04:03, Nithya Balachandran < >>>>>>>>>> nbalacha at redhat.com> ha scritto: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I see a lot of EC messages in the log but they don't seem very >>>>>>>>>> serious. Xavi, can you take a look? >>>>>>>>>> >>>>>>>>>> The only errors I see are: >>>>>>>>>> [2019-02-25 10:58:45.519871] E >>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>> called at 2019-02-25 10:57:47.429969 (xid=0xd26fe7) >>>>>>>>>> [2019-02-25 10:58:51.461493] E >>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>> 0-tier2-client-41: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>> called at 2019-02-25 10:57:47.499174 (xid=0xf47d6a) >>>>>>>>>> [2019-02-25 11:07:57.152874] E >>>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-70: connection to >>>>>>>>>> 192.168.0.55:49163 failed (Timeout della connessione); >>>>>>>>>> disconnecting socket >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Is the df -h operation still slow? If yes, can you take a tcpdump >>>>>>>>>> of the client while running df -h and send that across? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Nithya >>>>>>>>>> >>>>>>>>>> On Mon, 25 Feb 2019 at 17:27, Mauro Tridici < >>>>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Sorry, some minutes after my last mail message, I noticed that >>>>>>>>>>> ?df -h? command hanged for a while before returns the prompt. >>>>>>>>>>> Yesterday, everything was ok in the gluster client log, but, >>>>>>>>>>> today, I see a lot of errors (please, take a look to the attached file). >>>>>>>>>>> >>>>>>>>>>> On the client node, I detected an important RAM e NETWORK usage. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Do you think that the errors have been caused by the client >>>>>>>>>>> resources usage? >>>>>>>>>>> >>>>>>>>>>> Thank you in advance, >>>>>>>>>>> Mauro >>>>>>>>>>> >>>>>>>>>>> >>>>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhishpaliwal at gmail.com Tue Mar 12 05:28:46 2019 From: abhishpaliwal at gmail.com (ABHISHEK PALIWAL) Date: Tue, 12 Mar 2019 10:58:46 +0530 Subject: [Gluster-devel] [Gluster-users] Glusterfsd crashed with SIGSEGV In-Reply-To: References: Message-ID: Hi Amar, Below are the requested logs pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libglusterfs.so not a dynamic executable pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libgfrpc.so not a dynamic executable root at 128:/# gdb /usr/sbin/glusterd core.1099 GNU gdb (GDB) 7.10.1 Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64-wrs-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/glusterd...(no debugging symbols found)...done. [New LWP 1109] [New LWP 1101] [New LWP 1105] [New LWP 1110] [New LWP 1099] [New LWP 1107] [New LWP 1119] [New LWP 1103] [New LWP 1112] [New LWP 1116] [New LWP 1104] [New LWP 1239] [New LWP 1106] [New LWP 1111] [New LWP 1108] [New LWP 1117] [New LWP 1102] [New LWP 1118] [New LWP 1100] [New LWP 1114] [New LWP 1113] [New LWP 1115] warning: Could not load shared library symbols for linux-vdso64.so.1. Do you need "set solib-search-path" or "set sysroot"? [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 --volfile-id gv0.128.224.95.140.tmp-bric'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, bytes=bytes at entry=36) at malloc.c:3327 3327 { [Current thread is 1 (Thread 0x3fffb1689160 (LWP 1109))] (gdb) bt full #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, bytes=bytes at entry=36) at malloc.c:3327 nb = idx = bin = victim = size = victim_index = remainder = remainder_size = block = bit = map = fwd = bck = errstr = 0x0 __func__ = "_int_malloc" #1 0x00003fffb76a93dc in __GI___libc_malloc (bytes=36) at malloc.c:2921 ar_ptr = 0x3fffa8000020 victim = hook = __func__ = "__libc_malloc" #2 0x00003fffb7764fd0 in x_inline (xdrs=0x3fffb1686d20, len=) at xdr_sizeof.c:89 len = 36 xdrs = 0x3fffb1686d20 #3 0x00003fffb7842488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #4 0x00003fffb7842e84 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #5 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa81099f0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa8109aa0 "\265\256\373\200\f\206\361j" stat = #6 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa81099f0, obj_size=, xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #7 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #8 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa8109870, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa8109920 "\232\373\377\315\352\325\005\271" stat = #9 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa8109870, obj_size=, xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #10 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #11 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa81096f0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa81097a0 "\241X\372!\216\256=\342" stat = ---Type to continue, or q to quit--- #12 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa81096f0, obj_size=, xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #13 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #14 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa8109570, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa8109620 "\265\205\003Vu'\002L" stat = #15 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa8109570, obj_size=, xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #16 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #17 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa81093f0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa81094a0 "\200L\027F'\177\366D" stat = #18 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa81093f0, obj_size=, xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #19 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #20 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa8109270, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa8109320 "\217{dK(\001E\220" stat = #21 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa8109270, obj_size=, xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #22 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #23 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa81090f0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa81091a0 "\217\275\067\336\232\300(\005" stat = #24 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa81090f0, obj_size=, xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #25 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #26 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa8108f70, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa8109020 "\260.\025\b\244\352IT" stat = #27 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, objpp=0x3fffa8108f70, obj_size=, xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #28 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #29 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, pp=0x3fffa8108df0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa8108ea0 "\212GS\203l\035\n\\" ---Type to continue, or q to quit--- Regards, Abhishek On Mon, Mar 11, 2019 at 7:10 PM Amar Tumballi Suryanarayan < atumball at redhat.com> wrote: > Hi Abhishek, > > Can you check and get back to us? > > ``` > bash# ldd /usr/lib64/libglusterfs.so > bash# ldd /usr/lib64/libgfrpc.so > > ``` > > Also considering you have the core, can you do `(gdb) thr apply all bt > full` and pass it on? > > Thanks & Regards, > Amar > > On Mon, Mar 11, 2019 at 3:41 PM ABHISHEK PALIWAL > wrote: > >> Hi Team, >> >> COuld you please provide some pointer to debug it further. >> >> Regards, >> Abhishek >> >> On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL >> wrote: >> >>> Hi Team, >>> >>> I am using Glusterfs 5.4, where after setting the gluster mount point >>> when trying to access it, glusterfsd is getting crashed and mount point >>> through the "Transport endpoint is not connected error. >>> >>> Here I are the gdb log for the core file >>> >>> warning: Could not load shared library symbols for linux-vdso64.so.1. >>> Do you need "set solib-search-path" or "set sysroot"? >>> [Thread debugging using libthread_db enabled] >>> Using host libthread_db library "/lib64/libthread_db.so.1". >>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>> Program terminated with signal SIGSEGV, Segmentation fault. >>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>> bytes=bytes at entry=36) at malloc.c:3327 >>> 3327 { >>> [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))] >>> (gdb) >>> (gdb) >>> (gdb) bt >>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>> bytes=bytes at entry=36) at malloc.c:3327 >>> #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at malloc.c:2921 >>> #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, len=>> out>) at xdr_sizeof.c:89 >>> #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0 >>> #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> #5 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>> pp=0x3fff7c132020, size=, proc=) at >>> xdr_ref.c:84 >>> #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>> objpp=0x3fff7c132020, obj_size=, >>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> #8 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>> pp=0x3fff7c131ea0, size=, proc=) at >>> xdr_ref.c:84 >>> #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>> objpp=0x3fff7c131ea0, obj_size=, >>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> #11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>> pp=0x3fff7c131d20, size=, proc=) at >>> xdr_ref.c:84 >>> #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>> objpp=0x3fff7c131d20, obj_size=, >>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> #14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>> pp=0x3fff7c131ba0, size=, proc=) at >>> xdr_ref.c:84 >>> #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>> objpp=0x3fff7c131ba0, obj_size=, >>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> #17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>> pp=0x3fff7c131a20, size=, proc=) at >>> xdr_ref.c:84 >>> #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>> objpp=0x3fff7c131a20, obj_size=, >>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> #20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>> pp=0x3fff7c1318a0, size=, proc=) at >>> xdr_ref.c:84 >>> #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>> objpp=0x3fff7c1318a0, obj_size=, >>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> #23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>> pp=0x3fff7c131720, size=, proc=) at >>> xdr_ref.c:84 >>> #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>> objpp=0x3fff7c131720, obj_size=, >>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> #26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>> pp=0x3fff7c1315a0, size=, proc=) at >>> xdr_ref.c:84 >>> #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>> objpp=0x3fff7c1315a0, obj_size=, >>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> #29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>> pp=0x3fff7c131420, size=, proc=) at >>> xdr_ref.c:84 >>> #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>> objpp=0x3fff7c131420, obj_size=, >>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> #32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>> pp=0x3fff7c1312a0, size=, proc=) at >>> xdr_ref.c:84 >>> #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>> objpp=0x3fff7c1312a0, obj_size=, >>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> >>> Frames are getting repeated, could any one please me. >>> -- >>> Regards >>> Abhishek Paliwal >>> >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > Amar Tumballi (amarts) > -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: From dkhandel at redhat.com Tue Mar 12 08:55:52 2019 From: dkhandel at redhat.com (Deepshikha Khandelwal) Date: Tue, 12 Mar 2019 14:25:52 +0530 Subject: [Gluster-devel] [Gluster-infra] Softserve is up and running Message-ID: Hello folks, Softserve is deployed back today with AWS stack to loan centos machines for regression testing. I've tested them a few times today to confirm it works as expected. In the past, Softserve[1] machines would be a clean Centos 7 image. Now, we have an AMI image with all the dependencies installed and *almost* setup to run regressions. It just needs a few steps run on them and we have a simplified playbook that will setup *just* those steps. The instructions are on softserve wiki[2] Please let us know if you face troubles by filing a bug.[3] [1]: https://softserve.gluster.org/ [2]: https://github.com/gluster/softserve /wiki/Running-Regressions-on-loaned-Softserve-instances [3]: https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS&component=project-infrastructure Thanks, Deepshikha -------------- next part -------------- An HTML attachment was scrubbed... URL: From ravishankar at redhat.com Tue Mar 12 09:30:27 2019 From: ravishankar at redhat.com (Ravishankar N) Date: Tue, 12 Mar 2019 15:00:27 +0530 Subject: [Gluster-devel] gfapi: add function to set client-pid Message-ID: Hello, I'm planning to expose setting client-pid for gfapi clients via a new api, something like `glfs_set_client_pid (fs, pid)`. The functionality already exists for fuse mounts via the --client-pid=$PID option, where the value is captured in glusterfs_ctx_t->cmd_args->client_pid. Background: If the glusterfs eventing framework is enabled, AFR sends child-up/child down events (via the gf_event() call) in the notify code path whenever there is a connect/disconnect at AFR level. While this is okay for normal client processes, it does not make much sense if the event is coming from say glfsheal, which is a gfapi based program (having the AFR xlator) that is invoked when you run the heal info set of commands. Many applications periodically run heal info to monitor the heals and display it on the dashboard (like tendryl), leading to a flood of child up/ down messages to the application monitoring these events. We need to add a unique key=value to all such gf_event() calls in AFR, based on which the consumer of the events can decide to ignore them if needed. This key-value can be client-pid=$PID, where PID can be GF_CLIENT_PID_SELF_HEALD for selfheal daemon, GF_CLIENT_PID_GLFS_HEAL for glfsheal etc (these values are already defined in the code). This is why we need a way to set the client-pid for gfapi clients as well. Another approach would be to add an xlator option (say 'client-name') specific to AFR? and use that as the key-value pair but it seems to be an overkill to do that just for the sake of eventing purposes. Besides, the pid approach can also be extended to other gluster processes like rebalance, shd and other daemons where AFR is loaded but AFR child-up/down events from it are not of any particular interest.These daemons will now have to be spawned by glusterd with the --client-pid option. Regards, Ravi From ndevos at redhat.com Tue Mar 12 10:50:41 2019 From: ndevos at redhat.com (Niels de Vos) Date: Tue, 12 Mar 2019 11:50:41 +0100 Subject: [Gluster-devel] gfapi: add function to set client-pid In-Reply-To: References: Message-ID: <20190312105041.GI3535@ndevos-x270> On Tue, Mar 12, 2019 at 03:00:27PM +0530, Ravishankar N wrote: > Hello, > > I'm planning to expose setting client-pid for gfapi clients via a new api, > something like `glfs_set_client_pid (fs, pid)`. > The functionality already exists for fuse mounts via the --client-pid=$PID > option, where the value is captured in > glusterfs_ctx_t->cmd_args->client_pid. > > Background: > > If the glusterfs eventing framework is enabled, AFR sends child-up/child > down events (via the gf_event() call) in the notify code path whenever there > is a connect/disconnect at AFR level. While this is okay for normal client > processes, it does not make much sense if the event is coming from say > glfsheal, which is a gfapi based program (having the AFR xlator) that is > invoked when you run the heal info set of commands. Many applications > periodically run heal info to monitor the heals and display it on the > dashboard (like tendryl), leading to a flood of child up/ down messages to > the application monitoring these events. > > We need to add a unique key=value to all such gf_event() calls in AFR, based > on which the consumer of the events can decide to ignore them if needed. > This key-value can be client-pid=$PID, where PID can be > GF_CLIENT_PID_SELF_HEALD for selfheal daemon, GF_CLIENT_PID_GLFS_HEAL for > glfsheal etc (these values are already defined in the code). This is why we > need a way to set the client-pid for gfapi clients as well. > > Another approach would be to add an xlator option (say 'client-name') > specific to AFR? and use that as the key-value pair but it seems to be an > overkill to do that just for the sake of eventing purposes. Besides, the pid > approach can also be extended to other gluster processes like rebalance, shd > and other daemons where AFR is loaded but AFR child-up/down events from it > are not of any particular interest.These daemons will now have to be spawned > by glusterd with the --client-pid option. Sounds good to me. This probably should be a function that is not available to all gfapi consumers, so please use api/src/glfs-internal.h for that. With clear documentation as written in the email, it should be obvious that only Gluster internal processes may use it. Adding the integration mailinglist on CC, as that is where all discussions around gfapi should be archived. Thanks, Niels From abhishpaliwal at gmail.com Wed Mar 13 04:00:08 2019 From: abhishpaliwal at gmail.com (ABHISHEK PALIWAL) Date: Wed, 13 Mar 2019 09:30:08 +0530 Subject: [Gluster-devel] [Gluster-users] Glusterfsd crashed with SIGSEGV In-Reply-To: References: Message-ID: Hi Amar, did you get time to check the logs? Regards, Abhishek On Tue, Mar 12, 2019 at 10:58 AM ABHISHEK PALIWAL wrote: > Hi Amar, > > Below are the requested logs > > pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libglusterfs.so > not a dynamic executable > > pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libgfrpc.so > not a dynamic executable > > root at 128:/# gdb /usr/sbin/glusterd core.1099 > GNU gdb (GDB) 7.10.1 > Copyright (C) 2015 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later < > http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "powerpc64-wrs-linux". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > . > Find the GDB manual and other documentation resources online at: > . > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from /usr/sbin/glusterd...(no debugging symbols > found)...done. > [New LWP 1109] > [New LWP 1101] > [New LWP 1105] > [New LWP 1110] > [New LWP 1099] > [New LWP 1107] > [New LWP 1119] > [New LWP 1103] > [New LWP 1112] > [New LWP 1116] > [New LWP 1104] > [New LWP 1239] > [New LWP 1106] > [New LWP 1111] > [New LWP 1108] > [New LWP 1117] > [New LWP 1102] > [New LWP 1118] > [New LWP 1100] > [New LWP 1114] > [New LWP 1113] > [New LWP 1115] > > warning: Could not load shared library symbols for linux-vdso64.so.1. > Do you need "set solib-search-path" or "set sysroot"? > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 --volfile-id > gv0.128.224.95.140.tmp-bric'. > Program terminated with signal SIGSEGV, Segmentation fault. > #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, > bytes=bytes at entry=36) at malloc.c:3327 > 3327 { > [Current thread is 1 (Thread 0x3fffb1689160 (LWP 1109))] > (gdb) bt full > #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, > bytes=bytes at entry=36) at malloc.c:3327 > nb = > idx = > bin = > victim = > size = > victim_index = > remainder = > remainder_size = > block = > bit = > map = > fwd = > bck = > errstr = 0x0 > __func__ = "_int_malloc" > #1 0x00003fffb76a93dc in __GI___libc_malloc (bytes=36) at malloc.c:2921 > ar_ptr = 0x3fffa8000020 > victim = > hook = > __func__ = "__libc_malloc" > #2 0x00003fffb7764fd0 in x_inline (xdrs=0x3fffb1686d20, len= out>) at xdr_sizeof.c:89 > len = 36 > xdrs = 0x3fffb1686d20 > #3 0x00003fffb7842488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0 > No symbol table info available. > #4 0x00003fffb7842e84 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > No symbol table info available. > #5 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, > pp=0x3fffa81099f0, size=, proc=) at > xdr_ref.c:84 > loc = 0x3fffa8109aa0 "\265\256\373\200\f\206\361j" > stat = > #6 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, > objpp=0x3fffa81099f0, obj_size=, > xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > more_data = 1 > #7 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > No symbol table info available. > #8 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, > pp=0x3fffa8109870, size=, proc=) at > xdr_ref.c:84 > loc = 0x3fffa8109920 "\232\373\377\315\352\325\005\271" > stat = > #9 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, > objpp=0x3fffa8109870, obj_size=, > xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > more_data = 1 > #10 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > No symbol table info available. > #11 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, > pp=0x3fffa81096f0, size=, proc=) at > xdr_ref.c:84 > loc = 0x3fffa81097a0 "\241X\372!\216\256=\342" > stat = > ---Type to continue, or q to quit--- > #12 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, > objpp=0x3fffa81096f0, obj_size=, > xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > more_data = 1 > #13 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > No symbol table info available. > #14 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, > pp=0x3fffa8109570, size=, proc=) at > xdr_ref.c:84 > loc = 0x3fffa8109620 "\265\205\003Vu'\002L" > stat = > #15 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, > objpp=0x3fffa8109570, obj_size=, > xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > more_data = 1 > #16 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > No symbol table info available. > #17 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, > pp=0x3fffa81093f0, size=, proc=) at > xdr_ref.c:84 > loc = 0x3fffa81094a0 "\200L\027F'\177\366D" > stat = > #18 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, > objpp=0x3fffa81093f0, obj_size=, > xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > more_data = 1 > #19 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > No symbol table info available. > #20 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, > pp=0x3fffa8109270, size=, proc=) at > xdr_ref.c:84 > loc = 0x3fffa8109320 "\217{dK(\001E\220" > stat = > #21 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, > objpp=0x3fffa8109270, obj_size=, > xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > more_data = 1 > #22 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > No symbol table info available. > #23 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, > pp=0x3fffa81090f0, size=, proc=) at > xdr_ref.c:84 > loc = 0x3fffa81091a0 "\217\275\067\336\232\300(\005" > stat = > #24 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, > objpp=0x3fffa81090f0, obj_size=, > xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > more_data = 1 > #25 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > No symbol table info available. > #26 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, > pp=0x3fffa8108f70, size=, proc=) at > xdr_ref.c:84 > loc = 0x3fffa8109020 "\260.\025\b\244\352IT" > stat = > #27 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, > objpp=0x3fffa8108f70, obj_size=, > xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at > xdr_ref.c:135 > more_data = 1 > #28 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from > /usr/lib64/libgfxdr.so.0 > No symbol table info available. > #29 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, > pp=0x3fffa8108df0, size=, proc=) at > xdr_ref.c:84 > loc = 0x3fffa8108ea0 "\212GS\203l\035\n\\" > ---Type to continue, or q to quit--- > > > Regards, > Abhishek > > On Mon, Mar 11, 2019 at 7:10 PM Amar Tumballi Suryanarayan < > atumball at redhat.com> wrote: > >> Hi Abhishek, >> >> Can you check and get back to us? >> >> ``` >> bash# ldd /usr/lib64/libglusterfs.so >> bash# ldd /usr/lib64/libgfrpc.so >> >> ``` >> >> Also considering you have the core, can you do `(gdb) thr apply all bt >> full` and pass it on? >> >> Thanks & Regards, >> Amar >> >> On Mon, Mar 11, 2019 at 3:41 PM ABHISHEK PALIWAL >> wrote: >> >>> Hi Team, >>> >>> COuld you please provide some pointer to debug it further. >>> >>> Regards, >>> Abhishek >>> >>> On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL >>> wrote: >>> >>>> Hi Team, >>>> >>>> I am using Glusterfs 5.4, where after setting the gluster mount point >>>> when trying to access it, glusterfsd is getting crashed and mount point >>>> through the "Transport endpoint is not connected error. >>>> >>>> Here I are the gdb log for the core file >>>> >>>> warning: Could not load shared library symbols for linux-vdso64.so.1. >>>> Do you need "set solib-search-path" or "set sysroot"? >>>> [Thread debugging using libthread_db enabled] >>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>> bytes=bytes at entry=36) at malloc.c:3327 >>>> 3327 { >>>> [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))] >>>> (gdb) >>>> (gdb) >>>> (gdb) bt >>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>> bytes=bytes at entry=36) at malloc.c:3327 >>>> #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at malloc.c:2921 >>>> #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, len=>>> out>) at xdr_sizeof.c:89 >>>> #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from >>>> /usr/lib64/libgfxdr.so.0 >>>> #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> #5 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>> pp=0x3fff7c132020, size=, proc=) at >>>> xdr_ref.c:84 >>>> #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>> objpp=0x3fff7c132020, obj_size=, >>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> #8 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>> pp=0x3fff7c131ea0, size=, proc=) at >>>> xdr_ref.c:84 >>>> #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>> objpp=0x3fff7c131ea0, obj_size=, >>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> #11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>> pp=0x3fff7c131d20, size=, proc=) at >>>> xdr_ref.c:84 >>>> #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>> objpp=0x3fff7c131d20, obj_size=, >>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> #14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>> pp=0x3fff7c131ba0, size=, proc=) at >>>> xdr_ref.c:84 >>>> #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>> objpp=0x3fff7c131ba0, obj_size=, >>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> #17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>> pp=0x3fff7c131a20, size=, proc=) at >>>> xdr_ref.c:84 >>>> #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>> objpp=0x3fff7c131a20, obj_size=, >>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> #20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>> pp=0x3fff7c1318a0, size=, proc=) at >>>> xdr_ref.c:84 >>>> #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>> objpp=0x3fff7c1318a0, obj_size=, >>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> #23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>> pp=0x3fff7c131720, size=, proc=) at >>>> xdr_ref.c:84 >>>> #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>> objpp=0x3fff7c131720, obj_size=, >>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> #26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>> pp=0x3fff7c1315a0, size=, proc=) at >>>> xdr_ref.c:84 >>>> #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>> objpp=0x3fff7c1315a0, obj_size=, >>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> #29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>> pp=0x3fff7c131420, size=, proc=) at >>>> xdr_ref.c:84 >>>> #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>> objpp=0x3fff7c131420, obj_size=, >>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> #32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>> pp=0x3fff7c1312a0, size=, proc=) at >>>> xdr_ref.c:84 >>>> #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>> objpp=0x3fff7c1312a0, obj_size=, >>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> >>>> Frames are getting repeated, could any one please me. >>>> -- >>>> Regards >>>> Abhishek Paliwal >>>> >>> >>> >>> -- >>> >>> >>> >>> >>> Regards >>> Abhishek Paliwal >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> -- >> Amar Tumballi (amarts) >> > > > -- > > > > > Regards > Abhishek Paliwal > -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: From atumball at redhat.com Wed Mar 13 04:20:00 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Wed, 13 Mar 2019 09:50:00 +0530 Subject: [Gluster-devel] [Gluster-users] Glusterfsd crashed with SIGSEGV In-Reply-To: References: Message-ID: Hi Abhishek, Few more questions, > On Tue, Mar 12, 2019 at 10:58 AM ABHISHEK PALIWAL > wrote: > >> Hi Amar, >> >> Below are the requested logs >> >> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libglusterfs.so >> not a dynamic executable >> >> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libgfrpc.so >> not a dynamic executable >> >> Can you please add a * at the end, so it gets the linked library list from the actual files (ideally this is a symlink, but I expected it to resolve like in Fedora). > root at 128:/# gdb /usr/sbin/glusterd core.1099 >> GNU gdb (GDB) 7.10.1 >> Copyright (C) 2015 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later < >> http://gnu.org/licenses/gpl.html> >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "powerpc64-wrs-linux". >> Type "show configuration" for configuration details. >> For bug reporting instructions, please see: >> . >> Find the GDB manual and other documentation resources online at: >> . >> For help, type "help". >> Type "apropos word" to search for commands related to "word"... >> Reading symbols from /usr/sbin/glusterd...(no debugging symbols >> found)...done. >> [New LWP 1109] >> [New LWP 1101] >> [New LWP 1105] >> [New LWP 1110] >> [New LWP 1099] >> [New LWP 1107] >> [New LWP 1119] >> [New LWP 1103] >> [New LWP 1112] >> [New LWP 1116] >> [New LWP 1104] >> [New LWP 1239] >> [New LWP 1106] >> [New LWP 1111] >> [New LWP 1108] >> [New LWP 1117] >> [New LWP 1102] >> [New LWP 1118] >> [New LWP 1100] >> [New LWP 1114] >> [New LWP 1113] >> [New LWP 1115] >> >> warning: Could not load shared library symbols for linux-vdso64.so.1. >> Do you need "set solib-search-path" or "set sysroot"? >> [Thread debugging using libthread_db enabled] >> Using host libthread_db library "/lib64/libthread_db.so.1". >> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >> --volfile-id gv0.128.224.95.140.tmp-bric'. >> Program terminated with signal SIGSEGV, Segmentation fault. >> #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >> bytes=bytes at entry=36) at malloc.c:3327 >> 3327 { >> [Current thread is 1 (Thread 0x3fffb1689160 (LWP 1109))] >> (gdb) bt full >> > This is backtrace of one particular thread. I need output of command (gdb) thread apply all bt full Also, considering this is a crash in the malloc library call itself, would like to know the details of OS, Kernel version and gcc versions. Regards, Amar #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >> bytes=bytes at entry=36) at malloc.c:3327 >> nb = >> idx = >> bin = >> victim = >> size = >> victim_index = >> remainder = >> remainder_size = >> block = >> bit = >> map = >> fwd = >> bck = >> errstr = 0x0 >> __func__ = "_int_malloc" >> #1 0x00003fffb76a93dc in __GI___libc_malloc (bytes=36) at malloc.c:2921 >> ar_ptr = 0x3fffa8000020 >> victim = >> hook = >> __func__ = "__libc_malloc" >> #2 0x00003fffb7764fd0 in x_inline (xdrs=0x3fffb1686d20, len=> out>) at xdr_sizeof.c:89 >> len = 36 >> xdrs = 0x3fffb1686d20 >> #3 0x00003fffb7842488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0 >> No symbol table info available. >> #4 0x00003fffb7842e84 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> No symbol table info available. >> #5 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >> pp=0x3fffa81099f0, size=, proc=) at >> xdr_ref.c:84 >> loc = 0x3fffa8109aa0 "\265\256\373\200\f\206\361j" >> stat = >> #6 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >> objpp=0x3fffa81099f0, obj_size=, >> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> more_data = 1 >> #7 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> No symbol table info available. >> #8 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >> pp=0x3fffa8109870, size=, proc=) at >> xdr_ref.c:84 >> loc = 0x3fffa8109920 "\232\373\377\315\352\325\005\271" >> stat = >> #9 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >> objpp=0x3fffa8109870, obj_size=, >> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> more_data = 1 >> #10 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> No symbol table info available. >> #11 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >> pp=0x3fffa81096f0, size=, proc=) at >> xdr_ref.c:84 >> loc = 0x3fffa81097a0 "\241X\372!\216\256=\342" >> stat = >> ---Type to continue, or q to quit--- >> #12 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >> objpp=0x3fffa81096f0, obj_size=, >> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> more_data = 1 >> #13 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> No symbol table info available. >> #14 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >> pp=0x3fffa8109570, size=, proc=) at >> xdr_ref.c:84 >> loc = 0x3fffa8109620 "\265\205\003Vu'\002L" >> stat = >> #15 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >> objpp=0x3fffa8109570, obj_size=, >> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> more_data = 1 >> #16 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> No symbol table info available. >> #17 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >> pp=0x3fffa81093f0, size=, proc=) at >> xdr_ref.c:84 >> loc = 0x3fffa81094a0 "\200L\027F'\177\366D" >> stat = >> #18 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >> objpp=0x3fffa81093f0, obj_size=, >> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> more_data = 1 >> #19 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> No symbol table info available. >> #20 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >> pp=0x3fffa8109270, size=, proc=) at >> xdr_ref.c:84 >> loc = 0x3fffa8109320 "\217{dK(\001E\220" >> stat = >> #21 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >> objpp=0x3fffa8109270, obj_size=, >> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> more_data = 1 >> #22 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> No symbol table info available. >> #23 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >> pp=0x3fffa81090f0, size=, proc=) at >> xdr_ref.c:84 >> loc = 0x3fffa81091a0 "\217\275\067\336\232\300(\005" >> stat = >> #24 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >> objpp=0x3fffa81090f0, obj_size=, >> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> more_data = 1 >> #25 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> No symbol table info available. >> #26 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >> pp=0x3fffa8108f70, size=, proc=) at >> xdr_ref.c:84 >> loc = 0x3fffa8109020 "\260.\025\b\244\352IT" >> stat = >> #27 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >> objpp=0x3fffa8108f70, obj_size=, >> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >> xdr_ref.c:135 >> more_data = 1 >> #28 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >> /usr/lib64/libgfxdr.so.0 >> No symbol table info available. >> #29 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >> pp=0x3fffa8108df0, size=, proc=) at >> xdr_ref.c:84 >> loc = 0x3fffa8108ea0 "\212GS\203l\035\n\\" >> ---Type to continue, or q to quit--- >> >> >> Regards, >> Abhishek >> >> On Mon, Mar 11, 2019 at 7:10 PM Amar Tumballi Suryanarayan < >> atumball at redhat.com> wrote: >> >>> Hi Abhishek, >>> >>> Can you check and get back to us? >>> >>> ``` >>> bash# ldd /usr/lib64/libglusterfs.so >>> bash# ldd /usr/lib64/libgfrpc.so >>> >>> ``` >>> >>> Also considering you have the core, can you do `(gdb) thr apply all bt >>> full` and pass it on? >>> >>> Thanks & Regards, >>> Amar >>> >>> On Mon, Mar 11, 2019 at 3:41 PM ABHISHEK PALIWAL < >>> abhishpaliwal at gmail.com> wrote: >>> >>>> Hi Team, >>>> >>>> COuld you please provide some pointer to debug it further. >>>> >>>> Regards, >>>> Abhishek >>>> >>>> On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL < >>>> abhishpaliwal at gmail.com> wrote: >>>> >>>>> Hi Team, >>>>> >>>>> I am using Glusterfs 5.4, where after setting the gluster mount point >>>>> when trying to access it, glusterfsd is getting crashed and mount point >>>>> through the "Transport endpoint is not connected error. >>>>> >>>>> Here I are the gdb log for the core file >>>>> >>>>> warning: Could not load shared library symbols for linux-vdso64.so.1. >>>>> Do you need "set solib-search-path" or "set sysroot"? >>>>> [Thread debugging using libthread_db enabled] >>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>>>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>> 3327 { >>>>> [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))] >>>>> (gdb) >>>>> (gdb) >>>>> (gdb) bt >>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>> #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at >>>>> malloc.c:2921 >>>>> #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, >>>>> len=) at xdr_sizeof.c:89 >>>>> #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> #5 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>> pp=0x3fff7c132020, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>> objpp=0x3fff7c132020, obj_size=, >>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> #8 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>> pp=0x3fff7c131ea0, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>> objpp=0x3fff7c131ea0, obj_size=, >>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> #11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>> pp=0x3fff7c131d20, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>> objpp=0x3fff7c131d20, obj_size=, >>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> #14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>> pp=0x3fff7c131ba0, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>> objpp=0x3fff7c131ba0, obj_size=, >>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> #17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>> pp=0x3fff7c131a20, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>> objpp=0x3fff7c131a20, obj_size=, >>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> #20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>> pp=0x3fff7c1318a0, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>> objpp=0x3fff7c1318a0, obj_size=, >>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> #23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>> pp=0x3fff7c131720, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>> objpp=0x3fff7c131720, obj_size=, >>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> #26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>> pp=0x3fff7c1315a0, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>> objpp=0x3fff7c1315a0, obj_size=, >>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> #29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>> pp=0x3fff7c131420, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>> objpp=0x3fff7c131420, obj_size=, >>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> #32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>> pp=0x3fff7c1312a0, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>> objpp=0x3fff7c1312a0, obj_size=, >>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> >>>>> Frames are getting repeated, could any one please me. >>>>> -- >>>>> Regards >>>>> Abhishek Paliwal >>>>> >>>> >>>> >>>> -- >>>> >>>> >>>> >>>> >>>> Regards >>>> Abhishek Paliwal >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >>> >>> >>> -- >>> Amar Tumballi (amarts) >>> >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> > > > -- > > > > > Regards > Abhishek Paliwal > -- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhishpaliwal at gmail.com Wed Mar 13 04:32:51 2019 From: abhishpaliwal at gmail.com (ABHISHEK PALIWAL) Date: Wed, 13 Mar 2019 10:02:51 +0530 Subject: [Gluster-devel] [Gluster-users] Glusterfsd crashed with SIGSEGV In-Reply-To: References: Message-ID: Here are the logs: pabhishe at arn-build3$ldd ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.* ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0: not a dynamic executable ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0.0.1: not a dynamic executable pabhishe at arn-build3$ldd ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0.0.1 not a dynamic executable For backtraces I have attached the core_logs.txt file. Regards, Abhishek On Wed, Mar 13, 2019 at 9:51 AM Amar Tumballi Suryanarayan < atumball at redhat.com> wrote: > Hi Abhishek, > > Few more questions, > > >> On Tue, Mar 12, 2019 at 10:58 AM ABHISHEK PALIWAL < >> abhishpaliwal at gmail.com> wrote: >> >>> Hi Amar, >>> >>> Below are the requested logs >>> >>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libglusterfs.so >>> not a dynamic executable >>> >>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libgfrpc.so >>> not a dynamic executable >>> >>> > Can you please add a * at the end, so it gets the linked library list from > the actual files (ideally this is a symlink, but I expected it to resolve > like in Fedora). > > > >> root at 128:/# gdb /usr/sbin/glusterd core.1099 >>> GNU gdb (GDB) 7.10.1 >>> Copyright (C) 2015 Free Software Foundation, Inc. >>> License GPLv3+: GNU GPL version 3 or later < >>> http://gnu.org/licenses/gpl.html> >>> This is free software: you are free to change and redistribute it. >>> There is NO WARRANTY, to the extent permitted by law. Type "show >>> copying" >>> and "show warranty" for details. >>> This GDB was configured as "powerpc64-wrs-linux". >>> Type "show configuration" for configuration details. >>> For bug reporting instructions, please see: >>> . >>> Find the GDB manual and other documentation resources online at: >>> . >>> For help, type "help". >>> Type "apropos word" to search for commands related to "word"... >>> Reading symbols from /usr/sbin/glusterd...(no debugging symbols >>> found)...done. >>> [New LWP 1109] >>> [New LWP 1101] >>> [New LWP 1105] >>> [New LWP 1110] >>> [New LWP 1099] >>> [New LWP 1107] >>> [New LWP 1119] >>> [New LWP 1103] >>> [New LWP 1112] >>> [New LWP 1116] >>> [New LWP 1104] >>> [New LWP 1239] >>> [New LWP 1106] >>> [New LWP 1111] >>> [New LWP 1108] >>> [New LWP 1117] >>> [New LWP 1102] >>> [New LWP 1118] >>> [New LWP 1100] >>> [New LWP 1114] >>> [New LWP 1113] >>> [New LWP 1115] >>> >>> warning: Could not load shared library symbols for linux-vdso64.so.1. >>> Do you need "set solib-search-path" or "set sysroot"? >>> [Thread debugging using libthread_db enabled] >>> Using host libthread_db library "/lib64/libthread_db.so.1". >>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>> Program terminated with signal SIGSEGV, Segmentation fault. >>> #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >>> bytes=bytes at entry=36) at malloc.c:3327 >>> 3327 { >>> [Current thread is 1 (Thread 0x3fffb1689160 (LWP 1109))] >>> (gdb) bt full >>> >> > This is backtrace of one particular thread. I need output of command > > (gdb) thread apply all bt full > > > Also, considering this is a crash in the malloc library call itself, would > like to know the details of OS, Kernel version and gcc versions. > > Regards, > Amar > > #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >>> bytes=bytes at entry=36) at malloc.c:3327 >>> nb = >>> idx = >>> bin = >>> victim = >>> size = >>> victim_index = >>> remainder = >>> remainder_size = >>> block = >>> bit = >>> map = >>> fwd = >>> bck = >>> errstr = 0x0 >>> __func__ = "_int_malloc" >>> #1 0x00003fffb76a93dc in __GI___libc_malloc (bytes=36) at malloc.c:2921 >>> ar_ptr = 0x3fffa8000020 >>> victim = >>> hook = >>> __func__ = "__libc_malloc" >>> #2 0x00003fffb7764fd0 in x_inline (xdrs=0x3fffb1686d20, len=>> out>) at xdr_sizeof.c:89 >>> len = 36 >>> xdrs = 0x3fffb1686d20 >>> #3 0x00003fffb7842488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0 >>> No symbol table info available. >>> #4 0x00003fffb7842e84 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> No symbol table info available. >>> #5 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>> pp=0x3fffa81099f0, size=, proc=) at >>> xdr_ref.c:84 >>> loc = 0x3fffa8109aa0 "\265\256\373\200\f\206\361j" >>> stat = >>> #6 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>> objpp=0x3fffa81099f0, obj_size=, >>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> more_data = 1 >>> #7 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> No symbol table info available. >>> #8 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>> pp=0x3fffa8109870, size=, proc=) at >>> xdr_ref.c:84 >>> loc = 0x3fffa8109920 "\232\373\377\315\352\325\005\271" >>> stat = >>> #9 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>> objpp=0x3fffa8109870, obj_size=, >>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> more_data = 1 >>> #10 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> No symbol table info available. >>> #11 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>> pp=0x3fffa81096f0, size=, proc=) at >>> xdr_ref.c:84 >>> loc = 0x3fffa81097a0 "\241X\372!\216\256=\342" >>> stat = >>> ---Type to continue, or q to quit--- >>> #12 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>> objpp=0x3fffa81096f0, obj_size=, >>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> more_data = 1 >>> #13 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> No symbol table info available. >>> #14 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>> pp=0x3fffa8109570, size=, proc=) at >>> xdr_ref.c:84 >>> loc = 0x3fffa8109620 "\265\205\003Vu'\002L" >>> stat = >>> #15 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>> objpp=0x3fffa8109570, obj_size=, >>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> more_data = 1 >>> #16 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> No symbol table info available. >>> #17 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>> pp=0x3fffa81093f0, size=, proc=) at >>> xdr_ref.c:84 >>> loc = 0x3fffa81094a0 "\200L\027F'\177\366D" >>> stat = >>> #18 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>> objpp=0x3fffa81093f0, obj_size=, >>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> more_data = 1 >>> #19 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> No symbol table info available. >>> #20 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>> pp=0x3fffa8109270, size=, proc=) at >>> xdr_ref.c:84 >>> loc = 0x3fffa8109320 "\217{dK(\001E\220" >>> stat = >>> #21 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>> objpp=0x3fffa8109270, obj_size=, >>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> more_data = 1 >>> #22 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> No symbol table info available. >>> #23 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>> pp=0x3fffa81090f0, size=, proc=) at >>> xdr_ref.c:84 >>> loc = 0x3fffa81091a0 "\217\275\067\336\232\300(\005" >>> stat = >>> #24 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>> objpp=0x3fffa81090f0, obj_size=, >>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> more_data = 1 >>> #25 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> No symbol table info available. >>> #26 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>> pp=0x3fffa8108f70, size=, proc=) at >>> xdr_ref.c:84 >>> loc = 0x3fffa8109020 "\260.\025\b\244\352IT" >>> stat = >>> #27 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>> objpp=0x3fffa8108f70, obj_size=, >>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>> xdr_ref.c:135 >>> more_data = 1 >>> #28 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>> /usr/lib64/libgfxdr.so.0 >>> No symbol table info available. >>> #29 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>> pp=0x3fffa8108df0, size=, proc=) at >>> xdr_ref.c:84 >>> loc = 0x3fffa8108ea0 "\212GS\203l\035\n\\" >>> ---Type to continue, or q to quit--- >>> >>> >>> Regards, >>> Abhishek >>> >>> On Mon, Mar 11, 2019 at 7:10 PM Amar Tumballi Suryanarayan < >>> atumball at redhat.com> wrote: >>> >>>> Hi Abhishek, >>>> >>>> Can you check and get back to us? >>>> >>>> ``` >>>> bash# ldd /usr/lib64/libglusterfs.so >>>> bash# ldd /usr/lib64/libgfrpc.so >>>> >>>> ``` >>>> >>>> Also considering you have the core, can you do `(gdb) thr apply all bt >>>> full` and pass it on? >>>> >>>> Thanks & Regards, >>>> Amar >>>> >>>> On Mon, Mar 11, 2019 at 3:41 PM ABHISHEK PALIWAL < >>>> abhishpaliwal at gmail.com> wrote: >>>> >>>>> Hi Team, >>>>> >>>>> COuld you please provide some pointer to debug it further. >>>>> >>>>> Regards, >>>>> Abhishek >>>>> >>>>> On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL < >>>>> abhishpaliwal at gmail.com> wrote: >>>>> >>>>>> Hi Team, >>>>>> >>>>>> I am using Glusterfs 5.4, where after setting the gluster mount point >>>>>> when trying to access it, glusterfsd is getting crashed and mount point >>>>>> through the "Transport endpoint is not connected error. >>>>>> >>>>>> Here I are the gdb log for the core file >>>>>> >>>>>> warning: Could not load shared library symbols for linux-vdso64.so.1. >>>>>> Do you need "set solib-search-path" or "set sysroot"? >>>>>> [Thread debugging using libthread_db enabled] >>>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>>>>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>>>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>> 3327 { >>>>>> [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))] >>>>>> (gdb) >>>>>> (gdb) >>>>>> (gdb) bt >>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>> #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at >>>>>> malloc.c:2921 >>>>>> #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, >>>>>> len=) at xdr_sizeof.c:89 >>>>>> #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> #5 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>> pp=0x3fff7c132020, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>> objpp=0x3fff7c132020, obj_size=, >>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> #8 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>> pp=0x3fff7c131ea0, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>> objpp=0x3fff7c131ea0, obj_size=, >>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> #11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>> pp=0x3fff7c131d20, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>> objpp=0x3fff7c131d20, obj_size=, >>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> #14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>> pp=0x3fff7c131ba0, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>> objpp=0x3fff7c131ba0, obj_size=, >>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> #17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>> pp=0x3fff7c131a20, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>> objpp=0x3fff7c131a20, obj_size=, >>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> #20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>> pp=0x3fff7c1318a0, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>> objpp=0x3fff7c1318a0, obj_size=, >>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> #23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>> pp=0x3fff7c131720, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>> objpp=0x3fff7c131720, obj_size=, >>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> #26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>> pp=0x3fff7c1315a0, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>> objpp=0x3fff7c1315a0, obj_size=, >>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> #29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>> pp=0x3fff7c131420, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>> objpp=0x3fff7c131420, obj_size=, >>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> #32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>> pp=0x3fff7c1312a0, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>> objpp=0x3fff7c1312a0, obj_size=, >>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> >>>>>> Frames are getting repeated, could any one please me. >>>>>> -- >>>>>> Regards >>>>>> Abhishek Paliwal >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> >>>>> >>>>> >>>>> Regards >>>>> Abhishek Paliwal >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>>> >>>> >>>> -- >>>> Amar Tumballi (amarts) >>>> >>> >>> >>> -- >>> >>> >>> >>> >>> Regards >>> Abhishek Paliwal >>> >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> > > > -- > Amar Tumballi (amarts) > -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- warning: Could not load shared library symbols for linux-vdso64.so.1. Do you need "set solib-search-path" or "set sysroot"? [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 --volfile-id gv0.128.224.95.140.tmp-bric'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00003fffb3570d48 in _int_malloc (av=av at entry=0x3fffa4000020, bytes=bytes at entry=36) at malloc.c:3327 3327 { [Current thread is 1 (Thread 0x3fffad553160 (LWP 554))] (gdb) thread apply all bt full Thread 23 (Thread 0x3fffad513160 (LWP 555)): #0 0x00003fffb36ab7b0 in __pthread_cond_wait (cond=0x3fffa8074ff8, mutex=0x3fffa8074fd0) at pthread_cond_wait.c:186 r4 = 128 r7 = 2 r5 = 1 r8 = 2 arg3 = 1 r0 = 221 r3 = 512 r6 = 0 arg4 = 0 __err = __ret = futex_val = 1 buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fffad5125e0, __canceltype = 16383, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x3fffa8074ff8, mutex = 0x3fffa8074fd0, bc_seq = 0} err = pshared = 0 pi_flag = 0 val = seq = 0 #1 0x00003fffaf2762ec in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/bitrot-stub.so No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fffad513160) at pthread_create.c:462 pd = 0x3fffad513160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {8589934592, 70367268262048, 70367268262048, 70367268267296, 3419478345679794793, 7163871753673912110, 7452460622125822566, 8299977387227111790, 8388357161025011712, 0 }, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 22 (Thread 0x3fffac80f160 (LWP 557)): #0 0x00003fffb36ab7b0 in __pthread_cond_wait (cond=0x3fffa807a298, mutex=0x3fffa807a270) at pthread_cond_wait.c:186 r4 = 128 r7 = 2 r5 = 1 r8 = 2 arg3 = 1 r0 = 221 ---Type to continue, or q to quit--- r3 = 512 r6 = 0 arg4 = 0 __err = __ret = futex_val = 1 buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fffac80e560, __canceltype = 0, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x3fffa807a298, mutex = 0x3fffa807a270, bc_seq = 0} err = pshared = 0 pi_flag = 0 val = seq = 0 #1 0x00003fffaf2b4a9c in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/changelog.so No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fffac80f160) at pthread_create.c:462 pd = 0x3fffac80f160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {70367268283192, 70367268283192, 70367268283808, 1, 0, -4995072473058770944, 149, 206158430208, 64, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 88, 64, 70367388068184, 0, 70367268267328, 70367268288960, 268615696, 70367461387880, -4995072473058770944, 101, 171798691840, 15, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 3419478345679794793, 7163871753673900218, -5913212017086300160, 133, 171798691840, 37, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 3419478345679794793, 7163871753673912110, 7452460622125822566, 8299961938229487461, 7813577622142234096, 936748722493063168, 0, 133, 695784701952, 40, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0xbaadf00d00000000, 0x0, 0x95, 0x8100000000}, data = {prev = 0xbaadf00d00000000, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 21 (Thread 0x3fffacd13160 (LWP 556)): #0 0x00003fffb36ab7b0 in __pthread_cond_wait (cond=0x3fffa8075070, mutex=0x3fffa8075048) at pthread_cond_wait.c:186 r4 = 128 r7 = 2 r5 = 1 r8 = 2 arg3 = 1 r0 = 221 r3 = 512 r6 = 0 arg4 = 0 __err = __ret = futex_val = 1 ---Type to continue, or q to quit--- buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fffacd125e0, __canceltype = 16383, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x3fffa8075070, mutex = 0x3fffa8075048, bc_seq = 0} err = pshared = 0 pi_flag = 0 val = seq = 0 #1 0x00003fffaf27385c in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/bitrot-stub.so No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fffacd13160) at pthread_create.c:462 pd = 0x3fffacd13160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0 }, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 20 (Thread 0x3fffaef53160 (LWP 551)): #0 0x00003fffb36ab7b0 in __pthread_cond_wait (cond=0x3fffa803f770, mutex=0x3fffa803f748) at pthread_cond_wait.c:186 r4 = 128 r7 = 2 r5 = 1 r8 = 2 arg3 = 1 r0 = 221 r3 = 512 r6 = 0 arg4 = 0 __err = __ret = futex_val = 1 buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fffaef525e0, __canceltype = 16383, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x3fffa803f770, mutex = 0x3fffa803f748, bc_seq = 0} err = pshared = 0 pi_flag = 0 val = seq = 0 #1 0x00003fffb373825c in ?? () from /usr/lib64/libgfrpc.so.0 No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fffaef53160) at pthread_create.c:462 ---Type to continue, or q to quit--- pd = 0x3fffaef53160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {70367268043272, 70367268042552, 70367268042552, 0, 1, 0, 0, 0, 1, 1, 0, 0, 70367268042568, 8589934592, 70367384514912, 4294967297, -4995072473058770944, 309, 16, 0, 1, 0, -1, 0, -1, 0 , 117, 691489734656, 32, 70367268057072, -3819410108757049344, 0, 0, 0, 0}, mask_was_saved = 16383}}, priv = {pad = {0x0, 0x0, 0xbaadf00d00000000, 0x145}, data = {prev = 0x0, cleanup = 0x0, canceltype = -1163005939}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 19 (Thread 0x3fff966c7160 (LWP 561)): #0 0x00003fffb35b4044 in .__nanosleep () at ../sysdeps/unix/syscall-template.S:84 No locals. #1 0x00003fffb35b3e0c in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138 ts = {tv_sec = 1, tv_nsec = 79283866} set = {__val = {65536, 0 }} oset = {__val = {18446744066193095191, 0, 0, 0, 0, 70366972896928, 0, 70367388722224, 0, 0, 70367389062912, 0, 0, 0, 0, 0}} result = #2 0x00003fffaf356678 in ?? () from /usr/lib64/glusterfs/5.4/xlator/storage/posix.so No symbol table info available. #3 0x00003fffb36a3b30 in start_thread (arg=0x3fff966c7160) at pthread_create.c:462 pd = 0x3fff966c7160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {70367389029784, 70367389029760, 70367389029688, 70367389029664, 0, 70367389029640, 0, 0, 0, 459, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 196, 0 , 1023, 0, 32, 0, 599, 0, 0, 0, 3, 0, 4, 0, 0, 0, 1, 0, 0, 0, 0, 0, 599, 0, 0, 0, 139, 0, 0, 0, 6375, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" #4 0x0000000000000000 in ?? () No symbol table info available. Thread 18 (Thread 0x3fff94ec7160 (LWP 564)): #0 0x00003fffb36ab7b0 in __pthread_cond_wait (cond=0x3fffa80843e8, mutex=0x3fffa80843c0) at pthread_cond_wait.c:186 r4 = 128 r7 = 2 r5 = 1 r8 = 2 arg3 = 1 r0 = 221 ---Type to continue, or q to quit--- r3 = 512 r6 = 0 arg4 = 0 __err = __ret = futex_val = 1 buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fff94ec64e0, __canceltype = 0, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x3fffa80843e8, mutex = 0x3fffa80843c0, bc_seq = 0} err = pshared = 0 pi_flag = 0 val = seq = 0 #1 0x00003fffaf35699c in ?? () from /usr/lib64/glusterfs/5.4/xlator/storage/posix.so No symbol table info available. #2 0x00003fffaf356cc4 in ?? () from /usr/lib64/glusterfs/5.4/xlator/storage/posix.so No symbol table info available. #3 0x00003fffb36a3b30 in start_thread (arg=0x3fff94ec7160) at pthread_create.c:462 pd = 0x3fff94ec7160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {70367268323904, 70367268323904, 1215, 608, 607, 607, 70367268323968, 8589934592, 0, 1, 0, 0, 0, 0, 75374666, 0, 72057594037927936, 70366956122464, 72057594037927936, 70367268357728, 70367268324832, 36, 25694, 5, 72018011619328, 0, 0, 100, 4096, 0, 263, 395262763, 308, 258307626, 308, 258307626, 0, 0, 0, 7539334392389520705, -8557774763185766717, 0, 0, 0, 0, 70366947733856, 70367268324272, 70367268324272, 0, 1, 0, 0, 0, 1, 1, 0, 0, 70367268324288, 8589934592, 3, 80384, 0, 128849018890, 70366964511072}, mask_was_saved = 16777216}}, priv = {pad = {0x3fff966c7160, 0x100000000000001, 0x0, 0x1ff}, data = {prev = 0x3fff966c7160, cleanup = 0x100000000000001, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" #4 0x00003fffa8080c50 in ?? () No symbol table info available. Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 17 (Thread 0x3fff95ec7160 (LWP 562)): #0 0x00003fffb35b4044 in .__nanosleep () at ../sysdeps/unix/syscall-template.S:84 No locals. #1 0x00003fffb35b3e0c in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138 ts = {tv_sec = 12, tv_nsec = 78972105} set = {__val = {65536, 0 }} oset = {__val = {18446744066193095191, 0, 0, 0, 0, 0, 0, 68719476736, 0, 70366964507520, 256, 0, 0, 0, 0, 0}} result = #2 0x00003fffaf355e08 in ?? () from /usr/lib64/glusterfs/5.4/xlator/storage/posix.so No symbol table info available. #3 0x00003fffb36a3b30 in start_thread (arg=0x3fff95ec7160) at pthread_create.c:462 ---Type to continue, or q to quit--- pd = 0x3fff95ec7160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {70367389029784, 70367389029760, 70367389029688, 70367389029664, 0, 70367389029640, 0, 0, 0, 459, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 196, 0 , 1023, 0, 32, 0, 599, 0, 0, 0, 3, 0, 4, 0, 0, 0, 1, 0, 0, 0, 0, 0, 599, 0, 0, 0, 139, 0, 0, 0, 6375, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" #4 0x0000000000000000 in ?? () No symbol table info available. Thread 16 (Thread 0x3fffb1c9e160 (LWP 546)): #0 0x00003fffb36b19a0 in do_sigwait (set=, set at entry=0x3fffb1c9d660, sig=sig at entry=0x3fffb1c9d6e0) at ../sysdeps/unix/sysv/linux/sigwait.c:61 r4 = 0 r7 = 2 arg2 = 0 r5 = 0 r8 = 2 arg3 = 0 r0 = 176 r3 = 4 r6 = 8 arg4 = 8 ret = tmpset = {__val = {0, 0, 0, 0, 0, 0, 0, 70367459258040, 268515760, 268516016, 268516264, 268612384, 70367432038576, 268478552, 70367432005088, 0}} err = #1 0x00003fffb36b1a88 in __sigwait (set=0x3fffb1c9d660, sig=0x3fffb1c9d6e0) at ../sysdeps/unix/sysv/linux/sigwait.c:96 oldtype = 0 result = sig = 0x3fffb1c9d6e0 set = 0x3fffb1c9d660 #2 0x000000001000a82c in .glusterfs_sigwaiter () No symbol table info available. #3 0x00003fffb36a3b30 in start_thread (arg=0x3fffb1c9e160) at pthread_create.c:462 pd = 0x3fffb1c9e160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {8674967983565600856, 70367459482624, 8674967983542736380, 0, 0, 70367423623168, 70367432008224, 8388608, 70367459442720, 70367774066608, 268616336, 70367459468248, 268606096, 3, 0, 70367459468264, 70367774065808, 70367774065864, 4001536, 70367459443736, 70367432005440, -3187653500, 0 }, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = ---Type to continue, or q to quit--- freesize = __PRETTY_FUNCTION__ = "start_thread" #4 0x00003fffb35ee17c in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96 No locals. Thread 15 (Thread 0x3fffb249e160 (LWP 545)): #0 0x00003fffb36b1150 in .__nanosleep () at ../sysdeps/unix/syscall-template.S:84 No locals. #1 0x00003fffb37a3b9c in ?? () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fffb249e160) at pthread_create.c:462 pd = 0x3fffb249e160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-1, 0, -1, 0 , 49, 1, 0, 0, 0, 0, 369, 70367461387792, 70367267785088, 0, 70367461387880, 0, 0, 70367461387920, 0, 0, 70367461387960, 0, 0, 70367461388000, 0, 0, 70367461388040, 0, 0, 70367461388080, 0, 0, 70367461388120, 0, 0}, mask_was_saved = 16383}}, priv = {pad = {0x0, 0x3fffb38a2fa8, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x3fffb38a2fa8, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 14 (Thread 0x3fffad653160 (LWP 553)): #0 0x00003fffb36ab7b0 in __pthread_cond_wait (cond=0x3fffa804ed00, mutex=0x3fffa804ecd8) at pthread_cond_wait.c:186 r4 = 128 r7 = 2 r5 = 25 r8 = 2 arg3 = 25 r0 = 221 r3 = 512 r6 = 0 arg4 = 0 __err = __ret = futex_val = 25 buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fffad6525e0, __canceltype = 16383, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x3fffa804ed00, mutex = 0x3fffa804ecd8, bc_seq = 0} err = pshared = 0 pi_flag = 0 val = seq = 12 #1 0x00003fffaf0a41d0 in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/index.so ---Type to continue, or q to quit--- No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fffad653160) at pthread_create.c:462 pd = 0x3fffad653160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-7104728266551728278, 70367268105416, 70367268105416, 0, 1, 0, 0, 0, 25, 13, 12, 12, 70367268105432, 8589934592, 70367268106968, 70367268106344, 70367268105864, 0, 70367358300512, 0, -4995072473058770944, 101, 682899800064, 1, 70367268135648, -3819410108757049344, 0, 0, 0, 0, 124603186227970048, 0, 0, 37, 70367267824576, 0, 0, 117, 196092424128, 18, 70367268092080, -3819410108757049344, 0, 0, 0, 0, 8390898194479146030, 7018422612882190964, 8719174134808772608, 0, 0, 277, 3405691582, 0, 70367267785088, 34359738368, 268862464, 3, 4294967299, 1, 70367268105960, 70367268107992, 0, 0}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x3fffa804f6d8}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 13 (Thread 0x3fff96fff160 (LWP 560)): #0 0x00003fffb35e4764 in .__select () at ../sysdeps/unix/syscall-template.S:84 No locals. #1 0x00003fffaf2b4df0 in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/changelog.so No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fff96fff160) at pthread_create.c:462 pd = 0x3fff96fff160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 0, 0, 0, 0, 70367268283136, 70367268283136, 0, 0, 0, 0, 0, 70367268283192, 70367268283192, 70367268283808, 1, 0, -4995072473058770944, 149, 206158430208, 64, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 88, 64, 70367388068184, 0, 70367268267328, 70367268288960, 268615696, 70367461387880, -4995072473058770944, 101, 171798691840, 15, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 3419478345679794793, 7163871753673900218, -5913212017086300160, 133, 171798691840, 37, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 3419478345679794793, 7163871753673912110, 7452460622125822566, 8299961938229487461, 7813577622142234096, 936748722493063168, 0}, mask_was_saved = 0}}, priv = {pad = {0x28, 0x3fffa8076810, 0xcafebabe00000000, 0x0}, data = {prev = 0x28, cleanup = 0x3fffa8076810, canceltype = -889275714}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" #3 0x0000000000000000 in ?? () No symbol table info available. Thread 12 (Thread 0x3fffb049e160 (LWP 549)): #0 0x00003fffb36abccc in __pthread_cond_timedwait (cond=0x100705a8, mutex=0x10070580, abstime=0x3fffb049d670) at pthread_cond_timedwait.c:198 r4 = 393 r7 = 0 arg5 = 0 arg2 = ---Type to continue, or q to quit--- r5 = 26 r8 = 4294967295 arg6 = 4294967295 arg3 = 26 r0 = 221 r3 = 516 r6 = 70367406839408 arg4 = 70367406839408 arg1 = 268895660 __err = __ret = futex_val = 26 buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fffb049d540, __canceltype = 0, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x100705a8, mutex = 0x10070580, bc_seq = 6} result = 0 pshared = 0 pi_flag = 0 err = val = seq = 12 #1 0x00003fffb37dee54 in ?? () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #2 0x00003fffb37dfd4c in ?? () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #3 0x00003fffb36a3b30 in start_thread (arg=0x3fffb049e160) at pthread_create.c:462 pd = 0x3fffb049e160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0 , 268872368, 70367460585420, 70367415227936, 70367459256832, 0, 70367133610272, 1, 70367459451856, 1, 70366462478528, 70367267922320, 1, 0, 70367457788880, 70367415261360, 0, 0, 70367406845952, 70367415231008, 8388608, 70367459442720, 268872128, 268872128, 70367459468248, 70367461341232, 3, 0, 70367459468264, 70367774066048, 70367774066104, 4001536, 268872128, 70367133610160, 70367460585448, 0, 0, 70367457788880, 70367460585448, 0, 1107313796, 0, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = { 0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 11 (Thread 0x3fff977ff160 (LWP 559)): #0 0x00003fffb35e4764 in .__select () at ../sysdeps/unix/syscall-template.S:84 No locals. #1 0x00003fffaf2b4df0 in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/changelog.so No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fff977ff160) at pthread_create.c:462 ---Type to continue, or q to quit--- pd = 0x3fff977ff160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 0, 0, 0, 0, 70367268283136, 70367268283136, 0, 0, 0, 0, 0, 70367268283192, 70367268283192, 70367268283808, 1, 0, -4995072473058770944, 149, 206158430208, 64, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 88, 64, 70367388068184, 0, 70367268267328, 70367268288960, 268615696, 70367461387880, -4995072473058770944, 101, 171798691840, 15, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 3419478345679794793, 7163871753673900218, -5913212017086300160, 133, 171798691840, 37, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 3419478345679794793, 7163871753673912110, 7452460622125822566, 8299961938229487461, 7813577622142234096, 936748722493063168, 0}, mask_was_saved = 0}}, priv = {pad = {0x28, 0x3fffa8076810, 0xcafebabe00000000, 0x0}, data = {prev = 0x28, cleanup = 0x3fffa8076810, canceltype = -889275714}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" #3 0x0000000000000000 in ?? () No symbol table info available. Thread 10 (Thread 0x3fffb149e160 (LWP 547)): #0 0x00003fffb35b4044 in .__nanosleep () at ../sysdeps/unix/syscall-template.S:84 No locals. #1 0x00003fffb35b3e0c in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138 ts = {tv_sec = 6, tv_nsec = 35069955} set = {__val = {65536, 0 }} oset = {__val = {18446744066193095191, 0, 0, 0, 0, 0, 70367423608448, 8388608, 70367423616680, 70367423616664, 70367423608456, 0, 70367461387752, 70367461387824, 70367461387864, 3133061822}} result = #2 0x00003fffb37c675c in ?? () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #3 0x00003fffb36a3b30 in start_thread (arg=0x3fffb149e160) at pthread_create.c:462 pd = 0x3fffb149e160 now = unwind_buf = not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 9 (Thread 0x3fffae753160 (LWP 552)): #0 0x00003fffb36ab7b0 in __pthread_cond_wait (cond=0x3fffa803fa50, mutex=0x3fffa803fa28) at pthread_cond_wait.c:186 r4 = 128 r7 = 2 r5 = 27457 r8 = 2 arg3 = 27457 ---Type to continue, or q to quit--- r0 = 221 r3 = 512 r6 = 0 arg4 = 0 __err = __ret = futex_val = 27457 buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fffae7525e0, __canceltype = 16383, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x3fffa803fa50, mutex = 0x3fffa803fa28, bc_seq = 0} err = pshared = 0 pi_flag = 0 val = seq = 13728 #1 0x00003fffb373825c in ?? () from /usr/lib64/libgfrpc.so.0 No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fffae753160) at pthread_create.c:462 pd = 0x3fffae753160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {70367268036144, 70367268043288, 70367268043288, 0, 1, 0, 0, 0, 27457, 13729, 13728, 13728, 70367268043304, 8589934592, 70367376126304, 4294967297, -4995072473058770944, 309, 16, 0, 1, 0, -1, 0, -1, 0 , 117, 691489734656, 32, 70367268057072, -3819410108757049344, 0, 0, 0, 0}, mask_was_saved = 16383}}, priv = {pad = {0x0, 0x0, 0xbaadf00d00000000, 0x145}, data = {prev = 0x0, cleanup = 0x0, canceltype = -1163005939}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 8 (Thread 0x3fffafba2160 (LWP 550)): #0 0x00003fffb35ee8f4 in .epoll_wait () at ../sysdeps/unix/syscall-template.S:84 No locals. #1 0x00003fffb3807730 in ?? () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fffafba2160) at pthread_create.c:462 pd = 0x3fffafba2160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0 }, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" ---Type to continue, or q to quit--- Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 7 (Thread 0x3fff9452c160 (LWP 674)): #0 0x00003fffb36abccc in __pthread_cond_timedwait (cond=0x3fffa805c908, mutex=0x3fffa805c8e0, abstime=0x3fff9452b6d0) at pthread_cond_timedwait.c:198 r4 = 393 r7 = 0 arg5 = 0 arg2 = r5 = 26867 r8 = 4294967295 arg6 = 4294967295 arg3 = 26867 r0 = 221 r3 = 516 r6 = 70366937659088 arg4 = 70366937659088 arg1 = 70367268161804 __err = __ret = futex_val = 26867 buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fff9452b590, __canceltype = 16383, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x3fffa805c908, mutex = 0x3fffa805c8e0, bc_seq = 0} result = 0 pshared = 0 pi_flag = 0 err = val = seq = 13432 #1 0x00003fffaf13d380 in ?? () from /usr/lib64/glusterfs/5.4/xlator/performance/io-threads.so No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fff9452c160) at pthread_create.c:462 pd = 0x3fff9452c160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {70367268161864, 70367268161880, 70367268161880, 70367268161896, 70367268161896, 70367268161912, 70367268161912, 70367268161928, 70367268161928, 70367268161944, 70367268161944, 70367268161960, 70367268161960, 70367268161976, 70367268161976, 70367268161992, 70367268161992, 70367268162008, 70367268162008, 70367268162024, 70367268162024, 70367268162040, 70367268162040, 68719476752, 68719476737, 4294967296, 0, 0, 0, 0, 0, 0, 4096, 0, 262144, 0, 0, 72057594037927936, 70367267891312, 262144, 282574488338432, 0, 0, 0, -4995072473058770944, 309, 16, 0, 1, 0, -1, 0, -1, 0 }, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" #3 0x0000000000000000 in ?? () No symbol table info available. ---Type to continue, or q to quit--- Thread 6 (Thread 0x3fffb38d5000 (LWP 544)): #0 0x00003fffb36a5084 in pthread_join (threadid=70367397421408, thread_return=0x0) at pthread_join.c:90 r4 = 0 r7 = 2 arg2 = 0 r5 = 550 r8 = 2 arg3 = 550 r0 = 221 r3 = 512 r6 = 0 arg4 = 0 arg1 = 70367397421616 __err = __ret = __tid = 550 _buffer = {__routine = @0x3fffb36c8478: 0x3fffb36a4f70 , __arg = 0x3fffafba2588, __canceltype = 1735157363, __prev = 0x0} oldtype = 0 self = 0x3fffb38d5000 result = 0 #1 0x00003fffb3806be0 in ?? () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #2 0x00003fffb37c5214 in .event_dispatch () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #3 0x0000000010006358 in .main () No symbol table info available. Thread 5 (Thread 0x3fff9456c160 (LWP 671)): #0 0x00003fffb36abccc in __pthread_cond_timedwait (cond=0x3fffa805c908, mutex=0x3fffa805c8e0, abstime=0x3fff9456b6d0) at pthread_cond_timedwait.c:198 r4 = 393 r7 = 0 arg5 = 0 arg2 = r5 = 26865 r8 = 4294967295 arg6 = 4294967295 arg3 = 26865 r0 = 221 r3 = 516 r6 = 70366937921232 arg4 = 70366937921232 arg1 = 70367268161804 __err = __ret = ---Type to continue, or q to quit--- futex_val = 26865 buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fff9456b590, __canceltype = 16383, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x3fffa805c908, mutex = 0x3fffa805c8e0, bc_seq = 0} result = 0 pshared = 0 pi_flag = 0 err = val = seq = 13431 #1 0x00003fffaf13d380 in ?? () from /usr/lib64/glusterfs/5.4/xlator/performance/io-threads.so No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fff9456c160) at pthread_create.c:462 pd = 0x3fff9456c160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {70367268161864, 70367268161880, 70367268161880, 70367268161896, 70367268161896, 70367268161912, 70367268161912, 70367268161928, 70367268161928, 70367268161944, 70367268161944, 70367268161960, 70367268161960, 70367268161976, 70367268161976, 70367268161992, 70367268161992, 70367268162008, 70367268162008, 70367268162024, 70367268162024, 70367268162040, 70367268162040, 68719476752, 68719476737, 4294967296, 0, 0, 0, 0, 0, 0, 4096, 0, 262144, 0, 0, 72057594037927936, 70367267891312, 262144, 282574488338432, 0, 0, 0, -4995072473058770944, 309, 16, 0, 1, 0, -1, 0, -1, 0 }, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" #3 0x0000000000000000 in ?? () No symbol table info available. Thread 4 (Thread 0x3fff956c7160 (LWP 563)): #0 0x00003fffb36abccc in __pthread_cond_timedwait (cond=0x3fffa8084250, mutex=0x3fffa8084280, abstime=0x3fff956c66a0) at pthread_cond_timedwait.c:198 r4 = 393 r7 = 0 arg5 = 0 arg2 = r5 = 1215 r8 = 4294967295 arg6 = 4294967295 arg3 = 1215 r0 = 221 r3 = 516 r6 = 70366956119712 arg4 = 70366956119712 arg1 = 70367268323924 __err = __ret = futex_val = 1215 ---Type to continue, or q to quit--- buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fff956c6560, __canceltype = 0, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x3fffa8084250, mutex = 0x3fffa8084280, bc_seq = 0} result = 0 pshared = 0 pi_flag = 0 err = val = seq = 607 #1 0x00003fffaf351278 in ?? () from /usr/lib64/glusterfs/5.4/xlator/storage/posix.so No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fff956c7160) at pthread_create.c:462 pd = 0x3fff956c7160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {75374666, 0, 72057594037927936, 70366956122464, 72057594037927936, 70367268357728, 70367268324832, 36, 25694, 5, 72018011619328, 0, 0, 100, 4096, 0, 263, 395262763, 308, 258307626, 308, 258307626, 0, 0, 0, 7539334392389520705, -8557774763185766717, 0, 0, 0, 0, 70366947733856, 70367268324272, 70367268324272, 0, 1, 0, 0, 0, 1, 1, 0, 0, 70367268324288, 8589934592, 3, 80384, 0, 128849018890, 70366964511072, 72057594037927937, 0, 70366972899680, 72057594037927937, 0, 511, 2194728288356, 0, -4995072473058770944, 341, 193273528320, 256, 70367268310096, -3819410108757049344}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x3132382e3232342e, 0x39352e3134300000}, data = {prev = 0x0, cleanup = 0x0, canceltype = 825374766}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" #3 0x0000000000000000 in ?? () No symbol table info available. Thread 3 (Thread 0x3fff97fff160 (LWP 558)): #0 0x00003fffb35e4764 in .__select () at ../sysdeps/unix/syscall-template.S:84 No locals. #1 0x00003fffaf2b4df0 in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/changelog.so No symbol table info available. #2 0x00003fffb36a3b30 in start_thread (arg=0x3fff97fff160) at pthread_create.c:462 pd = 0x3fff97fff160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 0, 0, 0, 0, 70367268283136, 70367268283136, 0, 0, 0, 0, 0, 70367268283192, 70367268283192, 70367268283808, 1, 0, -4995072473058770944, 149, 206158430208, 64, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 88, 64, 70367388068184, 0, 70367268267328, 70367268288960, 268615696, 70367461387880, -4995072473058770944, 101, 171798691840, 15, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 3419478345679794793, 7163871753673900218, -5913212017086300160, 133, 171798691840, 37, 70367268268048, -3819410108757049344, 0, 0, 0, 0, 3419478345679794793, 7163871753673912110, 7452460622125822566, 8299961938229487461, 7813577622142234096, 936748722493063168, 0}, mask_was_saved = 0}}, priv = {pad = {0x28, 0x3fffa8076810, 0xcafebabe00000000, 0x0}, data = {prev = 0x28, cleanup = 0x3fffa8076810, canceltype = -889275714}}} not_first_call = pagesize_m1 = sp = freesize = ---Type to continue, or q to quit--- __PRETTY_FUNCTION__ = "start_thread" #3 0x0000000000000000 in ?? () No symbol table info available. Thread 2 (Thread 0x3fffb0c9e160 (LWP 548)): #0 0x00003fffb36abccc in __pthread_cond_timedwait (cond=0x100705a8, mutex=0x10070580, abstime=0x3fffb0c9d670) at pthread_cond_timedwait.c:198 r4 = 393 r7 = 0 arg5 = 0 arg2 = r5 = 25 r8 = 4294967295 arg6 = 4294967295 arg3 = 25 r0 = 221 r3 = 516 r6 = 70367415228016 arg4 = 70367415228016 arg1 = 268895660 __err = __ret = futex_val = 25 buffer = {__routine = @0x3fffb36c8b50: 0x3fffb36ab400 <__condvar_cleanup>, __arg = 0x3fffb0c9d540, __canceltype = 16383, __prev = 0x0} cbuffer = {oldtype = 0, cond = 0x100705a8, mutex = 0x10070580, bc_seq = 6} result = 0 pshared = 0 pi_flag = 0 err = val = seq = 12 #1 0x00003fffb37dee54 in ?? () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #2 0x00003fffb37dfd4c in ?? () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #3 0x00003fffb36a3b30 in start_thread (arg=0x3fffb0c9e160) at pthread_create.c:462 pd = 0x3fffb0c9e160 now = unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0 , 268872368, 70367460585420, 70367415227936, 70367459256832, 0, 70367133610272, 1, 70367459451856, 1, 70366462478528, 70367267922320, 1, 0, 70367457788880, 70367415261360, 0, 0, 70367406845952, 70367415231008, 8388608, 70367459442720, 268872128, 268872128, 70367459468248, 70367461341232, 3, 0, 70367459468264, 70367774066048, 70367774066104, 4001536, 268872128, 70367133610160, 70367460585448, 0, 0, 70367457788880, 70367460585448, 0, 1107313796, 0, 0, 0, 0, 0, 0, 0, 0, 0}, mask_was_saved = 0}}, priv = {pad = { 0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = ---Type to continue, or q to quit--- freesize = __PRETTY_FUNCTION__ = "start_thread" Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 1 (Thread 0x3fffad553160 (LWP 554)): #0 0x00003fffb3570d48 in _int_malloc (av=av at entry=0x3fffa4000020, bytes=bytes at entry=36) at malloc.c:3327 nb = idx = bin = victim = size = victim_index = remainder = remainder_size = block = bit = map = fwd = bck = errstr = 0x0 __func__ = "_int_malloc" #1 0x00003fffb35733dc in __GI___libc_malloc (bytes=36) at malloc.c:2921 ar_ptr = 0x3fffa4000020 victim = hook = __func__ = "__libc_malloc" #2 0x00003fffb362efd0 in x_inline (xdrs=0x3fffad550d20, len=) at xdr_sizeof.c:89 len = 36 xdrs = 0x3fffad550d20 #3 0x00003fffb370c488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #4 0x00003fffb370ce84 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #5 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e9c10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e9cc0 "\275\270^m\371j\233O" stat = #6 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e9c10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #7 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #8 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e9a90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e9b40 "\241\315\264rh<\b\274" stat = #9 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e9a90, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #10 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #11 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e9910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e99c0 "\266+W$\331o6\310" stat = #12 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e9910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #13 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #14 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e9790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e9840 "\262hF[\200\331\236\336" stat = #15 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e9790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #16 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #17 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e9610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e96c0 "\224T\fu\021Iw\021" stat = #18 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e9610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #19 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #20 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e9490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e9540 "\237n(\216\211\246[\261" stat = #21 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e9490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #22 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #23 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e9310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e93c0 "\235N~8\213&\221\356" stat = #24 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e9310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #25 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #26 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e9190, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40e9240 "\221\351\207\227\a \346" stat = #27 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e9190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #28 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #29 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e9010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e90c0 "\241\250?\337\037\355\v\r" stat = #30 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e9010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #31 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #32 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e8e90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e8f40 "\261\033(\201\022%q," stat = #33 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e8e90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #34 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #35 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e8d10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e8dc0 "\273\372\242K\351T\035\360" stat = #36 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e8d10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #37 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #38 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e8b90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e8c40 "\200z~h\330-\342}" stat = #39 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e8b90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #40 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #41 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e8a10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e8ac0 "\264\016\246~\222\031z7" stat = #42 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e8a10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #43 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #44 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e8890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e8940 "\202 #45 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e8890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #46 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #47 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e8710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e87c0 "\227\341\252" stat = #48 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e8710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #49 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #50 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e8590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e8640 "\206\322F,= #51 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e8590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #52 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #53 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e8410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e84c0 "\272Y\210ot7\004j" stat = #54 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e8410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #55 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #56 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e8290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e8340 "\200\306\001\317\375\a\307\206" stat = #57 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e8290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #58 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #59 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e8110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e81c0 "\203\067\327\224x\037\370\021" stat = ---Type to continue, or q to quit--- #60 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e8110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #61 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #62 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e7f90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e8040 "\203\344p\225\033\322\345W" stat = #63 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e7f90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #64 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #65 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e7e10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e7ec0 "\233\234\326iS\306\236\277" stat = #66 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e7e10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #67 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #68 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e7c90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e7d40 "\211\350\030R\344g#\303" stat = #69 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e7c90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #70 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #71 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e7b10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e7bc0 "\245=\277\374\036M|\202" stat = #72 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e7b10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #73 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #74 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e7990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e7a40 "\206N\261\372\320\341\371\365" stat = #75 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e7990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #76 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #77 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e7810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e78c0 "\224\271\271\216|\204~%" stat = #78 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e7810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #79 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #80 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e7690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e7740 "\224\351\333\274\354A-\233" stat = #81 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e7690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #82 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #83 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e7510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e75c0 "\220\345\357\325" stat = #84 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e7510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #85 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #86 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e7390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e7440 "\212\216I\320\006\244\335\032" stat = #87 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e7390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #88 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #89 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e7210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e72c0 "\201\b[\314X6[\273" stat = #90 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e7210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #91 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #92 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e7090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e7140 "\214#2$\210>\303\f" stat = #93 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e7090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #94 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #95 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e6f10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e6fc0 "\250\067\071\327\244\334lx" stat = #96 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e6f10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #97 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #98 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e6d90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e6e40 "\236\250\337\202\246\307\003\367" stat = #99 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e6d90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #100 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #101 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e6c10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e6cc0 "\222E\305\310>\362<\300" stat = #102 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e6c10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #103 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #104 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e6a90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e6b40 "\225C\346\322P\322f-" stat = #105 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e6a90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #106 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #107 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e6910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e69c0 "\214W85\017\033\273\200" stat = #108 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e6910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #109 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #110 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e6790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e6840 "\276U\v(\n\301\360$" ---Type to continue, or q to quit--- stat = #111 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e6790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #112 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #113 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e6610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e66c0 "\261\271\005j\334<9Y" stat = #114 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e6610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #115 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #116 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e6490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e6540 "\215jE2\222\067\070\004" stat = #117 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e6490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #118 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #119 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e6310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e63c0 "\236\307n\021n\314\003\213" stat = #120 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e6310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #121 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #122 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e6190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e6240 "\234n\363T\021\235\243\312" stat = #123 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e6190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #124 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #125 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e6010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e60c0 "\202\222A\233\225\001\327U" stat = #126 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e6010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #127 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #128 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e5e90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e5f40 "\274\344\270t\346%r\021" stat = #129 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e5e90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #130 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #131 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e5d10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e5dc0 "\215\agF\332\337\310W" stat = #132 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e5d10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #133 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #134 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e5b90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e5c40 "\234\066\022\270\226\t\221\065" stat = #135 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e5b90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #136 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #137 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e5a10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e5ac0 "\237\210\006o[lc\335" stat = #138 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e5a10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #139 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #140 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e5890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e5940 "\202\326\256[\256!\341g" stat = #141 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e5890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #142 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #143 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e5710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e57c0 "\226\262V\r\325\037;p" stat = #144 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e5710, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #145 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #146 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e5590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e5640 "\223\017uz\213\243\302\247" stat = #147 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e5590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #148 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #149 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e5410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e54c0 "\264\302\346Z\330\231\363\060" stat = #150 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e5410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #151 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #152 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e5290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e5340 "\210s\223" stat = #153 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e5290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #154 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #155 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e5110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e51c0 "\204\204\333\254\330>\376\070" stat = #156 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e5110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #157 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #158 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e4f90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e5040 "\223\254+j\250\266\263l" stat = #159 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e4f90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #160 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #161 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e4e10, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40e4ec0 "\223\062\203\341q\237ke" stat = #162 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e4e10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #163 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #164 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e4c90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e4d40 "\231\362\336x\303\267\221\213" stat = #165 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e4c90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #166 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #167 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e4b10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e4bc0 "\276R\036\201w\r\304\236" stat = #168 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e4b10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #169 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #170 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e4990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e4a40 "\211\373E\335;\017/*" stat = #171 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e4990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #172 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #173 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e4810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e48c0 "\226=\227\311\006\311\372\333" stat = #174 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e4810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #175 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #176 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e4690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e4740 "\220sT0\031\303\274+" stat = #177 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e4690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #178 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #179 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e4510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e45c0 "\222\313\062\250\264)\256\070" stat = #180 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e4510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #181 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #182 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e4390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e4440 "\252~2 \203\223\234T" stat = #183 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e4390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #184 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #185 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e4210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e42c0 "\256D\t\257_Nk\016" stat = #186 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e4210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #187 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #188 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e4090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e4140 "\262\272\256\212\362\020\365\226" stat = #189 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e4090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #190 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #191 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e3f10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e3fc0 "\227bN\323\201T\022\320" stat = #192 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e3f10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #193 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #194 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e3d90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e3e40 "\206 \204\217\350\213\236M" stat = ---Type to continue, or q to quit--- #195 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e3d90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #196 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #197 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e3c10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e3cc0 "\201\355\370v\023\320\204\375" stat = #198 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e3c10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #199 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #200 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e3a90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e3b40 "\200n+\246\245\317-\247" stat = #201 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e3a90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #202 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #203 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e3910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e39c0 "\274;\263l\350\257\205\060" stat = #204 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e3910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #205 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #206 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e3790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e3840 "\231\312:\316\346\345\245," stat = #207 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e3790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #208 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #209 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e3610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e36c0 "\251\321RL\306N\324~" stat = #210 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e3610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #211 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #212 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e3490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e3540 "\225W\320\300\334\327/ " stat = #213 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e3490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #214 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #215 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e3310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e33c0 "\210&\\2E\023+H" stat = #216 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e3310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #217 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #218 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e3190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e3240 "\275C\335\217\215\033E\303" stat = #219 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e3190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #220 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #221 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e3010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e30c0 "\225\252\265b\344\237+\r" stat = #222 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e3010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #223 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #224 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e2e90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e2f40 "\224\341\066+ \226\241\205" stat = #225 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e2e90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #226 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #227 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e2d10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e2dc0 "\217\222\032\263IX19" stat = #228 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e2d10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #229 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #230 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e2b90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e2c40 "\261\223\312\217\352I\021\222" stat = #231 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e2b90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #232 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #233 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e2a10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e2ac0 "\246\316\372\217\277\341\213\244" stat = #234 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e2a10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #235 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #236 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e2890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e2940 "\205v`p\320a\225\364" stat = #237 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e2890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #238 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #239 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e2710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e27c0 "\233\025o\347\060\215e\023" stat = #240 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e2710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #241 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #242 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e2590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e2640 "\270\253\255O+\260\214)" stat = #243 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e2590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #244 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #245 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e2410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e24c0 "\256JQ6\330\226\317M" ---Type to continue, or q to quit--- stat = #246 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e2410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #247 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #248 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e2290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e2340 "\240eJ`\313&\t\245" stat = #249 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e2290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #250 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #251 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e2110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e21c0 "\237\260F\345\004\307\020\357" stat = #252 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e2110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #253 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #254 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e1f90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e2040 "\200\336\062\333\201\"`@" stat = #255 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e1f90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #256 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #257 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e1e10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e1ec0 "\250\207\236\067\066\062\210\260" stat = #258 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e1e10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #259 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #260 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e1c90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e1d40 "\244\064\206\260\342\344\221\224" stat = #261 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e1c90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #262 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #263 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e1b10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e1bc0 "\242\003\033\361\025!Q\250" stat = #264 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e1b10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #265 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #266 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e1990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e1a40 "\202\337/\243\367MD\243" stat = #267 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e1990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #268 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #269 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e1810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e18c0 "\255W\234\020\250kC\240" stat = #270 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e1810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #271 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #272 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e1690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e1740 "\207V\235E\362\363\203n" stat = #273 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e1690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #274 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #275 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e1510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e15c0 "\211\340\210\261$\005\250#" stat = #276 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e1510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #277 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #278 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e1390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e1440 "\217K\a\003d\r\r*" stat = #279 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e1390, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #280 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #281 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e1210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e12c0 "\233\027\354\247\256R\227'" stat = #282 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e1210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #283 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #284 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e1090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e1140 "\246\061\177\032}17\232" stat = #285 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e1090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #286 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #287 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e0f10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e0fc0 "\241\060\023\v\026\221S\370" stat = #288 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e0f10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #289 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #290 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e0d90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e0e40 "\272\306\256{\261\232+j" stat = #291 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e0d90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #292 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #293 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e0c10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e0cc0 "\233\344\271\357\336\301\066\017" stat = #294 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e0c10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #295 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #296 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e0a90, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40e0b40 "\220\213\277V\263\207z\311" stat = #297 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e0a90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #298 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #299 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e0910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e09c0 "\246\346\232" stat = #300 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e0910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #301 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #302 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e0790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e0840 "\236\272\261\342<\230\243~" stat = #303 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e0790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #304 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #305 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e0610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e06c0 "\204c\333Jw\333\220\357" stat = #306 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e0610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #307 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #308 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e0490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e0540 "\216\227\360]]7\355H" stat = #309 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e0490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #310 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #311 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e0310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e03c0 "\244\337\226^ 8y[" stat = #312 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e0310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #313 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #314 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e0190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e0240 "\221h\026b1%\337\276" stat = #315 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e0190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #316 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #317 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40e0010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40e00c0 "\216\340\201\341\323\016C\257" stat = #318 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40e0010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #319 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #320 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dfe90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dff40 "\264s\326\325\302Yi\311" stat = #321 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dfe90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #322 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #323 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dfd10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dfdc0 "\262H\230\322_\231\v\275" stat = #324 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dfd10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #325 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #326 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dfb90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dfc40 "\212\216\017\260\250jn\271" stat = #327 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dfb90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #328 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #329 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dfa10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dfac0 "\245\275\002\323" stat = ---Type to continue, or q to quit--- #330 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dfa10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #331 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #332 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40df890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40df940 "\236\255\n$\316I\274+" stat = #333 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40df890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #334 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #335 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40df710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40df7c0 "\215\343<\227\221\306\241\306" stat = #336 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40df710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #337 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #338 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40df590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40df640 "\245.\221\223\067\374\245\350" stat = #339 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40df590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #340 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #341 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40df410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40df4c0 "\223\347\225O\003\341W\363" stat = #342 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40df410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #343 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #344 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40df290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40df340 "\245\322Yf\344i!\241" stat = #345 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40df290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #346 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #347 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40df110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40df1c0 "\274,\306\273\322\031\301\062" stat = #348 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40df110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #349 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #350 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40def90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40df040 "\210y+c\243\236\024l" stat = #351 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40def90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #352 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #353 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dee10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40deec0 "\251?\201rKcz\313" stat = #354 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dee10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #355 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #356 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dec90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ded40 "\205\022\220u\307\334\330\036" stat = #357 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dec90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #358 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #359 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40deb10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40debc0 "\266\003\001\234)s\b0" stat = #360 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40deb10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #361 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #362 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40de990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dea40 "\227\241\305Fh\366\217\206" stat = #363 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40de990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #364 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #365 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40de810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40de8c0 "\200W\030\344 \233i\b" stat = #366 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40de810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #367 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #368 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40de690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40de740 "\267\323s\257\214CJH" stat = #369 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40de690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #370 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #371 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40de510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40de5c0 "\204\232\213<\307\303Z(" stat = #372 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40de510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #373 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #374 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40de390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40de440 "\251)\353UBX\215\223" stat = #375 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40de390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #376 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #377 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40de210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40de2c0 "\237\370\230\212\217Zx\307" stat = #378 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40de210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #379 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #380 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40de090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40de140 "\233\327dk\247 9\345" ---Type to continue, or q to quit--- stat = #381 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40de090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #382 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #383 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ddf10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ddfc0 "\222k #384 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ddf10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #385 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #386 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ddd90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dde40 "\237\272\370\207\301\225J$" stat = #387 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ddd90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #388 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #389 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ddc10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ddcc0 "\206\247\351\306\026\321Vz" stat = #390 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ddc10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #391 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #392 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dda90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ddb40 "\275q\267\254\330\247\315\317" stat = #393 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dda90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #394 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #395 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dd910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dd9c0 "\214\216\224\311\ah\265\242" stat = #396 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dd910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #397 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #398 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dd790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dd840 "\237G3\274>z1K" stat = #399 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dd790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #400 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #401 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dd610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dd6c0 "\261\005 at m\372\326{A" stat = #402 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dd610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #403 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #404 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dd490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dd540 "\232\177\242\251\350\a\023," stat = #405 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dd490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #406 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #407 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dd310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dd3c0 "\242\337\352[\234/\306\224" stat = #408 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dd310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #409 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #410 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dd190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dd240 "\201\t\247\333t\340\312\341" stat = #411 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dd190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #412 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #413 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dd010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dd0c0 "\262\356\027p!\347\227X" stat = #414 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dd010, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #415 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #416 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dce90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dcf40 "\227\333r\267\342\030SX" stat = #417 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dce90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #418 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #419 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dcd10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dcdc0 "\230/\005Z\256]C^" stat = #420 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dcd10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #421 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #422 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dcb90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dcc40 "\244\017\360" stat = #423 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dcb90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #424 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #425 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dca10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dcac0 "\240\023\311|\221\216\324\234" stat = #426 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dca10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #427 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #428 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dc890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dc940 "\242\347Y\346\033\346 \365" stat = #429 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dc890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #430 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #431 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dc710, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40dc7c0 "\237)mZ\223\016\376n" stat = #432 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dc710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #433 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #434 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dc590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dc640 "\221\"\224\352U/\313\342" stat = #435 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dc590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #436 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #437 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dc410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dc4c0 "\203\314H\017.P #438 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dc410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #439 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #440 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dc290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dc340 "\260\320 \356\310\326\365\277" stat = #441 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dc290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #442 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #443 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dc110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dc1c0 "\273h\020*j\241\377\354" stat = #444 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dc110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #445 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #446 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dbf90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dc040 "\250\316\a\370'\362Gs" stat = #447 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dbf90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #448 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #449 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dbe10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dbec0 "\246)i\207\064_yr" stat = #450 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dbe10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #451 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #452 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dbc90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dbd40 "\262\246\312\331T*yx" stat = #453 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dbc90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #454 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #455 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dbb10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dbbc0 "\234\200\350\306\272\360\345\255" stat = #456 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dbb10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #457 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #458 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40db990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dba40 "\274\034?\nS<\r\"" stat = #459 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40db990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #460 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #461 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40db810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40db8c0 "\240Ky3\310;s\246" stat = #462 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40db810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #463 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #464 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40db690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40db740 "\233\231\025\311X\327\370\351" stat = ---Type to continue, or q to quit--- #465 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40db690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #466 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #467 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40db510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40db5c0 "\235)\313/\245&\314W" stat = #468 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40db510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #469 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #470 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40db390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40db440 "\237\033\356Yi@?\342" stat = #471 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40db390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #472 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #473 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40db210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40db2c0 "\256\060\364Vw\245\345J" stat = #474 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40db210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #475 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #476 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40db090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40db140 "\247\351$dk\036\064\t" stat = #477 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40db090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #478 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #479 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40daf10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dafc0 "\226\066\231\236\363\201\326\266" stat = #480 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40daf10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #481 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #482 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dad90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dae40 "\213\204\r\262\004t\377\235" stat = #483 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dad90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #484 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #485 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40dac10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dacc0 "\250E\017\061\303F\314\325" stat = #486 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40dac10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #487 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #488 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40daa90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40dab40 "\270>\\\177\071\035\371(" stat = #489 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40daa90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #490 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #491 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40da910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40da9c0 "\245\024\363\071\216\267\b " stat = #492 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40da910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #493 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #494 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40da790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40da840 "\200\346\203\362y\312\070\351" stat = #495 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40da790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #496 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #497 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40da610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40da6c0 "\211\246\312N3\205\364\232" stat = #498 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40da610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #499 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #500 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40da490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40da540 "\212\273\207\300{;L4" stat = #501 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40da490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #502 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #503 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40da310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40da3c0 "\224\017\250)\315\066bG" stat = #504 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40da310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #505 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #506 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40da190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40da240 "\275\252}\353xW\354\230" stat = #507 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40da190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #508 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #509 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40da010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40da0c0 "\266U\251\202\342\314\001\022" stat = #510 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40da010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #511 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #512 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d9e90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d9f40 "\241\v\242\061\257\361\033\242" stat = #513 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d9e90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #514 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #515 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d9d10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d9dc0 "\211p\"\225\332+\177\373" ---Type to continue, or q to quit--- stat = #516 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d9d10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #517 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #518 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d9b90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d9c40 "\277\370\004\204\302c\016\225" stat = #519 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d9b90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #520 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #521 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d9a10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d9ac0 "\217\375\255A\017\266;\270" stat = #522 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d9a10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #523 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #524 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d9890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d9940 "\277\212p\226w \212\315" stat = #525 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d9890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #526 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #527 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d9710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d97c0 "\232\364J9_\v\327F" stat = #528 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d9710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #529 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #530 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d9590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d9640 "\276L\237\240V\272\070j" stat = #531 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d9590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #532 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #533 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d9410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d94c0 "\276j\026\342>1\244\335" stat = #534 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d9410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #535 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #536 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d9290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d9340 "\213\367\a\270\227:\233\334" stat = #537 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d9290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #538 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #539 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d9110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d91c0 "\234\333J\331qOaD" stat = #540 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d9110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #541 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #542 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d8f90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d9040 "\255\304\300\032[\207ux" stat = #543 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d8f90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #544 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #545 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d8e10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d8ec0 "\210\244\346\263\037yx\027" stat = #546 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d8e10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #547 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #548 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d8c90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d8d40 "\257\356\263\370\335\200\242]" stat = #549 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d8c90, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #550 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #551 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d8b10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d8bc0 "\272j\357^\377M\267K" stat = #552 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d8b10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #553 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #554 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d8990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d8a40 "\265\v\025\002\231x\210=" stat = #555 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d8990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #556 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #557 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d8810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d88c0 "\247vE\350S\022R\337" stat = #558 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d8810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #559 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #560 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d8690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d8740 "\232\353\374\334\240\315$j" stat = #561 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d8690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #562 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #563 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d8510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d85c0 "\215Q\v\200s\320\066H" stat = #564 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d8510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #565 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #566 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d8390, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40d8440 "\232F`kTw\313\221" stat = #567 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d8390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #568 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #569 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d8210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d82c0 "\250\227!\251w\354\276e" stat = #570 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d8210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #571 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #572 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d8090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d8140 "\210\352\370\225\221Y\243B" stat = #573 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d8090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #574 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #575 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d7f10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d7fc0 "\255\061F\363\206\277\004\320" stat = #576 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d7f10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #577 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #578 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d7d90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d7e40 "\200\217\025k5:\263[" stat = #579 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d7d90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #580 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #581 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d7c10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d7cc0 "\267\211n~\356\364\062\200" stat = #582 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d7c10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #583 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #584 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d7a90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d7b40 "\276\342\326\f\236Ty(" stat = #585 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d7a90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #586 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #587 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d7910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d79c0 "\231\342\230\a\364\031\232\376" stat = #588 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d7910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #589 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #590 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d7790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d7840 "\262\264_\313\327\341\364\f" stat = #591 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d7790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #592 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #593 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d7610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d76c0 "\237\061[\\\306\371iQ" stat = #594 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d7610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #595 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #596 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d7490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d7540 "\216P\240xV\270\365-" stat = #597 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d7490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #598 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #599 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d7310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d73c0 "\212\233\060cI\204;x" stat = ---Type to continue, or q to quit--- #600 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d7310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #601 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #602 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d7190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d7240 "\275av\323\273\004g\304" stat = #603 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d7190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #604 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #605 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d7010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d70c0 "\276XQ\310\070\067~\311" stat = #606 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d7010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #607 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #608 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d6e90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d6f40 "\243\231\270>\366/A\331" stat = #609 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d6e90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #610 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #611 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d6d10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d6dc0 "\246\345*\220\177j\233\340" stat = #612 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d6d10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #613 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #614 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d6b90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d6c40 "\227\325N\320\227b\306k" stat = #615 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d6b90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #616 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #617 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d6a10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d6ac0 "\207_:6\277\367 #618 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d6a10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #619 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #620 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d6890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d6940 "\274\314.\216\002f\344." stat = #621 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d6890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #622 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #623 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d6710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d67c0 "\256zq\321\353\346\341\276" stat = #624 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d6710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #625 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #626 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d6590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d6640 "\246\001\r\262\351\061\064\n" stat = #627 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d6590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #628 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #629 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d6410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d64c0 "\277\357\253\237\266'\341\305" stat = #630 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d6410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #631 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #632 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d6290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d6340 "\245\233:\002D#\261\366" stat = #633 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d6290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #634 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #635 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d6110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d61c0 "\263\002N\033\330\035\022\353" stat = #636 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d6110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #637 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #638 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d5f90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d6040 "\215\314\375$\336\200\060\377" stat = #639 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d5f90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #640 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #641 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d5e10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d5ec0 "\276;,F\032\304#\003" stat = #642 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d5e10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #643 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #644 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d5c90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d5d40 "\251H+\300\301\217\366\263" stat = #645 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d5c90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #646 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #647 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d5b10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d5bc0 "\205\326w\235\310\022\361\375" stat = #648 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d5b10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #649 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #650 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d5990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d5a40 "\202\t\252\r4\"\241\226" ---Type to continue, or q to quit--- stat = #651 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d5990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #652 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #653 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d5810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d58c0 "\211\250TR\230\316\322;" stat = #654 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d5810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #655 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #656 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d5690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d5740 "\275\027\207$\031\061\207\266" stat = #657 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d5690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #658 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #659 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d5510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d55c0 "\243*ew)t\222\064" stat = #660 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d5510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #661 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #662 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d5390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d5440 "\247\067\335\360\342\236\207 " stat = #663 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d5390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #664 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #665 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d5210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d52c0 "\265y\r\347\261\060\204G" stat = #666 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d5210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #667 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #668 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d5090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d5140 "\241\315^\327v*\006\232" stat = #669 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d5090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #670 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #671 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d4f10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d4fc0 "\234\353(S=\214\267v" stat = #672 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d4f10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #673 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #674 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d4d90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d4e40 "\240F\320Z\214\277\262W" stat = #675 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d4d90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #676 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #677 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d4c10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d4cc0 "\262}\341\266\031\071\b\244" stat = #678 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d4c10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #679 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #680 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d4a90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d4b40 "\244\214\334\b\325t\205\316" stat = #681 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d4a90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #682 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #683 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d4910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d49c0 "\220!\b!\264W\377\251" stat = #684 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d4910, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #685 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #686 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d4790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d4840 "\251.\373e\232|\323\235" stat = #687 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d4790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #688 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #689 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d4610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d46c0 "\223\066\351\004\277\333\022Y" stat = #690 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d4610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #691 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #692 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d4490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d4540 "\252\303\005\372\311$\226\267" stat = #693 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d4490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #694 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #695 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d4310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d43c0 "\201j \032\306\232\017\224" stat = #696 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d4310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #697 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #698 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d4190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d4240 "\226\353}\340\213\"1\236" stat = #699 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d4190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #700 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #701 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d4010, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40d40c0 "\271\271\241u)\215\235\364" stat = #702 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d4010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #703 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #704 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d3e90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d3f40 "\222|QJ{\221\335\320" stat = #705 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d3e90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #706 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #707 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d3d10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d3dc0 "\232-\351 at +V<\b" stat = #708 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d3d10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #709 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #710 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d3b90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d3c40 "\267\310\276\071\255\266\065\314" stat = #711 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d3b90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #712 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #713 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d3a10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d3ac0 "\211L\321\361\365\327\272\337" stat = #714 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d3a10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #715 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #716 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d3890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d3940 "\216\227\207\205l\023\240\062" stat = #717 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d3890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #718 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #719 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d3710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d37c0 "\236\031\241\255\344d\237\227" stat = #720 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d3710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #721 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #722 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d3590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d3640 "\246/\314\340\277\266-\316" stat = #723 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d3590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #724 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #725 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d3410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d34c0 "\252\035\bi\310\006\332\242" stat = #726 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d3410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #727 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #728 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d3290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d3340 "\222\246\062\202+n\344H" stat = #729 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d3290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #730 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #731 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d3110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d31c0 "\262o\031\231DW\020\371" stat = #732 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d3110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #733 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #734 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d2f90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d3040 "\254U%\236\a]\350+" stat = ---Type to continue, or q to quit--- #735 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d2f90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #736 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #737 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d2e10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d2ec0 "\236\070\242\205\332Ud\217" stat = #738 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d2e10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #739 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #740 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d2c90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d2d40 "\203l\243\377)\202\202\214" stat = #741 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d2c90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #742 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #743 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d2b10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d2bc0 "\243Uq\262\362e" stat = #744 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d2b10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #745 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #746 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d2990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d2a40 "\260\274\337\371\220\372\326'" stat = #747 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d2990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #748 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #749 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d2810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d28c0 "\202\233\262\231\340\f($" stat = #750 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d2810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #751 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #752 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d2690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d2740 "\260\033:\344\071\v\252\271" stat = #753 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d2690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #754 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #755 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d2510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d25c0 "\221\274\206\004|>\253N" stat = #756 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d2510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #757 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #758 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d2390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d2440 "\267v\005\030\025\070\aq" stat = #759 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d2390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #760 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #761 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d2210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d22c0 "\235\270\345+\304 at 3a" stat = #762 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d2210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #763 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #764 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d2090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d2140 "\201Dj9t\326\201\232" stat = #765 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d2090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #766 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #767 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d1f10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d1fc0 "\231{\222\311\317\353\204H" stat = #768 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d1f10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #769 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #770 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d1d90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d1e40 "\271\323\241\064\367,\037\024" stat = #771 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d1d90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #772 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #773 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d1c10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d1cc0 "\243\063p\\\t\221RZ" stat = #774 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d1c10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #775 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #776 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d1a90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d1b40 "\237TH\361Y\203\301B" stat = #777 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d1a90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #778 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #779 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d1910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d19c0 "\257\250\231\317\242\352\262\272" stat = #780 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d1910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #781 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #782 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d1790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d1840 "\246\247%\a\360\037\244\257" stat = #783 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d1790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #784 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #785 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d1610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d16c0 "\246 -\327\064\347\374]" ---Type to continue, or q to quit--- stat = #786 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d1610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #787 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #788 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d1490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d1540 "\267\216\255\264N\037\031\254" stat = #789 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d1490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #790 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #791 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d1310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d13c0 "\214d\312Jhx/\026" stat = #792 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d1310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #793 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #794 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d1190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d1240 "\243J\303\355Zp=\345" stat = #795 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d1190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #796 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #797 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d1010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d10c0 "\270\037b\020Zl\242\246" stat = #798 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d1010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #799 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #800 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d0e90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d0f40 "\260\032\032^Nac\313" stat = #801 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d0e90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #802 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #803 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d0d10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d0dc0 "\260\230T\250\017~1x" stat = #804 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d0d10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #805 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #806 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d0b90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d0c40 "\200\005x\321\062S\005>" stat = #807 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d0b90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #808 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #809 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d0a10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d0ac0 "\225\302\037\356\030\f\224\257" stat = #810 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d0a10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #811 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #812 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d0890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d0940 "\260\036\024\036\335epN" stat = #813 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d0890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #814 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #815 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d0710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d07c0 "\223\336\255\002\247\364\361\373" stat = #816 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d0710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #817 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #818 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d0590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d0640 "\203\226\232@\245\322\352\032" stat = #819 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d0590, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #820 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #821 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d0410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d04c0 "\241\325\346O\005Or\305" stat = #822 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d0410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #823 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #824 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d0290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d0340 "\276;\325\302\221F+\330" stat = #825 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d0290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #826 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #827 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40d0110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d01c0 "\241)I\333\251\305\201\225" stat = #828 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40d0110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #829 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #830 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cff90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40d0040 "\270\064G\223\317)\277\306" stat = #831 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cff90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #832 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #833 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cfe10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cfec0 "\231\241\344\036\031\226\340\231" stat = #834 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cfe10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #835 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #836 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cfc90, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40cfd40 "\231\373>\364>\021\316i" stat = #837 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cfc90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #838 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #839 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cfb10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cfbc0 "\264\250\236\266pe\273Z" stat = #840 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cfb10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #841 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #842 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cf990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cfa40 "\255\225\023\037\260c\025\370" stat = #843 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cf990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #844 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #845 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cf810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cf8c0 "\214(f9\360'\204\234" stat = #846 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cf810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #847 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #848 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cf690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cf740 "\271\b\346\310?\266\b4" stat = #849 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cf690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #850 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #851 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cf510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cf5c0 "\205A\333\270NqC\233" stat = #852 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cf510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #853 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #854 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cf390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cf440 "\242\363\205\363\347\266>\231" stat = #855 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cf390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #856 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #857 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cf210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cf2c0 "\216p\001b\276\334\017\265" stat = #858 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cf210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #859 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #860 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cf090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cf140 "\235\263\001\241\202\325d\226" stat = #861 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cf090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #862 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #863 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cef10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cefc0 "\211iZ\202\265\372\330f" stat = #864 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cef10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #865 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #866 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ced90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cee40 "\241b\356\364\217?\033\311" stat = #867 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ced90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #868 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #869 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cec10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cecc0 "\216!\333\233m\217N^" stat = ---Type to continue, or q to quit--- #870 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cec10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #871 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #872 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cea90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ceb40 "\271\314sR\342\036K\202" stat = #873 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cea90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #874 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #875 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ce910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ce9c0 "\242w\004\274\063\373\375\274" stat = #876 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ce910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #877 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #878 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ce790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ce840 "\245\235\263\254\214\214\036\361" stat = #879 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ce790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #880 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #881 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ce610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ce6c0 "\237\327\257\222y\347_\\" stat = #882 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ce610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #883 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #884 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ce490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ce540 "\251\244\033+\300\006\036~" stat = #885 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ce490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #886 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #887 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ce310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ce3c0 "\222\266\362]\b\343\257\210" stat = #888 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ce310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #889 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #890 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ce190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ce240 "\230\261\331\225~\r\364\333" stat = #891 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ce190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #892 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #893 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ce010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ce0c0 "\215\241\200\025\375E*\035" stat = #894 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ce010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #895 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #896 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cde90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cdf40 "\214_96\352\024\335\234" stat = #897 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cde90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #898 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #899 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cdd10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cddc0 "\272\272\261\235[\360\061\323" stat = #900 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cdd10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #901 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #902 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cdb90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cdc40 "\214\363\307\313\227\237\017\264" stat = #903 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cdb90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #904 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #905 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cda10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cdac0 "\220*oA<8\353%" stat = #906 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cda10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #907 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #908 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cd890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cd940 "\275\300\255*\356\255iH" stat = #909 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cd890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #910 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #911 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cd710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cd7c0 "\273!\001/\\\237\017S" stat = #912 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cd710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #913 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #914 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cd590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cd640 "\205q\203\223Y\224\364\334" stat = #915 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cd590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #916 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #917 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cd410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cd4c0 "\272\\\020\224\355I\264\300" stat = #918 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cd410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #919 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #920 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cd290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cd340 "\243\344\037\352i%\004\034" ---Type to continue, or q to quit--- stat = #921 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cd290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #922 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #923 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cd110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cd1c0 "\266\060\353\263Y\020P\351" stat = #924 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cd110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #925 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #926 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ccf90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cd040 "\264\223\v\004C\343\377\037" stat = #927 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ccf90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #928 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #929 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cce10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ccec0 "\257/\254\bU\024\060\272" stat = #930 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cce10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #931 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #932 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ccc90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ccd40 "\260\217\355\374\372R\026\236" stat = #933 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ccc90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #934 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #935 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ccb10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ccbc0 "\261\315\376\232\361\245;;" stat = #936 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ccb10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #937 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #938 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cc990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cca40 "\275\005\217/\317a\365" stat = #939 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cc990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #940 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #941 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cc810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cc8c0 "\222\210XN\335\305\247\252" stat = #942 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cc810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #943 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #944 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cc690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cc740 "\240\371\364N\216T\252\364" stat = #945 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cc690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #946 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #947 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cc510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cc5c0 "\266\343\356\362a\a\250\261" stat = #948 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cc510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #949 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #950 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cc390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cc440 "\214\006e\035QNp\210" stat = #951 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cc390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #952 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #953 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cc210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cc2c0 "\221\206\333\020\265X!{" stat = #954 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cc210, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #955 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #956 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cc090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cc140 "\231z #957 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cc090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #958 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #959 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cbf10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cbfc0 "\235x\263\243i\001\240\275" stat = #960 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cbf10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #961 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #962 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cbd90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cbe40 "\253\207\311x#\347H\376" stat = #963 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cbd90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #964 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #965 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cbc10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cbcc0 "\215\315p\235\354\206\370L" stat = #966 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cbc10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #967 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #968 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cba90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cbb40 "\246&\330oA\322\357\211" stat = #969 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cba90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #970 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #971 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cb910, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40cb9c0 "\257\313\252\376\322\064v\006" stat = #972 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cb910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #973 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #974 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cb790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cb840 "\212\022\f\317\231\343\330[" stat = #975 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cb790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #976 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #977 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cb610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cb6c0 "\240\314\234n\204\324\217/" stat = #978 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cb610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #979 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #980 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cb490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cb540 "\271\235O\256':\222\207" stat = #981 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cb490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #982 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #983 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cb310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cb3c0 "\257X\234\254x\022\030\027" stat = #984 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cb310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #985 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #986 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cb190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cb240 "\255\330\334\071J\245l\374" stat = #987 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cb190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #988 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #989 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cb010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cb0c0 "\217\063\275\337)\365\225\231" stat = #990 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cb010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #991 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #992 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cae90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40caf40 "\224vA\365C\245p\326" stat = #993 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cae90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #994 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #995 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cad10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cadc0 "\201\033\222?[\314nw" stat = #996 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cad10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #997 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #998 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40cab90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40cac40 "\247\236\023\333\200\220Ql" stat = #999 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40cab90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1000 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1001 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40caa10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40caac0 "\220[\020\036\213V6!" stat = #1002 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40caa10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1003 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1004 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ca890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ca940 "\234k:\036" stat = ---Type to continue, or q to quit--- #1005 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ca890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1006 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1007 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ca710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ca7c0 "\210`\230\371x\302p\362" stat = #1008 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ca710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1009 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1010 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ca590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ca640 "\207k=\254\333\244\r2" stat = #1011 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ca590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1012 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1013 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ca410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ca4c0 "\240!\242]0/\260;" stat = #1014 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ca410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1015 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1016 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ca290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ca340 "\267\001\001\224\203\216\223\200" stat = #1017 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ca290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1018 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1019 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ca110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ca1c0 "\243!\306G\222\303V\372" stat = #1020 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ca110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1021 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #1022 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c9f90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ca040 "\264\314\003\375\a\263\260Q" stat = #1023 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c9f90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1024 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1025 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c9e10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c9ec0 "\277\027\241\034\375)\207\r" stat = #1026 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c9e10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1027 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1028 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c9c90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c9d40 "\213\231H\003\235\322\226\305" stat = #1029 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c9c90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1030 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1031 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c9b10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c9bc0 "\243Q&&\200\a\324\241" stat = #1032 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c9b10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1033 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1034 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c9990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c9a40 "\215\234$\243\350\030\243\242" stat = #1035 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c9990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1036 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1037 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c9810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c98c0 "\224\206\207\361\343O8s" stat = #1038 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c9810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #1039 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1040 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c9690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c9740 "\254s\233\177\245I\372-" stat = #1041 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c9690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1042 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1043 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c9510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c95c0 "\220\270\310\201\035\033}\t" stat = #1044 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c9510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1045 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1046 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c9390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c9440 "\217j|\261\265O\211<" stat = #1047 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c9390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1048 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1049 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c9210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c92c0 "\241\305\016\254g`\273\274" stat = #1050 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c9210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1051 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1052 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c9090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c9140 "\250\226\260.P\026\071\036" stat = #1053 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c9090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1054 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1055 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c8f10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c8fc0 "\204\376\017\035\335\033\253\206" ---Type to continue, or q to quit--- stat = #1056 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c8f10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1057 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1058 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c8d90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c8e40 "\201\206\064Z>\240a" stat = #1059 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c8d90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1060 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1061 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c8c10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c8cc0 "\252%\264\201\332\336(\277" stat = #1062 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c8c10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1063 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1064 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c8a90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c8b40 "\216\372\231\252o\223\255$" stat = #1065 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c8a90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1066 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1067 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c8910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c89c0 "\230Y\302\b\365E\225y" stat = #1068 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c8910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1069 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1070 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c8790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c8840 "\224@\v\371\244\351\"\234" stat = #1071 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c8790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1072 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #1073 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c8610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c86c0 "\227\017\261R$\257uH" stat = #1074 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c8610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1075 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1076 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c8490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c8540 "\230$&\322\321\344\205\301" stat = #1077 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c8490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1078 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1079 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c8310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c83c0 "\275\304\255\344\300\001\211\365" stat = #1080 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c8310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1081 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1082 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c8190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c8240 "\256\333\333\347\070\304\343\251" stat = #1083 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c8190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1084 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1085 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c8010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c80c0 "\255\345\224x\260\331\252\005" stat = #1086 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c8010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1087 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1088 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c7e90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c7f40 "\213\212em\234\376\337\233" stat = #1089 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c7e90, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1090 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1091 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c7d10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c7dc0 "\260\206vH\213\202\235\357" stat = #1092 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c7d10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1093 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1094 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c7b90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c7c40 "\255\v\324\301%\206\252\357" stat = #1095 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c7b90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1096 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1097 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c7a10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c7ac0 "\271\221;C\002\200\265Q" stat = #1098 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c7a10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1099 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1100 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c7890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c7940 "\235\257\267\305\031(\f\207" stat = #1101 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c7890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1102 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1103 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c7710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c77c0 "\260\017\205\372g\273\037\343" stat = #1104 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c7710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1105 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1106 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c7590, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40c7640 "\273}m0\327\260\067\244" stat = #1107 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c7590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1108 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1109 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c7410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c74c0 "\273\022\306\226\207\337\374\241" stat = #1110 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c7410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1111 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1112 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c7290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c7340 "\246k\005\332\252\371\"z" stat = #1113 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c7290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1114 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1115 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c7110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c71c0 "\225R\\Z\020\341R\220" stat = #1116 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c7110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1117 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1118 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c6f90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c7040 "\260\362\001\003+ds\241" stat = #1119 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c6f90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1120 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1121 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c6e10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c6ec0 "\275\337e #1122 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c6e10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #1123 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1124 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c6c90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c6d40 "\255W\300\346\016\316\360\241" stat = #1125 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c6c90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1126 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1127 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c6b10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c6bc0 "\275?*\332\276k\203\332" stat = #1128 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c6b10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1129 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1130 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c6990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c6a40 "\234\022\066\034\322W\352\035" stat = #1131 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c6990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1132 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1133 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c6810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c68c0 "\204\264&\224^\233\025\006" stat = #1134 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c6810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1135 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1136 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c6690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c6740 "\213\020\067|\250\350\035Y" stat = #1137 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c6690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1138 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1139 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c6510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c65c0 "\275h\030\016\331E\025\363" stat = ---Type to continue, or q to quit--- #1140 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c6510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1141 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1142 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c6390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c6440 "\270\033'\263D\373\357\036" stat = #1143 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c6390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1144 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1145 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c6210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c62c0 "\262\336!p\272\022\r\210" stat = #1146 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c6210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1147 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1148 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c6090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c6140 "\243\225V9[\241\032\243" stat = #1149 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c6090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1150 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1151 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c5f10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c5fc0 "\210\354x\264\235\224X\307" stat = #1152 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c5f10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1153 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1154 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c5d90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c5e40 "\222\224\355\320\212\247\273\266" stat = #1155 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c5d90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1156 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #1157 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c5c10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c5cc0 "\252\324\230\266qL\321F" stat = #1158 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c5c10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1159 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1160 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c5a90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c5b40 "\200\025,\237\314#\033\"" stat = #1161 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c5a90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1162 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1163 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c5910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c59c0 "\256\036\033\314\220\235j\375" stat = #1164 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c5910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1165 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1166 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c5790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c5840 "\271]\354\003-z\247\364" stat = #1167 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c5790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1168 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1169 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c5610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c56c0 "\274\267\332#\f\017k\236" stat = #1170 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c5610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1171 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1172 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c5490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c5540 "\233\251\370WlX\231J" stat = #1173 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c5490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #1174 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1175 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c5310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c53c0 "\263\371\237Fc\371\240\240" stat = #1176 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c5310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1177 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1178 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c5190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c5240 "\241\343W\234`\264\224\344" stat = #1179 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c5190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1180 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1181 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c5010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c50c0 "\201\304\215\302\277\203\231)" stat = #1182 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c5010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1183 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1184 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c4e90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c4f40 "\212\322^\341O\v\334b" stat = #1185 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c4e90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1186 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1187 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c4d10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c4dc0 "\203\032g\207\314J\016\302" stat = #1188 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c4d10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1189 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1190 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c4b90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c4c40 "\257!\004\317\361\355P2" ---Type to continue, or q to quit--- stat = #1191 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c4b90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1192 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1193 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c4a10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c4ac0 "\212[\271:E>\310\334" stat = #1194 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c4a10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1195 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1196 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c4890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c4940 "\225e\331\234/\251iK" stat = #1197 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c4890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1198 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1199 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c4710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c47c0 "\261t\f\335\255\215\300\035" stat = #1200 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c4710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1201 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1202 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c4590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c4640 "\203\244=?\243}\223c" stat = #1203 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c4590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1204 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1205 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c4410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c44c0 "\216\364\330\207\016\341#\264" stat = #1206 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c4410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1207 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #1208 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c4290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c4340 "\276\024\303\257BoU\320" stat = #1209 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c4290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1210 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1211 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c4110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c41c0 "\225\323}G\307qN1" stat = #1212 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c4110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1213 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1214 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c3f90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c4040 "\232\223\253!\347\016')" stat = #1215 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c3f90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1216 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1217 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c3e10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c3ec0 "\223\312[\r\024t\340I" stat = #1218 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c3e10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1219 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1220 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c3c90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c3d40 "\231\226\354\067\027\352\363j" stat = #1221 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c3c90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1222 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1223 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c3b10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c3bc0 "\224P\232y~\217\070\230" stat = #1224 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c3b10, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1225 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1226 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c3990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c3a40 "\202o\"\275\320\373\205\314" stat = #1227 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c3990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1228 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1229 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c3810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c38c0 "\242\312qi\237q\374f" stat = #1230 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c3810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1231 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1232 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c3690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c3740 "\275\375+\320\230\354\365\216" stat = #1233 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c3690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1234 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1235 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c3510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c35c0 "\253|b\314J\274\257\017" stat = #1236 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c3510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1237 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1238 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c3390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c3440 "\206\327B\364\351xM\r" stat = #1239 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c3390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1240 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1241 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c3210, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40c32c0 "\250\017\061\321\373\363\376C" stat = #1242 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c3210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1243 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1244 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c3090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c3140 "\222\231\256\213\353\070\200\307" stat = #1245 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c3090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1246 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1247 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c2f10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c2fc0 "\262\017Q8\202So\263" stat = #1248 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c2f10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1249 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1250 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c2d90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c2e40 "\212\034\211\201Wr\234\275" stat = #1251 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c2d90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1252 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1253 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c2c10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c2cc0 "\206\221\203\256j\262\370+" stat = #1254 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c2c10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1255 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1256 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c2a90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c2b40 "\207rH\346\331o\222\016" stat = #1257 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c2a90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #1258 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1259 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c2910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c29c0 "\235\200.e\302j;\312" stat = #1260 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c2910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1261 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1262 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c2790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c2840 "\226\201\240\306?\246\204\371" stat = #1263 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c2790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1264 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1265 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c2610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c26c0 "\237\002\006\325\n\277\222\352" stat = #1266 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c2610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1267 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1268 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c2490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c2540 "\237\252\026\333*{\264\017" stat = #1269 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c2490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1270 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1271 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c2310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c23c0 "\233`w\357\371\312\372\224" stat = #1272 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c2310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1273 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1274 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c2190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c2240 "\222li\316\230W\261\247" stat = ---Type to continue, or q to quit--- #1275 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c2190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1276 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1277 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c2010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c20c0 "\247\a\035\216\337EOO" stat = #1278 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c2010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1279 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1280 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c1e90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c1f40 "\202F\277\060\230\060\304\207" stat = #1281 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c1e90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1282 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1283 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c1d10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c1dc0 "\245\246\213\v\016Q+\331" stat = #1284 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c1d10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1285 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1286 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c1b90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c1c40 "\260\377\066&\215\032Z\260" stat = #1287 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c1b90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1288 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1289 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c1a10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c1ac0 "\201 \016x\fje," stat = #1290 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c1a10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1291 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #1292 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c1890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c1940 "\267e\314?\r\307\213\001" stat = #1293 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c1890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1294 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1295 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c1710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c17c0 "\272q\227\202\324#\210," stat = #1296 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c1710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1297 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1298 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c1590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c1640 "\261\303A\356\205\247\005\247" stat = #1299 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c1590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1300 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1301 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c1410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c14c0 "\201c\312\321\313\333rP" stat = #1302 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c1410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1303 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1304 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c1290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c1340 "\231\225\314\302s\250\n0" stat = #1305 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c1290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1306 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1307 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c1110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c11c0 "\201\376\254\017\355\347\377\277" stat = #1308 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c1110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #1309 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1310 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c0f90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c1040 "\250\371]\254\027f\316\223" stat = #1311 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c0f90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1312 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1313 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c0e10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c0ec0 "\262\032^\201\324\320\v\243" stat = #1314 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c0e10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1315 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1316 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c0c90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c0d40 "\251\253\065\226\"x\350\a" stat = #1317 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c0c90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1318 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1319 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c0b10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c0bc0 "\224\330\340\024\361\224[\213" stat = #1320 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c0b10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1321 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1322 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c0990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c0a40 "\255\021\302\235\343%\021@" stat = #1323 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c0990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1324 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1325 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c0810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c08c0 "\274\375w\006\067d\350\272" ---Type to continue, or q to quit--- stat = #1326 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c0810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1327 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1328 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c0690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c0740 "\266\261\310\235\036;\306\277" stat = #1329 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c0690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1330 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1331 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c0510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c05c0 "\225\367\216K\016\233i\001" stat = #1332 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c0510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1333 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1334 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c0390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c0440 "\205\331\034'\v}\024w" stat = #1335 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c0390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1336 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1337 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c0210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c02c0 "\274\063\266\350\240\321h\257" stat = #1338 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c0210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1339 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1340 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40c0090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40c0140 "\275\360hu\372P\036 " stat = #1341 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40c0090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1342 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #1343 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bff10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bffc0 "\251d\314\366\277\301D\215" stat = #1344 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bff10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1345 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1346 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bfd90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bfe40 "\250\366$\335\023\230z(" stat = #1347 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bfd90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1348 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1349 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bfc10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bfcc0 "\270\200\373j\362\227\305\b" stat = #1350 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bfc10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1351 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1352 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bfa90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bfb40 "\257x\341^;\347\300\357" stat = #1353 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bfa90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1354 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1355 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bf910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bf9c0 "\255_|\261\221\201\267\064" stat = #1356 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bf910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1357 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1358 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bf790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bf840 "\270\312r\263~\240\020\231" stat = #1359 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bf790, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1360 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1361 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bf610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bf6c0 "\242~Rd.{EL" stat = #1362 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bf610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1363 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1364 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bf490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bf540 "\227]\261-\213u\242\305" stat = #1365 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bf490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1366 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1367 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bf310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bf3c0 "\235\244\273\020\364Lue" stat = #1368 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bf310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1369 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1370 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bf190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bf240 "\200\035\250\020f\373\273\002" stat = #1371 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bf190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1372 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1373 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bf010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bf0c0 "\262;\257\337uw6#" stat = #1374 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bf010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1375 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1376 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bee90, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40bef40 "\217D\267\370\067\230HS" stat = #1377 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bee90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1378 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1379 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bed10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bedc0 "\256\230\332\327\251G\026l" stat = #1380 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bed10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1381 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1382 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40beb90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bec40 "\223R\035\r\363\004\016\321" stat = #1383 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40beb90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1384 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1385 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bea10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40beac0 "\224F\016\363\313\356A6" stat = #1386 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bea10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1387 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1388 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40be890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40be940 "\237E\257\316=\344\061:" stat = #1389 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40be890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1390 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1391 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40be710, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40be7c0 "\266>\242j\032]t{" stat = #1392 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40be710, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #1393 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1394 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40be590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40be640 "\225q\241\032\312\272\276z" stat = #1395 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40be590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1396 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1397 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40be410, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40be4c0 "\247\242\036\234'\357\021s" stat = #1398 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40be410, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1399 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1400 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40be290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40be340 "\227\004c\266\020c$(" stat = #1401 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40be290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1402 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1403 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40be110, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40be1c0 "\200\217\221\035\244u\214j" stat = #1404 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40be110, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1405 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1406 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bdf90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40be040 "\252\265\346]\031\065\374\221" stat = #1407 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bdf90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1408 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1409 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bde10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bdec0 "\213h ---Type to continue, or q to quit--- #1410 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bde10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1411 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1412 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bdc90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bdd40 "\253\375\310\027\033\317t\233" stat = #1413 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bdc90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1414 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1415 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bdb10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bdbc0 "\261D\237\325\006X\277\323" stat = #1416 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bdb10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1417 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1418 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bd990, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bda40 "\211\375\212\026\320\021\307\\" stat = #1419 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bd990, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1420 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1421 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bd810, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bd8c0 "\262\375gUz\027\037\"" stat = #1422 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bd810, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1423 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1424 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bd690, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bd740 "\255\066\212_C\206\253S" stat = #1425 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bd690, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1426 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #1427 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bd510, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bd5c0 "\226~\341i{(x\217" stat = #1428 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bd510, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1429 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1430 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bd390, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bd440 "\207\312\005\200\364L\202\326" stat = #1431 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bd390, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1432 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1433 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bd210, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bd2c0 "\263g\326\353t\241s\215" stat = #1434 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bd210, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1435 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1436 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bd090, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bd140 "\245\035\227\273o\300^\001" stat = #1437 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bd090, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1438 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1439 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bcf10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bcfc0 "\267\001\241\265\027!\275\335" stat = #1440 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bcf10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1441 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1442 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bcd90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bce40 "\211\245\024C\267\212X\247" stat = #1443 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bcd90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #1444 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1445 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bcc10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bccc0 "\272_<#^\024d6" stat = #1446 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bcc10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1447 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1448 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bca90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bcb40 "\250A\r\324\341\360" stat = #1449 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bca90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1450 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1451 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bc910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bc9c0 "\203!n\264\b#{k" stat = #1452 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bc910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1453 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1454 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bc790, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bc840 "\261\031)9\004]hl" stat = #1455 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bc790, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1456 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1457 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bc610, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bc6c0 "\217\030Y\310\274\344\203\317" stat = #1458 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bc610, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1459 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1460 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bc490, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bc540 "\251\354\337\263\321\067B" ---Type to continue, or q to quit--- stat = #1461 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bc490, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1462 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1463 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bc310, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bc3c0 "\237&?\307\215)\332\267" stat = #1464 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bc310, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1465 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1466 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bc190, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bc240 "\265\304H\241%\341*\035" stat = #1467 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bc190, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1468 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1469 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bc010, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bc0c0 "\226h\272b$T\275\303" stat = #1470 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bc010, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1471 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1472 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bbe90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bbf40 "\261\221\312a\225\367/\312" stat = #1473 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bbe90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1474 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1475 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bbd10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bbdc0 "\223\200^,\223\037}\241" stat = #1476 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bbd10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1477 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #1478 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bbb90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bbc40 "\216\303l\245d\332\223'" stat = #1479 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bbb90, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1480 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1481 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bba10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bbac0 "\215Pr\263\374vF\315" stat = #1482 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bba10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1483 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1484 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bb890, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bb940 "\220\036\177\016\311A\273\266" stat = #1485 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bb890, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1486 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1487 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bb760, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bb7c0 "\215v)\232K\206t\t" stat = #1488 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bb760, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1489 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1490 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bb630, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bb690 "\257\257\252P\n\310\343\257" stat = #1491 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bb630, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1492 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1493 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bb500, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bb560 "\241\071P/\271\305q;" stat = #1494 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bb500, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1495 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1496 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bb3d0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bb430 "\237\027a\025\017%\205\006" stat = #1497 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bb3d0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1498 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1499 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bb2a0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bb300 "\205#\202\065\fay\026" stat = #1500 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bb2a0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1501 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1502 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bb170, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bb1d0 "\263\066@\003\276}" stat = #1503 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bb170, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1504 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1505 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bb040, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bb0a0 "\257\260Q\313<\320\351\245" stat = #1506 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bb040, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1507 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1508 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40baf10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40baf70 "\266\277\215\367\201\200\315e" stat = #1509 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40baf10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1510 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1511 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bade0, size=, proc=) at xdr_ref.c:84 ---Type to continue, or q to quit--- loc = 0x3fffa40bae40 "\237\064\367\311\313\355\265\217" stat = #1512 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bade0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1513 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1514 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bacb0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40bad10 "\233N\n\267{\377x7" stat = #1515 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bacb0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1516 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1517 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40bab80, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40babe0 "\215\336R\345\347\235q6" stat = #1518 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40bab80, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1519 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1520 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40baa50, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40baab0 "\264\303\356\270N\243pU" stat = #1521 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40baa50, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1522 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1523 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ba920, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ba980 "\245\031R\034\067n\217\201" stat = #1524 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ba920, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1525 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1526 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ba7f0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ba850 "\236\234&\bM\264\367\270" stat = #1527 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ba7f0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 ---Type to continue, or q to quit--- #1528 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1529 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ba6c0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ba720 "\250\326\310\235\022\355\251\257" stat = #1530 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ba6c0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1531 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1532 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ba590, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ba5f0 "\247oM at li\031\257" stat = #1533 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ba590, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1534 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1535 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ba460, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ba4c0 "\207\353\351\376\312\025\235\215" stat = #1536 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ba460, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1537 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1538 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ba330, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ba390 "\224\304\330>\247\343\214\342" stat = #1539 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ba330, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1540 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1541 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ba200, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ba260 "\220\376\305\316\315\026\307`" stat = #1542 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ba200, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1543 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1544 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40ba0d0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ba130 "\200\v\025\017\244\224W\367" stat = ---Type to continue, or q to quit--- #1545 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40ba0d0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1546 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1547 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b9fa0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40ba000 "\272\300\032\220 #1548 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b9fa0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1549 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1550 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b9e70, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b9ed0 "\261\330\275,\204\267\225v" stat = #1551 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b9e70, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1552 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1553 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b9d40, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b9da0 "\215\223~\224\325\315[/" stat = #1554 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b9d40, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1555 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1556 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b9c10, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b9c70 "\251\320\\\307\362\256\246#" stat = #1557 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b9c10, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1558 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1559 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b9ae0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b9b40 "\202\220h\300\372\207\362\004" stat = #1560 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b9ae0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1561 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. ---Type to continue, or q to quit--- #1562 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b99b0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b9a10 "\247m\a\217\276lx\252" stat = #1563 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b99b0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1564 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1565 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b9880, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b98e0 "\267t\331\213\351q\316\237" stat = #1566 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b9880, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1567 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1568 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b9750, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b97b0 "\201m\316\367\312i\316\070" stat = #1569 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b9750, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1570 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1571 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b9620, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b9680 "\270\217UtR\344\377z" stat = #1572 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b9620, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1573 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1574 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b94f0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b9550 "\205\331\305c\f\210\304O" stat = #1575 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b94f0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1576 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1577 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b93c0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b9420 "\262\215\320Uq\336\340n" stat = #1578 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b93c0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 ---Type to continue, or q to quit--- more_data = 1 #1579 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1580 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b9290, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b92f0 "\241\365" stat = #1581 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b9290, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1582 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1583 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b9160, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b91c0 "\274\232\245L\024c&5" stat = #1584 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b9160, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1585 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1586 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b9030, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b9090 "\270|\352\324\365\324n\017" stat = #1587 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b9030, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1588 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1589 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b8f00, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b8f60 "\200" stat = #1590 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b8f00, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1591 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1592 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b8dd0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b8e30 "\264B\016\032z!\307\370" stat = #1593 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b8dd0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1594 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1595 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b8ca0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b8d00 "\253T\266`\240\342\270z" ---Type to continue, or q to quit--- stat = #1596 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b8ca0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1597 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1598 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b8b70, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b8bd0 "\231\021\206\002\346\256\265\315" stat = #1599 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b8b70, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1600 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1601 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b8a40, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b8aa0 "\265\223\271;U. \326" stat = #1602 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b8a40, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1603 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1604 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b8910, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b8970 "\256\025\262\201_\277\356\026" stat = #1605 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b8910, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1606 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1607 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b87e0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b8840 "\255]s/\032\363\304S" stat = #1608 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b87e0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1609 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1610 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b86b0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b8710 "\257\062\222\243\070D\330\335" stat = #1611 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b86b0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1612 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 ---Type to continue, or q to quit--- No symbol table info available. #1613 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b8580, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b85e0 "\222\314\337z,1\376\004" stat = #1614 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b8580, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1615 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1616 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b8450, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b84b0 "\222\222\372\070\300\237\206_" stat = #1617 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b8450, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1618 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1619 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b8320, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b8380 "\241\202}\320aN\361\061" stat = #1620 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b8320, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1621 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1622 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b81f0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b8250 "\203\f\240\302t\221~\362" stat = #1623 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b81f0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1624 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1625 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b80c0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b8120 "\224\250\322]+\024\247\245" stat = #1626 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b80c0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1627 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1628 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b7f90, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b7ff0 "\233{d~@A>\235" stat = #1629 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b7f90, obj_size=, ---Type to continue, or q to quit--- xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1630 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1631 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b7e60, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b7ec0 "\225\016\237\257\"d!\210" stat = #1632 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b7e60, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1633 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1634 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffa40b7c30, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b7d90 "\377\377\377\377\377\377\377\377" stat = #1635 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffa40b7c30, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1636 0x00003fffb370cec0 in .xdr_gfx_dirplist () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1637 0x00003fffb362ec28 in __GI_xdr_reference (xdrs=0x3fffad550d20, pp=0x3fffad550fe0, size=, proc=) at xdr_ref.c:84 loc = 0x3fffa40b7b60 "" stat = #1638 0x00003fffb362ee04 in __GI_xdr_pointer (xdrs=0x3fffad550d20, objpp=0x3fffad550fe0, obj_size=, xdr_obj=@0x3fffb37294b0: 0x3fffb370cdc0 <.xdr_gfx_dirplist>) at xdr_ref.c:135 more_data = 1 #1639 0x00003fffb3711348 in .xdr_gfx_readdirp_rsp () from /usr/lib64/libgfxdr.so.0 No symbol table info available. #1640 0x00003fffb362f120 in __GI_xdr_sizeof (func=, data=) at xdr_sizeof.c:157 x = {x_op = XDR_ENCODE, x_ops = 0x3fffad550cd0, x_public = 0x3fffa8029d00 "", x_private = 0x3fffa403fba0 "", x_base = 0x24 , x_handy = 110908} ops = {x_getlong = @0x3fffb3691b20: 0x3fffb362eea0 , x_putlong = @0x3fffb3691b80: 0x3fffb362f030 <.x_putlong>, x_getbytes = @0x3fffb3691b20: 0x3fffb362eea0 , x_putbytes = @0x3fffb3691ad8: 0x3fffb362ee30 , x_getpostn = @0x3fffb3691af0: 0x3fffb362ee60 , x_setpostn = @0x3fffb3691b08: 0x3fffb362ee80 , x_inline = @0x3fffb3691b68: 0x3fffb362ef60 , x_destroy = @0x3fffb3691b50: 0x3fffb362eef0 , x_getint32 = @0x3fffb3691b20: 0x3fffb362eea0 , x_putint32 = @0x3fffb3691b38: 0x3fffb362eec0 } stat = #1641 0x00003fffaef8d92c in ?? () from /usr/lib64/glusterfs/5.4/xlator/protocol/server.so No symbol table info available. #1642 0x00003fffaef8db88 in ?? () from /usr/lib64/glusterfs/5.4/xlator/protocol/server.so No symbol table info available. #1643 0x00003fffaefe54c0 in ?? () from /usr/lib64/glusterfs/5.4/xlator/protocol/server.so No symbol table info available. #1644 0x00003fffaf04955c in ?? () from /usr/lib64/glusterfs/5.4/xlator/debug/io-stats.so No symbol table info available. ---Type to continue, or q to quit--- #1645 0x00003fffaf0f0984 in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/marker.so No symbol table info available. #1646 0x00003fffb383a098 in .default_readdirp_cbk () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #1647 0x00003fffaf150e94 in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/upcall.so No symbol table info available. #1648 0x00003fffaf20757c in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/locks.so No symbol table info available. #1649 0x00003fffaf25542c in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/access-control.so No symbol table info available. #1650 0x00003fffaf27fbe0 in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/bitrot-stub.so No symbol table info available. #1651 0x00003fffaf385ef0 in ?? () from /usr/lib64/glusterfs/5.4/xlator/storage/posix.so No symbol table info available. #1652 0x00003fffaf386990 in ?? () from /usr/lib64/glusterfs/5.4/xlator/storage/posix.so No symbol table info available. #1653 0x00003fffb384315c in .default_readdirp () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #1654 0x00003fffb384315c in .default_readdirp () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #1655 0x00003fffaf279b28 in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/bitrot-stub.so No symbol table info available. #1656 0x00003fffaf25056c in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/access-control.so No symbol table info available. #1657 0x00003fffaf1fdaa4 in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/locks.so No symbol table info available. #1658 0x00003fffb384315c in .default_readdirp () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #1659 0x00003fffb384315c in .default_readdirp () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #1660 0x00003fffb384315c in .default_readdirp () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #1661 0x00003fffaf15c37c in ?? () from /usr/lib64/glusterfs/5.4/xlator/features/upcall.so No symbol table info available. #1662 0x00003fffb38668a4 in .default_readdirp_resume () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #1663 0x00003fffb37c0c64 in .call_resume_wind () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #1664 0x00003fffb37c1694 in .call_resume () from /usr/lib64/libglusterfs.so.0 No symbol table info available. #1665 0x00003fffaf13d4c8 in ?? () from /usr/lib64/glusterfs/5.4/xlator/performance/io-threads.so No symbol table info available. #1666 0x00003fffb36a3b30 in start_thread (arg=0x3fffad553160) at pthread_create.c:462 pd = 0x3fffad553160 now = ---Type to continue, or q to quit--- unwind_buf = {cancel_jmp_buf = {{jmp_buf = {70367268161864, 70367268161880, 70367268161880, 70367268161896, 70367268161896, 70367268161912, 70367268161912, 70367268161928, 70367268161928, 70367268161944, 70367268161944, 70367268161960, 70367268161960, 70367268161976, 70367268161976, 70367268161992, 70367268161992, 70367268162008, 70367268162008, 70367268162024, 70367268162024, 70367268162040, 70367268162040, 68719476752, 68719476737, 4294967296, 0, 0, 0, 0, 0, 0, 4096, 0, 262144, 0, 0, 72057594037927936, 70367267891312, 262144, 282574488338432, 0, 0, 0, -4995072473058770944, 309, 16, 0, 1, 0, -1, 0, -1, 0 }, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}} not_first_call = pagesize_m1 = sp = freesize = __PRETTY_FUNCTION__ = "start_thread" Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) (gdb) (gdb) (gdb) From abhishpaliwal at gmail.com Wed Mar 13 05:12:56 2019 From: abhishpaliwal at gmail.com (ABHISHEK PALIWAL) Date: Wed, 13 Mar 2019 10:42:56 +0530 Subject: [Gluster-devel] [Gluster-users] Glusterfsd crashed with SIGSEGV In-Reply-To: References: Message-ID: logs for libgfrpc.so pabhishe at arn-build3$ldd ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.* ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.0: not a dynamic executable ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.0.0.1: not a dynamic executable On Wed, Mar 13, 2019 at 10:02 AM ABHISHEK PALIWAL wrote: > Here are the logs: > > > pabhishe at arn-build3$ldd > ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.* > ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0: > not a dynamic executable > ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0.0.1: > not a dynamic executable > pabhishe at arn-build3$ldd > ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0.0.1 > not a dynamic executable > > > For backtraces I have attached the core_logs.txt file. > > Regards, > Abhishek > > On Wed, Mar 13, 2019 at 9:51 AM Amar Tumballi Suryanarayan < > atumball at redhat.com> wrote: > >> Hi Abhishek, >> >> Few more questions, >> >> >>> On Tue, Mar 12, 2019 at 10:58 AM ABHISHEK PALIWAL < >>> abhishpaliwal at gmail.com> wrote: >>> >>>> Hi Amar, >>>> >>>> Below are the requested logs >>>> >>>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libglusterfs.so >>>> not a dynamic executable >>>> >>>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libgfrpc.so >>>> not a dynamic executable >>>> >>>> >> Can you please add a * at the end, so it gets the linked library list >> from the actual files (ideally this is a symlink, but I expected it to >> resolve like in Fedora). >> >> >> >>> root at 128:/# gdb /usr/sbin/glusterd core.1099 >>>> GNU gdb (GDB) 7.10.1 >>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>> License GPLv3+: GNU GPL version 3 or later < >>>> http://gnu.org/licenses/gpl.html> >>>> This is free software: you are free to change and redistribute it. >>>> There is NO WARRANTY, to the extent permitted by law. Type "show >>>> copying" >>>> and "show warranty" for details. >>>> This GDB was configured as "powerpc64-wrs-linux". >>>> Type "show configuration" for configuration details. >>>> For bug reporting instructions, please see: >>>> . >>>> Find the GDB manual and other documentation resources online at: >>>> . >>>> For help, type "help". >>>> Type "apropos word" to search for commands related to "word"... >>>> Reading symbols from /usr/sbin/glusterd...(no debugging symbols >>>> found)...done. >>>> [New LWP 1109] >>>> [New LWP 1101] >>>> [New LWP 1105] >>>> [New LWP 1110] >>>> [New LWP 1099] >>>> [New LWP 1107] >>>> [New LWP 1119] >>>> [New LWP 1103] >>>> [New LWP 1112] >>>> [New LWP 1116] >>>> [New LWP 1104] >>>> [New LWP 1239] >>>> [New LWP 1106] >>>> [New LWP 1111] >>>> [New LWP 1108] >>>> [New LWP 1117] >>>> [New LWP 1102] >>>> [New LWP 1118] >>>> [New LWP 1100] >>>> [New LWP 1114] >>>> [New LWP 1113] >>>> [New LWP 1115] >>>> >>>> warning: Could not load shared library symbols for linux-vdso64.so.1. >>>> Do you need "set solib-search-path" or "set sysroot"? >>>> [Thread debugging using libthread_db enabled] >>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>> #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >>>> bytes=bytes at entry=36) at malloc.c:3327 >>>> 3327 { >>>> [Current thread is 1 (Thread 0x3fffb1689160 (LWP 1109))] >>>> (gdb) bt full >>>> >>> >> This is backtrace of one particular thread. I need output of command >> >> (gdb) thread apply all bt full >> >> >> Also, considering this is a crash in the malloc library call itself, >> would like to know the details of OS, Kernel version and gcc versions. >> >> Regards, >> Amar >> >> #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >>>> bytes=bytes at entry=36) at malloc.c:3327 >>>> nb = >>>> idx = >>>> bin = >>>> victim = >>>> size = >>>> victim_index = >>>> remainder = >>>> remainder_size = >>>> block = >>>> bit = >>>> map = >>>> fwd = >>>> bck = >>>> errstr = 0x0 >>>> __func__ = "_int_malloc" >>>> #1 0x00003fffb76a93dc in __GI___libc_malloc (bytes=36) at malloc.c:2921 >>>> ar_ptr = 0x3fffa8000020 >>>> victim = >>>> hook = >>>> __func__ = "__libc_malloc" >>>> #2 0x00003fffb7764fd0 in x_inline (xdrs=0x3fffb1686d20, len=>>> out>) at xdr_sizeof.c:89 >>>> len = 36 >>>> xdrs = 0x3fffb1686d20 >>>> #3 0x00003fffb7842488 in .xdr_gfx_iattx () from >>>> /usr/lib64/libgfxdr.so.0 >>>> No symbol table info available. >>>> #4 0x00003fffb7842e84 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> No symbol table info available. >>>> #5 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>> pp=0x3fffa81099f0, size=, proc=) at >>>> xdr_ref.c:84 >>>> loc = 0x3fffa8109aa0 "\265\256\373\200\f\206\361j" >>>> stat = >>>> #6 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>> objpp=0x3fffa81099f0, obj_size=, >>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> more_data = 1 >>>> #7 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> No symbol table info available. >>>> #8 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>> pp=0x3fffa8109870, size=, proc=) at >>>> xdr_ref.c:84 >>>> loc = 0x3fffa8109920 "\232\373\377\315\352\325\005\271" >>>> stat = >>>> #9 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>> objpp=0x3fffa8109870, obj_size=, >>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> more_data = 1 >>>> #10 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> No symbol table info available. >>>> #11 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>> pp=0x3fffa81096f0, size=, proc=) at >>>> xdr_ref.c:84 >>>> loc = 0x3fffa81097a0 "\241X\372!\216\256=\342" >>>> stat = >>>> ---Type to continue, or q to quit--- >>>> #12 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>> objpp=0x3fffa81096f0, obj_size=, >>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> more_data = 1 >>>> #13 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> No symbol table info available. >>>> #14 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>> pp=0x3fffa8109570, size=, proc=) at >>>> xdr_ref.c:84 >>>> loc = 0x3fffa8109620 "\265\205\003Vu'\002L" >>>> stat = >>>> #15 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>> objpp=0x3fffa8109570, obj_size=, >>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> more_data = 1 >>>> #16 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> No symbol table info available. >>>> #17 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>> pp=0x3fffa81093f0, size=, proc=) at >>>> xdr_ref.c:84 >>>> loc = 0x3fffa81094a0 "\200L\027F'\177\366D" >>>> stat = >>>> #18 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>> objpp=0x3fffa81093f0, obj_size=, >>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> more_data = 1 >>>> #19 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> No symbol table info available. >>>> #20 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>> pp=0x3fffa8109270, size=, proc=) at >>>> xdr_ref.c:84 >>>> loc = 0x3fffa8109320 "\217{dK(\001E\220" >>>> stat = >>>> #21 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>> objpp=0x3fffa8109270, obj_size=, >>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> more_data = 1 >>>> #22 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> No symbol table info available. >>>> #23 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>> pp=0x3fffa81090f0, size=, proc=) at >>>> xdr_ref.c:84 >>>> loc = 0x3fffa81091a0 "\217\275\067\336\232\300(\005" >>>> stat = >>>> #24 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>> objpp=0x3fffa81090f0, obj_size=, >>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> more_data = 1 >>>> #25 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> No symbol table info available. >>>> #26 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>> pp=0x3fffa8108f70, size=, proc=) at >>>> xdr_ref.c:84 >>>> loc = 0x3fffa8109020 "\260.\025\b\244\352IT" >>>> stat = >>>> #27 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>> objpp=0x3fffa8108f70, obj_size=, >>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>> xdr_ref.c:135 >>>> more_data = 1 >>>> #28 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>> /usr/lib64/libgfxdr.so.0 >>>> No symbol table info available. >>>> #29 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>> pp=0x3fffa8108df0, size=, proc=) at >>>> xdr_ref.c:84 >>>> loc = 0x3fffa8108ea0 "\212GS\203l\035\n\\" >>>> ---Type to continue, or q to quit--- >>>> >>>> >>>> Regards, >>>> Abhishek >>>> >>>> On Mon, Mar 11, 2019 at 7:10 PM Amar Tumballi Suryanarayan < >>>> atumball at redhat.com> wrote: >>>> >>>>> Hi Abhishek, >>>>> >>>>> Can you check and get back to us? >>>>> >>>>> ``` >>>>> bash# ldd /usr/lib64/libglusterfs.so >>>>> bash# ldd /usr/lib64/libgfrpc.so >>>>> >>>>> ``` >>>>> >>>>> Also considering you have the core, can you do `(gdb) thr apply all bt >>>>> full` and pass it on? >>>>> >>>>> Thanks & Regards, >>>>> Amar >>>>> >>>>> On Mon, Mar 11, 2019 at 3:41 PM ABHISHEK PALIWAL < >>>>> abhishpaliwal at gmail.com> wrote: >>>>> >>>>>> Hi Team, >>>>>> >>>>>> COuld you please provide some pointer to debug it further. >>>>>> >>>>>> Regards, >>>>>> Abhishek >>>>>> >>>>>> On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL < >>>>>> abhishpaliwal at gmail.com> wrote: >>>>>> >>>>>>> Hi Team, >>>>>>> >>>>>>> I am using Glusterfs 5.4, where after setting the gluster mount >>>>>>> point when trying to access it, glusterfsd is getting crashed and mount >>>>>>> point through the "Transport endpoint is not connected error. >>>>>>> >>>>>>> Here I are the gdb log for the core file >>>>>>> >>>>>>> warning: Could not load shared library symbols for linux-vdso64.so.1. >>>>>>> Do you need "set solib-search-path" or "set sysroot"? >>>>>>> [Thread debugging using libthread_db enabled] >>>>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>>>>>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>>>>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>>> 3327 { >>>>>>> [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))] >>>>>>> (gdb) >>>>>>> (gdb) >>>>>>> (gdb) bt >>>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>>> #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at >>>>>>> malloc.c:2921 >>>>>>> #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, >>>>>>> len=) at xdr_sizeof.c:89 >>>>>>> #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> #5 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>> pp=0x3fff7c132020, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>> objpp=0x3fff7c132020, obj_size=, >>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> #8 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>> pp=0x3fff7c131ea0, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>> objpp=0x3fff7c131ea0, obj_size=, >>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> #11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>> pp=0x3fff7c131d20, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>> objpp=0x3fff7c131d20, obj_size=, >>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> #14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>> pp=0x3fff7c131ba0, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>> objpp=0x3fff7c131ba0, obj_size=, >>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> #17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>> pp=0x3fff7c131a20, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>> objpp=0x3fff7c131a20, obj_size=, >>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> #20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>> pp=0x3fff7c1318a0, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>> objpp=0x3fff7c1318a0, obj_size=, >>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> #23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>> pp=0x3fff7c131720, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>> objpp=0x3fff7c131720, obj_size=, >>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> #26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>> pp=0x3fff7c1315a0, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>> objpp=0x3fff7c1315a0, obj_size=, >>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> #29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>> pp=0x3fff7c131420, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>> objpp=0x3fff7c131420, obj_size=, >>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> #32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>> pp=0x3fff7c1312a0, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>> objpp=0x3fff7c1312a0, obj_size=, >>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> >>>>>>> Frames are getting repeated, could any one please me. >>>>>>> -- >>>>>>> Regards >>>>>>> Abhishek Paliwal >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Regards >>>>>> Abhishek Paliwal >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>>> >>>>> >>>>> -- >>>>> Amar Tumballi (amarts) >>>>> >>>> >>>> >>>> -- >>>> >>>> >>>> >>>> >>>> Regards >>>> Abhishek Paliwal >>>> >>> >>> >>> -- >>> >>> >>> >>> >>> Regards >>> Abhishek Paliwal >>> >> >> >> -- >> Amar Tumballi (amarts) >> > > > -- > > > > > Regards > Abhishek Paliwal > -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhishpaliwal at gmail.com Wed Mar 13 05:24:32 2019 From: abhishpaliwal at gmail.com (ABHISHEK PALIWAL) Date: Wed, 13 Mar 2019 10:54:32 +0530 Subject: [Gluster-devel] [Gluster-users] Glusterfsd crashed with SIGSEGV In-Reply-To: References: Message-ID: Hi Amar, this problem seems to be configuration issue due to librpc. Could you please let me know what should be configuration I need to use? Regards, Abhishek On Wed, Mar 13, 2019 at 10:42 AM ABHISHEK PALIWAL wrote: > logs for libgfrpc.so > > pabhishe at arn-build3$ldd > ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.* > ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.0: > not a dynamic executable > ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.0.0.1: > not a dynamic executable > > > On Wed, Mar 13, 2019 at 10:02 AM ABHISHEK PALIWAL > wrote: > >> Here are the logs: >> >> >> pabhishe at arn-build3$ldd >> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.* >> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0: >> not a dynamic executable >> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0.0.1: >> not a dynamic executable >> pabhishe at arn-build3$ldd >> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0.0.1 >> not a dynamic executable >> >> >> For backtraces I have attached the core_logs.txt file. >> >> Regards, >> Abhishek >> >> On Wed, Mar 13, 2019 at 9:51 AM Amar Tumballi Suryanarayan < >> atumball at redhat.com> wrote: >> >>> Hi Abhishek, >>> >>> Few more questions, >>> >>> >>>> On Tue, Mar 12, 2019 at 10:58 AM ABHISHEK PALIWAL < >>>> abhishpaliwal at gmail.com> wrote: >>>> >>>>> Hi Amar, >>>>> >>>>> Below are the requested logs >>>>> >>>>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libglusterfs.so >>>>> not a dynamic executable >>>>> >>>>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libgfrpc.so >>>>> not a dynamic executable >>>>> >>>>> >>> Can you please add a * at the end, so it gets the linked library list >>> from the actual files (ideally this is a symlink, but I expected it to >>> resolve like in Fedora). >>> >>> >>> >>>> root at 128:/# gdb /usr/sbin/glusterd core.1099 >>>>> GNU gdb (GDB) 7.10.1 >>>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>>> License GPLv3+: GNU GPL version 3 or later < >>>>> http://gnu.org/licenses/gpl.html> >>>>> This is free software: you are free to change and redistribute it. >>>>> There is NO WARRANTY, to the extent permitted by law. Type "show >>>>> copying" >>>>> and "show warranty" for details. >>>>> This GDB was configured as "powerpc64-wrs-linux". >>>>> Type "show configuration" for configuration details. >>>>> For bug reporting instructions, please see: >>>>> . >>>>> Find the GDB manual and other documentation resources online at: >>>>> . >>>>> For help, type "help". >>>>> Type "apropos word" to search for commands related to "word"... >>>>> Reading symbols from /usr/sbin/glusterd...(no debugging symbols >>>>> found)...done. >>>>> [New LWP 1109] >>>>> [New LWP 1101] >>>>> [New LWP 1105] >>>>> [New LWP 1110] >>>>> [New LWP 1099] >>>>> [New LWP 1107] >>>>> [New LWP 1119] >>>>> [New LWP 1103] >>>>> [New LWP 1112] >>>>> [New LWP 1116] >>>>> [New LWP 1104] >>>>> [New LWP 1239] >>>>> [New LWP 1106] >>>>> [New LWP 1111] >>>>> [New LWP 1108] >>>>> [New LWP 1117] >>>>> [New LWP 1102] >>>>> [New LWP 1118] >>>>> [New LWP 1100] >>>>> [New LWP 1114] >>>>> [New LWP 1113] >>>>> [New LWP 1115] >>>>> >>>>> warning: Could not load shared library symbols for linux-vdso64.so.1. >>>>> Do you need "set solib-search-path" or "set sysroot"? >>>>> [Thread debugging using libthread_db enabled] >>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>>>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>>> #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>> 3327 { >>>>> [Current thread is 1 (Thread 0x3fffb1689160 (LWP 1109))] >>>>> (gdb) bt full >>>>> >>>> >>> This is backtrace of one particular thread. I need output of command >>> >>> (gdb) thread apply all bt full >>> >>> >>> Also, considering this is a crash in the malloc library call itself, >>> would like to know the details of OS, Kernel version and gcc versions. >>> >>> Regards, >>> Amar >>> >>> #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>> nb = >>>>> idx = >>>>> bin = >>>>> victim = >>>>> size = >>>>> victim_index = >>>>> remainder = >>>>> remainder_size = >>>>> block = >>>>> bit = >>>>> map = >>>>> fwd = >>>>> bck = >>>>> errstr = 0x0 >>>>> __func__ = "_int_malloc" >>>>> #1 0x00003fffb76a93dc in __GI___libc_malloc (bytes=36) at >>>>> malloc.c:2921 >>>>> ar_ptr = 0x3fffa8000020 >>>>> victim = >>>>> hook = >>>>> __func__ = "__libc_malloc" >>>>> #2 0x00003fffb7764fd0 in x_inline (xdrs=0x3fffb1686d20, >>>>> len=) at xdr_sizeof.c:89 >>>>> len = 36 >>>>> xdrs = 0x3fffb1686d20 >>>>> #3 0x00003fffb7842488 in .xdr_gfx_iattx () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> No symbol table info available. >>>>> #4 0x00003fffb7842e84 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> No symbol table info available. >>>>> #5 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>> pp=0x3fffa81099f0, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> loc = 0x3fffa8109aa0 "\265\256\373\200\f\206\361j" >>>>> stat = >>>>> #6 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>> objpp=0x3fffa81099f0, obj_size=, >>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> more_data = 1 >>>>> #7 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> No symbol table info available. >>>>> #8 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>> pp=0x3fffa8109870, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> loc = 0x3fffa8109920 "\232\373\377\315\352\325\005\271" >>>>> stat = >>>>> #9 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>> objpp=0x3fffa8109870, obj_size=, >>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> more_data = 1 >>>>> #10 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> No symbol table info available. >>>>> #11 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>> pp=0x3fffa81096f0, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> loc = 0x3fffa81097a0 "\241X\372!\216\256=\342" >>>>> stat = >>>>> ---Type to continue, or q to quit--- >>>>> #12 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>> objpp=0x3fffa81096f0, obj_size=, >>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> more_data = 1 >>>>> #13 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> No symbol table info available. >>>>> #14 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>> pp=0x3fffa8109570, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> loc = 0x3fffa8109620 "\265\205\003Vu'\002L" >>>>> stat = >>>>> #15 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>> objpp=0x3fffa8109570, obj_size=, >>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> more_data = 1 >>>>> #16 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> No symbol table info available. >>>>> #17 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>> pp=0x3fffa81093f0, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> loc = 0x3fffa81094a0 "\200L\027F'\177\366D" >>>>> stat = >>>>> #18 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>> objpp=0x3fffa81093f0, obj_size=, >>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> more_data = 1 >>>>> #19 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> No symbol table info available. >>>>> #20 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>> pp=0x3fffa8109270, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> loc = 0x3fffa8109320 "\217{dK(\001E\220" >>>>> stat = >>>>> #21 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>> objpp=0x3fffa8109270, obj_size=, >>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> more_data = 1 >>>>> #22 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> No symbol table info available. >>>>> #23 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>> pp=0x3fffa81090f0, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> loc = 0x3fffa81091a0 "\217\275\067\336\232\300(\005" >>>>> stat = >>>>> #24 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>> objpp=0x3fffa81090f0, obj_size=, >>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> more_data = 1 >>>>> #25 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> No symbol table info available. >>>>> #26 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>> pp=0x3fffa8108f70, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> loc = 0x3fffa8109020 "\260.\025\b\244\352IT" >>>>> stat = >>>>> #27 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>> objpp=0x3fffa8108f70, obj_size=, >>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>> xdr_ref.c:135 >>>>> more_data = 1 >>>>> #28 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>> /usr/lib64/libgfxdr.so.0 >>>>> No symbol table info available. >>>>> #29 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>> pp=0x3fffa8108df0, size=, proc=) at >>>>> xdr_ref.c:84 >>>>> loc = 0x3fffa8108ea0 "\212GS\203l\035\n\\" >>>>> ---Type to continue, or q to quit--- >>>>> >>>>> >>>>> Regards, >>>>> Abhishek >>>>> >>>>> On Mon, Mar 11, 2019 at 7:10 PM Amar Tumballi Suryanarayan < >>>>> atumball at redhat.com> wrote: >>>>> >>>>>> Hi Abhishek, >>>>>> >>>>>> Can you check and get back to us? >>>>>> >>>>>> ``` >>>>>> bash# ldd /usr/lib64/libglusterfs.so >>>>>> bash# ldd /usr/lib64/libgfrpc.so >>>>>> >>>>>> ``` >>>>>> >>>>>> Also considering you have the core, can you do `(gdb) thr apply all >>>>>> bt full` and pass it on? >>>>>> >>>>>> Thanks & Regards, >>>>>> Amar >>>>>> >>>>>> On Mon, Mar 11, 2019 at 3:41 PM ABHISHEK PALIWAL < >>>>>> abhishpaliwal at gmail.com> wrote: >>>>>> >>>>>>> Hi Team, >>>>>>> >>>>>>> COuld you please provide some pointer to debug it further. >>>>>>> >>>>>>> Regards, >>>>>>> Abhishek >>>>>>> >>>>>>> On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL < >>>>>>> abhishpaliwal at gmail.com> wrote: >>>>>>> >>>>>>>> Hi Team, >>>>>>>> >>>>>>>> I am using Glusterfs 5.4, where after setting the gluster mount >>>>>>>> point when trying to access it, glusterfsd is getting crashed and mount >>>>>>>> point through the "Transport endpoint is not connected error. >>>>>>>> >>>>>>>> Here I are the gdb log for the core file >>>>>>>> >>>>>>>> warning: Could not load shared library symbols for >>>>>>>> linux-vdso64.so.1. >>>>>>>> Do you need "set solib-search-path" or "set sysroot"? >>>>>>>> [Thread debugging using libthread_db enabled] >>>>>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>>>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>>>>>>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>>>> 3327 { >>>>>>>> [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))] >>>>>>>> (gdb) >>>>>>>> (gdb) >>>>>>>> (gdb) bt >>>>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>>>> #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at >>>>>>>> malloc.c:2921 >>>>>>>> #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, >>>>>>>> len=) at xdr_sizeof.c:89 >>>>>>>> #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from >>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>> #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from >>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>> #5 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>> pp=0x3fff7c132020, size=, proc=) at >>>>>>>> xdr_ref.c:84 >>>>>>>> #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>> objpp=0x3fff7c132020, obj_size=, >>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>>> xdr_ref.c:135 >>>>>>>> #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>> #8 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>> pp=0x3fff7c131ea0, size=, proc=) at >>>>>>>> xdr_ref.c:84 >>>>>>>> #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>> objpp=0x3fff7c131ea0, obj_size=, >>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>>> xdr_ref.c:135 >>>>>>>> #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>> #11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>> pp=0x3fff7c131d20, size=, proc=) at >>>>>>>> xdr_ref.c:84 >>>>>>>> #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>> objpp=0x3fff7c131d20, obj_size=, >>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>>> xdr_ref.c:135 >>>>>>>> #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>> #14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>> pp=0x3fff7c131ba0, size=, proc=) at >>>>>>>> xdr_ref.c:84 >>>>>>>> #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>> objpp=0x3fff7c131ba0, obj_size=, >>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>>> xdr_ref.c:135 >>>>>>>> #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>> #17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>> pp=0x3fff7c131a20, size=, proc=) at >>>>>>>> xdr_ref.c:84 >>>>>>>> #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>> objpp=0x3fff7c131a20, obj_size=, >>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>>> xdr_ref.c:135 >>>>>>>> #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>> #20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>> pp=0x3fff7c1318a0, size=, proc=) at >>>>>>>> xdr_ref.c:84 >>>>>>>> #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>> objpp=0x3fff7c1318a0, obj_size=, >>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>>> xdr_ref.c:135 >>>>>>>> #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>> #23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>> pp=0x3fff7c131720, size=, proc=) at >>>>>>>> xdr_ref.c:84 >>>>>>>> #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>> objpp=0x3fff7c131720, obj_size=, >>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>>> xdr_ref.c:135 >>>>>>>> #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>> #26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>> pp=0x3fff7c1315a0, size=, proc=) at >>>>>>>> xdr_ref.c:84 >>>>>>>> #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>> objpp=0x3fff7c1315a0, obj_size=, >>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>>> xdr_ref.c:135 >>>>>>>> #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>> #29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>> pp=0x3fff7c131420, size=, proc=) at >>>>>>>> xdr_ref.c:84 >>>>>>>> #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>> objpp=0x3fff7c131420, obj_size=, >>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>>> xdr_ref.c:135 >>>>>>>> #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>> #32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>> pp=0x3fff7c1312a0, size=, proc=) at >>>>>>>> xdr_ref.c:84 >>>>>>>> #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>> objpp=0x3fff7c1312a0, obj_size=, >>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at >>>>>>>> xdr_ref.c:135 >>>>>>>> >>>>>>>> Frames are getting repeated, could any one please me. >>>>>>>> -- >>>>>>>> Regards >>>>>>>> Abhishek Paliwal >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards >>>>>>> Abhishek Paliwal >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Amar Tumballi (amarts) >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> >>>>> >>>>> >>>>> Regards >>>>> Abhishek Paliwal >>>>> >>>> >>>> >>>> -- >>>> >>>> >>>> >>>> >>>> Regards >>>> Abhishek Paliwal >>>> >>> >>> >>> -- >>> Amar Tumballi (amarts) >>> >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> > > > -- > > > > > Regards > Abhishek Paliwal > -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: From atumball at redhat.com Wed Mar 13 06:25:23 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Wed, 13 Mar 2019 11:55:23 +0530 Subject: [Gluster-devel] [Gluster-users] Glusterfsd crashed with SIGSEGV In-Reply-To: References: Message-ID: We recommend to use 'tirpc' in the later releases. use '--with-tirpc' while running ./configure On Wed, Mar 13, 2019 at 10:55 AM ABHISHEK PALIWAL wrote: > Hi Amar, > > this problem seems to be configuration issue due to librpc. > > Could you please let me know what should be configuration I need to use? > > Regards, > Abhishek > > On Wed, Mar 13, 2019 at 10:42 AM ABHISHEK PALIWAL > wrote: > >> logs for libgfrpc.so >> >> pabhishe at arn-build3$ldd >> ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.* >> ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.0: >> not a dynamic executable >> ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.0.0.1: >> not a dynamic executable >> >> >> On Wed, Mar 13, 2019 at 10:02 AM ABHISHEK PALIWAL < >> abhishpaliwal at gmail.com> wrote: >> >>> Here are the logs: >>> >>> >>> pabhishe at arn-build3$ldd >>> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.* >>> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0: >>> not a dynamic executable >>> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0.0.1: >>> not a dynamic executable >>> pabhishe at arn-build3$ldd >>> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0.0.1 >>> not a dynamic executable >>> >>> >>> For backtraces I have attached the core_logs.txt file. >>> >>> Regards, >>> Abhishek >>> >>> On Wed, Mar 13, 2019 at 9:51 AM Amar Tumballi Suryanarayan < >>> atumball at redhat.com> wrote: >>> >>>> Hi Abhishek, >>>> >>>> Few more questions, >>>> >>>> >>>>> On Tue, Mar 12, 2019 at 10:58 AM ABHISHEK PALIWAL < >>>>> abhishpaliwal at gmail.com> wrote: >>>>> >>>>>> Hi Amar, >>>>>> >>>>>> Below are the requested logs >>>>>> >>>>>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libglusterfs.so >>>>>> not a dynamic executable >>>>>> >>>>>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libgfrpc.so >>>>>> not a dynamic executable >>>>>> >>>>>> >>>> Can you please add a * at the end, so it gets the linked library list >>>> from the actual files (ideally this is a symlink, but I expected it to >>>> resolve like in Fedora). >>>> >>>> >>>> >>>>> root at 128:/# gdb /usr/sbin/glusterd core.1099 >>>>>> GNU gdb (GDB) 7.10.1 >>>>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>>>> License GPLv3+: GNU GPL version 3 or later < >>>>>> http://gnu.org/licenses/gpl.html> >>>>>> This is free software: you are free to change and redistribute it. >>>>>> There is NO WARRANTY, to the extent permitted by law. Type "show >>>>>> copying" >>>>>> and "show warranty" for details. >>>>>> This GDB was configured as "powerpc64-wrs-linux". >>>>>> Type "show configuration" for configuration details. >>>>>> For bug reporting instructions, please see: >>>>>> . >>>>>> Find the GDB manual and other documentation resources online at: >>>>>> . >>>>>> For help, type "help". >>>>>> Type "apropos word" to search for commands related to "word"... >>>>>> Reading symbols from /usr/sbin/glusterd...(no debugging symbols >>>>>> found)...done. >>>>>> [New LWP 1109] >>>>>> [New LWP 1101] >>>>>> [New LWP 1105] >>>>>> [New LWP 1110] >>>>>> [New LWP 1099] >>>>>> [New LWP 1107] >>>>>> [New LWP 1119] >>>>>> [New LWP 1103] >>>>>> [New LWP 1112] >>>>>> [New LWP 1116] >>>>>> [New LWP 1104] >>>>>> [New LWP 1239] >>>>>> [New LWP 1106] >>>>>> [New LWP 1111] >>>>>> [New LWP 1108] >>>>>> [New LWP 1117] >>>>>> [New LWP 1102] >>>>>> [New LWP 1118] >>>>>> [New LWP 1100] >>>>>> [New LWP 1114] >>>>>> [New LWP 1113] >>>>>> [New LWP 1115] >>>>>> >>>>>> warning: Could not load shared library symbols for linux-vdso64.so.1. >>>>>> Do you need "set solib-search-path" or "set sysroot"? >>>>>> [Thread debugging using libthread_db enabled] >>>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>>>>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>>>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>>>> #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>> 3327 { >>>>>> [Current thread is 1 (Thread 0x3fffb1689160 (LWP 1109))] >>>>>> (gdb) bt full >>>>>> >>>>> >>>> This is backtrace of one particular thread. I need output of command >>>> >>>> (gdb) thread apply all bt full >>>> >>>> >>>> Also, considering this is a crash in the malloc library call itself, >>>> would like to know the details of OS, Kernel version and gcc versions. >>>> >>>> Regards, >>>> Amar >>>> >>>> #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>> nb = >>>>>> idx = >>>>>> bin = >>>>>> victim = >>>>>> size = >>>>>> victim_index = >>>>>> remainder = >>>>>> remainder_size = >>>>>> block = >>>>>> bit = >>>>>> map = >>>>>> fwd = >>>>>> bck = >>>>>> errstr = 0x0 >>>>>> __func__ = "_int_malloc" >>>>>> #1 0x00003fffb76a93dc in __GI___libc_malloc (bytes=36) at >>>>>> malloc.c:2921 >>>>>> ar_ptr = 0x3fffa8000020 >>>>>> victim = >>>>>> hook = >>>>>> __func__ = "__libc_malloc" >>>>>> #2 0x00003fffb7764fd0 in x_inline (xdrs=0x3fffb1686d20, >>>>>> len=) at xdr_sizeof.c:89 >>>>>> len = 36 >>>>>> xdrs = 0x3fffb1686d20 >>>>>> #3 0x00003fffb7842488 in .xdr_gfx_iattx () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> No symbol table info available. >>>>>> #4 0x00003fffb7842e84 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> No symbol table info available. >>>>>> #5 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>> pp=0x3fffa81099f0, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> loc = 0x3fffa8109aa0 "\265\256\373\200\f\206\361j" >>>>>> stat = >>>>>> #6 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>> objpp=0x3fffa81099f0, obj_size=, >>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> more_data = 1 >>>>>> #7 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> No symbol table info available. >>>>>> #8 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>> pp=0x3fffa8109870, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> loc = 0x3fffa8109920 "\232\373\377\315\352\325\005\271" >>>>>> stat = >>>>>> #9 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>> objpp=0x3fffa8109870, obj_size=, >>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> more_data = 1 >>>>>> #10 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> No symbol table info available. >>>>>> #11 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>> pp=0x3fffa81096f0, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> loc = 0x3fffa81097a0 "\241X\372!\216\256=\342" >>>>>> stat = >>>>>> ---Type to continue, or q to quit--- >>>>>> #12 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>> objpp=0x3fffa81096f0, obj_size=, >>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> more_data = 1 >>>>>> #13 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> No symbol table info available. >>>>>> #14 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>> pp=0x3fffa8109570, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> loc = 0x3fffa8109620 "\265\205\003Vu'\002L" >>>>>> stat = >>>>>> #15 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>> objpp=0x3fffa8109570, obj_size=, >>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> more_data = 1 >>>>>> #16 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> No symbol table info available. >>>>>> #17 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>> pp=0x3fffa81093f0, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> loc = 0x3fffa81094a0 "\200L\027F'\177\366D" >>>>>> stat = >>>>>> #18 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>> objpp=0x3fffa81093f0, obj_size=, >>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> more_data = 1 >>>>>> #19 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> No symbol table info available. >>>>>> #20 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>> pp=0x3fffa8109270, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> loc = 0x3fffa8109320 "\217{dK(\001E\220" >>>>>> stat = >>>>>> #21 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>> objpp=0x3fffa8109270, obj_size=, >>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> more_data = 1 >>>>>> #22 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> No symbol table info available. >>>>>> #23 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>> pp=0x3fffa81090f0, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> loc = 0x3fffa81091a0 "\217\275\067\336\232\300(\005" >>>>>> stat = >>>>>> #24 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>> objpp=0x3fffa81090f0, obj_size=, >>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> more_data = 1 >>>>>> #25 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> No symbol table info available. >>>>>> #26 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>> pp=0x3fffa8108f70, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> loc = 0x3fffa8109020 "\260.\025\b\244\352IT" >>>>>> stat = >>>>>> #27 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>> objpp=0x3fffa8108f70, obj_size=, >>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>> xdr_ref.c:135 >>>>>> more_data = 1 >>>>>> #28 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>> /usr/lib64/libgfxdr.so.0 >>>>>> No symbol table info available. >>>>>> #29 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>> pp=0x3fffa8108df0, size=, proc=) at >>>>>> xdr_ref.c:84 >>>>>> loc = 0x3fffa8108ea0 "\212GS\203l\035\n\\" >>>>>> ---Type to continue, or q to quit--- >>>>>> >>>>>> >>>>>> Regards, >>>>>> Abhishek >>>>>> >>>>>> On Mon, Mar 11, 2019 at 7:10 PM Amar Tumballi Suryanarayan < >>>>>> atumball at redhat.com> wrote: >>>>>> >>>>>>> Hi Abhishek, >>>>>>> >>>>>>> Can you check and get back to us? >>>>>>> >>>>>>> ``` >>>>>>> bash# ldd /usr/lib64/libglusterfs.so >>>>>>> bash# ldd /usr/lib64/libgfrpc.so >>>>>>> >>>>>>> ``` >>>>>>> >>>>>>> Also considering you have the core, can you do `(gdb) thr apply all >>>>>>> bt full` and pass it on? >>>>>>> >>>>>>> Thanks & Regards, >>>>>>> Amar >>>>>>> >>>>>>> On Mon, Mar 11, 2019 at 3:41 PM ABHISHEK PALIWAL < >>>>>>> abhishpaliwal at gmail.com> wrote: >>>>>>> >>>>>>>> Hi Team, >>>>>>>> >>>>>>>> COuld you please provide some pointer to debug it further. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Abhishek >>>>>>>> >>>>>>>> On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL < >>>>>>>> abhishpaliwal at gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Team, >>>>>>>>> >>>>>>>>> I am using Glusterfs 5.4, where after setting the gluster mount >>>>>>>>> point when trying to access it, glusterfsd is getting crashed and mount >>>>>>>>> point through the "Transport endpoint is not connected error. >>>>>>>>> >>>>>>>>> Here I are the gdb log for the core file >>>>>>>>> >>>>>>>>> warning: Could not load shared library symbols for >>>>>>>>> linux-vdso64.so.1. >>>>>>>>> Do you need "set solib-search-path" or "set sysroot"? >>>>>>>>> [Thread debugging using libthread_db enabled] >>>>>>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>>>>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>>>>>>>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>>>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>>>>> 3327 { >>>>>>>>> [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))] >>>>>>>>> (gdb) >>>>>>>>> (gdb) >>>>>>>>> (gdb) bt >>>>>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>>>>> #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at >>>>>>>>> malloc.c:2921 >>>>>>>>> #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, >>>>>>>>> len=) at xdr_sizeof.c:89 >>>>>>>>> #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from >>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>> #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from >>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>> #5 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>>> pp=0x3fff7c132020, size=, proc=) at >>>>>>>>> xdr_ref.c:84 >>>>>>>>> #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>> objpp=0x3fff7c132020, obj_size=, >>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>> at xdr_ref.c:135 >>>>>>>>> #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>> #8 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>>> pp=0x3fff7c131ea0, size=, proc=) at >>>>>>>>> xdr_ref.c:84 >>>>>>>>> #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>> objpp=0x3fff7c131ea0, obj_size=, >>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>> at xdr_ref.c:135 >>>>>>>>> #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>> #11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>>> pp=0x3fff7c131d20, size=, proc=) at >>>>>>>>> xdr_ref.c:84 >>>>>>>>> #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>> objpp=0x3fff7c131d20, obj_size=, >>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>> at xdr_ref.c:135 >>>>>>>>> #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>> #14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>>> pp=0x3fff7c131ba0, size=, proc=) at >>>>>>>>> xdr_ref.c:84 >>>>>>>>> #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>> objpp=0x3fff7c131ba0, obj_size=, >>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>> at xdr_ref.c:135 >>>>>>>>> #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>> #17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>>> pp=0x3fff7c131a20, size=, proc=) at >>>>>>>>> xdr_ref.c:84 >>>>>>>>> #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>> objpp=0x3fff7c131a20, obj_size=, >>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>> at xdr_ref.c:135 >>>>>>>>> #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>> #20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>>> pp=0x3fff7c1318a0, size=, proc=) at >>>>>>>>> xdr_ref.c:84 >>>>>>>>> #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>> objpp=0x3fff7c1318a0, obj_size=, >>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>> at xdr_ref.c:135 >>>>>>>>> #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>> #23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>>> pp=0x3fff7c131720, size=, proc=) at >>>>>>>>> xdr_ref.c:84 >>>>>>>>> #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>> objpp=0x3fff7c131720, obj_size=, >>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>> at xdr_ref.c:135 >>>>>>>>> #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>> #26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>>> pp=0x3fff7c1315a0, size=, proc=) at >>>>>>>>> xdr_ref.c:84 >>>>>>>>> #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>> objpp=0x3fff7c1315a0, obj_size=, >>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>> at xdr_ref.c:135 >>>>>>>>> #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>> #29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>>> pp=0x3fff7c131420, size=, proc=) at >>>>>>>>> xdr_ref.c:84 >>>>>>>>> #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>> objpp=0x3fff7c131420, obj_size=, >>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>> at xdr_ref.c:135 >>>>>>>>> #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>> #32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20, >>>>>>>>> pp=0x3fff7c1312a0, size=, proc=) at >>>>>>>>> xdr_ref.c:84 >>>>>>>>> #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>> objpp=0x3fff7c1312a0, obj_size=, >>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>> at xdr_ref.c:135 >>>>>>>>> >>>>>>>>> Frames are getting repeated, could any one please me. >>>>>>>>> -- >>>>>>>>> Regards >>>>>>>>> Abhishek Paliwal >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Regards >>>>>>>> Abhishek Paliwal >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Amar Tumballi (amarts) >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Regards >>>>>> Abhishek Paliwal >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> >>>>> >>>>> >>>>> Regards >>>>> Abhishek Paliwal >>>>> >>>> >>>> >>>> -- >>>> Amar Tumballi (amarts) >>>> >>> >>> >>> -- >>> >>> >>> >>> >>> Regards >>> Abhishek Paliwal >>> >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> > > > -- > > > > > Regards > Abhishek Paliwal > -- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhishpaliwal at gmail.com Wed Mar 13 07:20:53 2019 From: abhishpaliwal at gmail.com (ABHISHEK PALIWAL) Date: Wed, 13 Mar 2019 12:50:53 +0530 Subject: [Gluster-devel] [Gluster-users] Glusterfsd crashed with SIGSEGV In-Reply-To: References: Message-ID: are you sure its '--witn-tirpc'? as with it i am getting WARNING: QA Issue: glusterfs: configure was passed unrecognised options: --with-tirpc [unknown-configure-option] also I tried with '--with-libtirpc' but result was same. Regards, Abhishek On Wed, Mar 13, 2019 at 11:56 AM Amar Tumballi Suryanarayan < atumball at redhat.com> wrote: > We recommend to use 'tirpc' in the later releases. use '--with-tirpc' > while running ./configure > > On Wed, Mar 13, 2019 at 10:55 AM ABHISHEK PALIWAL > wrote: > >> Hi Amar, >> >> this problem seems to be configuration issue due to librpc. >> >> Could you please let me know what should be configuration I need to use? >> >> Regards, >> Abhishek >> >> On Wed, Mar 13, 2019 at 10:42 AM ABHISHEK PALIWAL < >> abhishpaliwal at gmail.com> wrote: >> >>> logs for libgfrpc.so >>> >>> pabhishe at arn-build3$ldd >>> ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.* >>> ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.0: >>> not a dynamic executable >>> ./5.4-r0/packages-split/glusterfs/usr/lib64/libgfrpc.so.0.0.1: >>> not a dynamic executable >>> >>> >>> On Wed, Mar 13, 2019 at 10:02 AM ABHISHEK PALIWAL < >>> abhishpaliwal at gmail.com> wrote: >>> >>>> Here are the logs: >>>> >>>> >>>> pabhishe at arn-build3$ldd >>>> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.* >>>> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0: >>>> not a dynamic executable >>>> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0.0.1: >>>> not a dynamic executable >>>> pabhishe at arn-build3$ldd >>>> ./5.4-r0/sysroot-destdir/usr/lib64/libglusterfs.so.0.0.1 >>>> not a dynamic executable >>>> >>>> >>>> For backtraces I have attached the core_logs.txt file. >>>> >>>> Regards, >>>> Abhishek >>>> >>>> On Wed, Mar 13, 2019 at 9:51 AM Amar Tumballi Suryanarayan < >>>> atumball at redhat.com> wrote: >>>> >>>>> Hi Abhishek, >>>>> >>>>> Few more questions, >>>>> >>>>> >>>>>> On Tue, Mar 12, 2019 at 10:58 AM ABHISHEK PALIWAL < >>>>>> abhishpaliwal at gmail.com> wrote: >>>>>> >>>>>>> Hi Amar, >>>>>>> >>>>>>> Below are the requested logs >>>>>>> >>>>>>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libglusterfs.so >>>>>>> not a dynamic executable >>>>>>> >>>>>>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libgfrpc.so >>>>>>> not a dynamic executable >>>>>>> >>>>>>> >>>>> Can you please add a * at the end, so it gets the linked library list >>>>> from the actual files (ideally this is a symlink, but I expected it to >>>>> resolve like in Fedora). >>>>> >>>>> >>>>> >>>>>> root at 128:/# gdb /usr/sbin/glusterd core.1099 >>>>>>> GNU gdb (GDB) 7.10.1 >>>>>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>>>>> License GPLv3+: GNU GPL version 3 or later < >>>>>>> http://gnu.org/licenses/gpl.html> >>>>>>> This is free software: you are free to change and redistribute it. >>>>>>> There is NO WARRANTY, to the extent permitted by law. Type "show >>>>>>> copying" >>>>>>> and "show warranty" for details. >>>>>>> This GDB was configured as "powerpc64-wrs-linux". >>>>>>> Type "show configuration" for configuration details. >>>>>>> For bug reporting instructions, please see: >>>>>>> . >>>>>>> Find the GDB manual and other documentation resources online at: >>>>>>> . >>>>>>> For help, type "help". >>>>>>> Type "apropos word" to search for commands related to "word"... >>>>>>> Reading symbols from /usr/sbin/glusterd...(no debugging symbols >>>>>>> found)...done. >>>>>>> [New LWP 1109] >>>>>>> [New LWP 1101] >>>>>>> [New LWP 1105] >>>>>>> [New LWP 1110] >>>>>>> [New LWP 1099] >>>>>>> [New LWP 1107] >>>>>>> [New LWP 1119] >>>>>>> [New LWP 1103] >>>>>>> [New LWP 1112] >>>>>>> [New LWP 1116] >>>>>>> [New LWP 1104] >>>>>>> [New LWP 1239] >>>>>>> [New LWP 1106] >>>>>>> [New LWP 1111] >>>>>>> [New LWP 1108] >>>>>>> [New LWP 1117] >>>>>>> [New LWP 1102] >>>>>>> [New LWP 1118] >>>>>>> [New LWP 1100] >>>>>>> [New LWP 1114] >>>>>>> [New LWP 1113] >>>>>>> [New LWP 1115] >>>>>>> >>>>>>> warning: Could not load shared library symbols for linux-vdso64.so.1. >>>>>>> Do you need "set solib-search-path" or "set sysroot"? >>>>>>> [Thread debugging using libthread_db enabled] >>>>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>>>>>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>>>>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>>>>> #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >>>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>>> 3327 { >>>>>>> [Current thread is 1 (Thread 0x3fffb1689160 (LWP 1109))] >>>>>>> (gdb) bt full >>>>>>> >>>>>> >>>>> This is backtrace of one particular thread. I need output of command >>>>> >>>>> (gdb) thread apply all bt full >>>>> >>>>> >>>>> Also, considering this is a crash in the malloc library call itself, >>>>> would like to know the details of OS, Kernel version and gcc versions. >>>>> >>>>> Regards, >>>>> Amar >>>>> >>>>> #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020, >>>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>>> nb = >>>>>>> idx = >>>>>>> bin = >>>>>>> victim = >>>>>>> size = >>>>>>> victim_index = >>>>>>> remainder = >>>>>>> remainder_size = >>>>>>> block = >>>>>>> bit = >>>>>>> map = >>>>>>> fwd = >>>>>>> bck = >>>>>>> errstr = 0x0 >>>>>>> __func__ = "_int_malloc" >>>>>>> #1 0x00003fffb76a93dc in __GI___libc_malloc (bytes=36) at >>>>>>> malloc.c:2921 >>>>>>> ar_ptr = 0x3fffa8000020 >>>>>>> victim = >>>>>>> hook = >>>>>>> __func__ = "__libc_malloc" >>>>>>> #2 0x00003fffb7764fd0 in x_inline (xdrs=0x3fffb1686d20, >>>>>>> len=) at xdr_sizeof.c:89 >>>>>>> len = 36 >>>>>>> xdrs = 0x3fffb1686d20 >>>>>>> #3 0x00003fffb7842488 in .xdr_gfx_iattx () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> No symbol table info available. >>>>>>> #4 0x00003fffb7842e84 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> No symbol table info available. >>>>>>> #5 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>>> pp=0x3fffa81099f0, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> loc = 0x3fffa8109aa0 "\265\256\373\200\f\206\361j" >>>>>>> stat = >>>>>>> #6 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>>> objpp=0x3fffa81099f0, obj_size=, >>>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> more_data = 1 >>>>>>> #7 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> No symbol table info available. >>>>>>> #8 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>>> pp=0x3fffa8109870, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> loc = 0x3fffa8109920 "\232\373\377\315\352\325\005\271" >>>>>>> stat = >>>>>>> #9 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>>> objpp=0x3fffa8109870, obj_size=, >>>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> more_data = 1 >>>>>>> #10 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> No symbol table info available. >>>>>>> #11 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>>> pp=0x3fffa81096f0, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> loc = 0x3fffa81097a0 "\241X\372!\216\256=\342" >>>>>>> stat = >>>>>>> ---Type to continue, or q to quit--- >>>>>>> #12 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>>> objpp=0x3fffa81096f0, obj_size=, >>>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> more_data = 1 >>>>>>> #13 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> No symbol table info available. >>>>>>> #14 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>>> pp=0x3fffa8109570, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> loc = 0x3fffa8109620 "\265\205\003Vu'\002L" >>>>>>> stat = >>>>>>> #15 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>>> objpp=0x3fffa8109570, obj_size=, >>>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> more_data = 1 >>>>>>> #16 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> No symbol table info available. >>>>>>> #17 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>>> pp=0x3fffa81093f0, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> loc = 0x3fffa81094a0 "\200L\027F'\177\366D" >>>>>>> stat = >>>>>>> #18 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>>> objpp=0x3fffa81093f0, obj_size=, >>>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> more_data = 1 >>>>>>> #19 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> No symbol table info available. >>>>>>> #20 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>>> pp=0x3fffa8109270, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> loc = 0x3fffa8109320 "\217{dK(\001E\220" >>>>>>> stat = >>>>>>> #21 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>>> objpp=0x3fffa8109270, obj_size=, >>>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> more_data = 1 >>>>>>> #22 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> No symbol table info available. >>>>>>> #23 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>>> pp=0x3fffa81090f0, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> loc = 0x3fffa81091a0 "\217\275\067\336\232\300(\005" >>>>>>> stat = >>>>>>> #24 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>>> objpp=0x3fffa81090f0, obj_size=, >>>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> more_data = 1 >>>>>>> #25 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> No symbol table info available. >>>>>>> #26 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>>> pp=0x3fffa8108f70, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> loc = 0x3fffa8109020 "\260.\025\b\244\352IT" >>>>>>> stat = >>>>>>> #27 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20, >>>>>>> objpp=0x3fffa8108f70, obj_size=, >>>>>>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at >>>>>>> xdr_ref.c:135 >>>>>>> more_data = 1 >>>>>>> #28 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from >>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>> No symbol table info available. >>>>>>> #29 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20, >>>>>>> pp=0x3fffa8108df0, size=, proc=) at >>>>>>> xdr_ref.c:84 >>>>>>> loc = 0x3fffa8108ea0 "\212GS\203l\035\n\\" >>>>>>> ---Type to continue, or q to quit--- >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Abhishek >>>>>>> >>>>>>> On Mon, Mar 11, 2019 at 7:10 PM Amar Tumballi Suryanarayan < >>>>>>> atumball at redhat.com> wrote: >>>>>>> >>>>>>>> Hi Abhishek, >>>>>>>> >>>>>>>> Can you check and get back to us? >>>>>>>> >>>>>>>> ``` >>>>>>>> bash# ldd /usr/lib64/libglusterfs.so >>>>>>>> bash# ldd /usr/lib64/libgfrpc.so >>>>>>>> >>>>>>>> ``` >>>>>>>> >>>>>>>> Also considering you have the core, can you do `(gdb) thr apply all >>>>>>>> bt full` and pass it on? >>>>>>>> >>>>>>>> Thanks & Regards, >>>>>>>> Amar >>>>>>>> >>>>>>>> On Mon, Mar 11, 2019 at 3:41 PM ABHISHEK PALIWAL < >>>>>>>> abhishpaliwal at gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Team, >>>>>>>>> >>>>>>>>> COuld you please provide some pointer to debug it further. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Abhishek >>>>>>>>> >>>>>>>>> On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL < >>>>>>>>> abhishpaliwal at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Team, >>>>>>>>>> >>>>>>>>>> I am using Glusterfs 5.4, where after setting the gluster mount >>>>>>>>>> point when trying to access it, glusterfsd is getting crashed and mount >>>>>>>>>> point through the "Transport endpoint is not connected error. >>>>>>>>>> >>>>>>>>>> Here I are the gdb log for the core file >>>>>>>>>> >>>>>>>>>> warning: Could not load shared library symbols for >>>>>>>>>> linux-vdso64.so.1. >>>>>>>>>> Do you need "set solib-search-path" or "set sysroot"? >>>>>>>>>> [Thread debugging using libthread_db enabled] >>>>>>>>>> Using host libthread_db library "/lib64/libthread_db.so.1". >>>>>>>>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140 >>>>>>>>>> --volfile-id gv0.128.224.95.140.tmp-bric'. >>>>>>>>>> Program terminated with signal SIGSEGV, Segmentation fault. >>>>>>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>>>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>>>>>> 3327 { >>>>>>>>>> [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))] >>>>>>>>>> (gdb) >>>>>>>>>> (gdb) >>>>>>>>>> (gdb) bt >>>>>>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020, >>>>>>>>>> bytes=bytes at entry=36) at malloc.c:3327 >>>>>>>>>> #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at >>>>>>>>>> malloc.c:2921 >>>>>>>>>> #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20, >>>>>>>>>> len=) at xdr_sizeof.c:89 >>>>>>>>>> #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from >>>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>>> #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from >>>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>>> #5 0x00003fff95b6fc28 in __GI_xdr_reference >>>>>>>>>> (xdrs=0x3fff90391d20, pp=0x3fff7c132020, size=, >>>>>>>>>> proc=) at xdr_ref.c:84 >>>>>>>>>> #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>>> objpp=0x3fff7c132020, obj_size=, >>>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>>> at xdr_ref.c:135 >>>>>>>>>> #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>>> #8 0x00003fff95b6fc28 in __GI_xdr_reference >>>>>>>>>> (xdrs=0x3fff90391d20, pp=0x3fff7c131ea0, size=, >>>>>>>>>> proc=) at xdr_ref.c:84 >>>>>>>>>> #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>>> objpp=0x3fff7c131ea0, obj_size=, >>>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>>> at xdr_ref.c:135 >>>>>>>>>> #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>>> #11 0x00003fff95b6fc28 in __GI_xdr_reference >>>>>>>>>> (xdrs=0x3fff90391d20, pp=0x3fff7c131d20, size=, >>>>>>>>>> proc=) at xdr_ref.c:84 >>>>>>>>>> #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>>> objpp=0x3fff7c131d20, obj_size=, >>>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>>> at xdr_ref.c:135 >>>>>>>>>> #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>>> #14 0x00003fff95b6fc28 in __GI_xdr_reference >>>>>>>>>> (xdrs=0x3fff90391d20, pp=0x3fff7c131ba0, size=, >>>>>>>>>> proc=) at xdr_ref.c:84 >>>>>>>>>> #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>>> objpp=0x3fff7c131ba0, obj_size=, >>>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>>> at xdr_ref.c:135 >>>>>>>>>> #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>>> #17 0x00003fff95b6fc28 in __GI_xdr_reference >>>>>>>>>> (xdrs=0x3fff90391d20, pp=0x3fff7c131a20, size=, >>>>>>>>>> proc=) at xdr_ref.c:84 >>>>>>>>>> #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>>> objpp=0x3fff7c131a20, obj_size=, >>>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>>> at xdr_ref.c:135 >>>>>>>>>> #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>>> #20 0x00003fff95b6fc28 in __GI_xdr_reference >>>>>>>>>> (xdrs=0x3fff90391d20, pp=0x3fff7c1318a0, size=, >>>>>>>>>> proc=) at xdr_ref.c:84 >>>>>>>>>> #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>>> objpp=0x3fff7c1318a0, obj_size=, >>>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>>> at xdr_ref.c:135 >>>>>>>>>> #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>>> #23 0x00003fff95b6fc28 in __GI_xdr_reference >>>>>>>>>> (xdrs=0x3fff90391d20, pp=0x3fff7c131720, size=, >>>>>>>>>> proc=) at xdr_ref.c:84 >>>>>>>>>> #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>>> objpp=0x3fff7c131720, obj_size=, >>>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>>> at xdr_ref.c:135 >>>>>>>>>> #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>>> #26 0x00003fff95b6fc28 in __GI_xdr_reference >>>>>>>>>> (xdrs=0x3fff90391d20, pp=0x3fff7c1315a0, size=, >>>>>>>>>> proc=) at xdr_ref.c:84 >>>>>>>>>> #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>>> objpp=0x3fff7c1315a0, obj_size=, >>>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>>> at xdr_ref.c:135 >>>>>>>>>> #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>>> #29 0x00003fff95b6fc28 in __GI_xdr_reference >>>>>>>>>> (xdrs=0x3fff90391d20, pp=0x3fff7c131420, size=, >>>>>>>>>> proc=) at xdr_ref.c:84 >>>>>>>>>> #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>>> objpp=0x3fff7c131420, obj_size=, >>>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>>> at xdr_ref.c:135 >>>>>>>>>> #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from >>>>>>>>>> /usr/lib64/libgfxdr.so.0 >>>>>>>>>> #32 0x00003fff95b6fc28 in __GI_xdr_reference >>>>>>>>>> (xdrs=0x3fff90391d20, pp=0x3fff7c1312a0, size=, >>>>>>>>>> proc=) at xdr_ref.c:84 >>>>>>>>>> #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20, >>>>>>>>>> objpp=0x3fff7c1312a0, obj_size=, >>>>>>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) >>>>>>>>>> at xdr_ref.c:135 >>>>>>>>>> >>>>>>>>>> Frames are getting repeated, could any one please me. >>>>>>>>>> -- >>>>>>>>>> Regards >>>>>>>>>> Abhishek Paliwal >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> Abhishek Paliwal >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Amar Tumballi (amarts) >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards >>>>>>> Abhishek Paliwal >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Regards >>>>>> Abhishek Paliwal >>>>>> >>>>> >>>>> >>>>> -- >>>>> Amar Tumballi (amarts) >>>>> >>>> >>>> >>>> -- >>>> >>>> >>>> >>>> >>>> Regards >>>> Abhishek Paliwal >>>> >>> >>> >>> -- >>> >>> >>> >>> >>> Regards >>> Abhishek Paliwal >>> >> >> >> -- >> >> >> >> >> Regards >> Abhishek Paliwal >> > > > -- > Amar Tumballi (amarts) > -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Thu Mar 14 03:57:57 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Thu, 14 Mar 2019 09:27:57 +0530 Subject: [Gluster-devel] [Gluster-users] "rpc_clnt_ping_timer_expired" errors In-Reply-To: References: <96B07283-D8AB-4F06-909D-E00424625528@cmcc.it> <42758A0E-8BE9-497D-BDE3-55D7DC633BC7@cmcc.it> <6A8CF4A4-98EA-48C3-A059-D60D1B2721C7@cmcc.it> <2CF49168-9C1B-4931-8C34-8157262A137A@cmcc.it> <7A151CC9-A0AE-4A45-B450-A4063D216D9E@cmcc.it> <32D53ECE-3F49-4415-A6EE-241B351AC2BA@cmcc.it> <4685A75B-5978-4338-9C9F-4A02FB40B9BC@cmcc.it> <4D2E6B04-C2E8-4FD5-B72D-E7C05931C6F9@cmcc.it> <4E332A56-B318-4BC1-9F44-84AB4392A5DE@cmcc.it> <832FD362-3B14-40D8-8530-604419300476@cmcc.it> <8D926643-1093-48ED-823F-D8F117F702CF@cmcc.it> <9D0BE438-DF11-4D0A-AB85-B44357032F29@cmcc.it> <5F0AC378-8170-4342-8473-9C17159CAC1D@cmcc.it> <7A50E86D-9E27-4EA7-883B-45E9F973991A@cmcc.it> <58B5DB7F-DCB4-4FBF-B1F7-681315B1613A@cmcc.it> <6327B44F-4E7E-46BB-A74C-70F4457DD1EB@cmcc.it> <0167DF4A-8329-4A1A-B439-857DFA6F78BB@cmcc.it> <763F334E-35B8-4729-B8E1-D30866754EEE@cmcc.it> <91DFC9AC-4805-41EB-AC6F-5722E018DD6E@cmcc.it> <8A9752B8-B231-4570-8FF4-8D3D781E7D42@cmcc.it> <47A24804-F975-4EE6-9FA5-67FCDA18D707@cmcc.it> <637FF5D2-D1F4-4686-9D48-646A96F67B96@cmcc.it> <4A87495F-3755-48F7-8507-085869069C64@cmcc.it> <3854BBF6-5B98-4AB3-A67E-E7DE59E69A63@cmcc.it> <313DA021-9173-4899-96B0-831B10B00B61@cmcc.it> <17996AFD-DFC8-40F3-9D09-DB6DDAD5B7D6@cmcc.it> <7074B5D8-955A-4802-A9F3-606C99734417@cmcc.it> <83B84BF9-8334-4230-B6F8-0BC4BFBEFF15@cmcc.it> <133B9AE4-9792-4F72-AD91-D36E7B9EC711@cmcc.it> <6611C4B0-57FD-4390-88B5-BD373555D4C5@cmcc.it> Message-ID: On Wed, Mar 13, 2019 at 3:55 PM Mauro Tridici wrote: > Hi Raghavendra, > > Yes, server.event-thread has been changed from 4 to 8. > Was client.event-thread value too changed to 8? If not, I would like to know the results of including this tuning too. Also, if possible, can you get the output of following command from problematic clients and bricks (during the duration when load tends to be high and ping-timer-expiry is seen)? # top -bHd 3 This will help us to know CPU utilization of event-threads. And I forgot to ask, what version of Glusterfs are you using? During last days, I noticed that the error events are still here although > they have been considerably reduced. > > So, I used grep command against the log files in order to provide you a > global vision about the warning, error and critical events appeared today > at 06:xx (may be useful I hope). > I collected the info from s06 gluster server, but the behaviour is the the > almost the same on the other gluster servers. > > *ERRORS: * > *CWD: /var/log/glusterfs * > *COMMAND: grep " E " *.log |grep "2019-03-13 06:"* > > (I can see a lot of this kind of message in the same period but I'm > notifying you only one record for each type of error) > > glusterd.log:[2019-03-13 06:12:35.982863] E [MSGID: 101042] > [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of > /var/run/gluster/tier2_quota_list/ > > glustershd.log:[2019-03-13 06:14:28.666562] E > [rpc-clnt.c:350:saved_frames_unwind] (--> > /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a71ddcebb] (--> > /lib64/libgfr > pc.so.0(saved_frames_unwind+0x1de)[0x7f4a71ba1d9e] (--> > /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f4a71ba1ebe] (--> > /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup > +0x90)[0x7f4a71ba3640] (--> > /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7f4a71ba4130] ))))) > 0-tier2-client-55: forced unwinding frame type(GlusterFS 3.3) > op(INODELK(29)) > called at 2019-03-13 06:14:14.858441 (xid=0x17fddb50) > > glustershd.log:[2019-03-13 06:17:48.883825] E > [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to > 192.168.0.55:49158 failed (Connection timed out); disco > nnecting socket > glustershd.log:[2019-03-13 06:19:58.931798] E > [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to > 192.168.0.55:49158 failed (Connection timed out); disco > nnecting socket > glustershd.log:[2019-03-13 06:22:08.979829] E > [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to > 192.168.0.55:49158 failed (Connection timed out); disco > nnecting socket > glustershd.log:[2019-03-13 06:22:36.226847] E [MSGID: 114031] > [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote > operation failed [Transport endpoint > is not connected] > glustershd.log:[2019-03-13 06:22:36.306669] E [MSGID: 114031] > [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote > operation failed [Transport endpoint > is not connected] > glustershd.log:[2019-03-13 06:22:36.385257] E [MSGID: 114031] > [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote > operation failed [Transport endpoint > is not connected] > > *WARNINGS:* > *CWD: /var/log/glusterfs * > *COMMAND: grep " W " *.log |grep "2019-03-13 06:"* > > (I can see a lot of this kind of message in the same period but I'm > notifying you only one record for each type of warnings) > > glustershd.log:[2019-03-13 06:14:28.666772] W [MSGID: 114031] > [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-tier2-client-55: remote > operation failed. Path: 0f-f34d-4c25-bbe8-74bde0248d7e> (b6b35d0f-f34d-4c25-bbe8-74bde0248d7e). > Key: (null) [Transport endpoint is not connected] > > glustershd.log:[2019-03-13 06:14:31.421576] W [MSGID: 122035] > [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation > with some subvolumes unavailable (2) > > glustershd.log:[2019-03-13 06:15:31.547417] W [MSGID: 122032] > [ec-heald.c:266:ec_shd_index_sweep] 0-tier2-disperse-9: unable to get > index-dir on tier2-client-55 [Operation > now in progress] > > quota-mount-tier2.log:[2019-03-13 06:12:36.116277] W [MSGID: 101002] > [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is > deprecated, preferred is 'trans > port.address-family', continuing with correction > quota-mount-tier2.log:[2019-03-13 06:12:36.198430] W [MSGID: 101174] > [graph.c:363:_log_if_unknown_option] 0-tier2-readdir-ahead: option > 'parallel-readdir' is not recognized > quota-mount-tier2.log:[2019-03-13 06:12:37.945007] W > [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) > [0x7f340892be25] -->/usr/sbin/glusterfs(gluste > rfs_sigwaiter+0xe5) [0x55ef010164b5] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55ef0101632b] ) 0-: > received signum (15), shutting down > > *CRITICALS:* > *CWD: /var/log/glusterfs * > *COMMAND: grep " C " *.log |grep "2019-03-13 06:"* > > no critical errors at 06:xx > only one critical error during the day > > *[root at s06 glusterfs]# grep " C " *.log |grep "2019-03-13"* > glustershd.log:[2019-03-13 02:21:29.126279] C > [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server > 192.168.0.55:49158 has not responded in the last 42 seconds, > disconnecting. > > > Thank you very much for your help. > Regards, > Mauro > > On 12 Mar 2019, at 05:17, Raghavendra Gowdappa > wrote: > > Was the suggestion to increase server.event-thread values tried? If yes, > what were the results? > > On Mon, Mar 11, 2019 at 2:40 PM Mauro Tridici > wrote: > >> Dear All, >> >> do you have any suggestions about the right way to "debug" this issue? >> In attachment, the updated logs of ?s06" gluster server. >> >> I noticed a lot of intermittent warning and error messages. >> >> Thank you in advance, >> Mauro >> >> >> >> On 4 Mar 2019, at 18:45, Raghavendra Gowdappa >> wrote: >> >> >> +Gluster Devel , +Gluster-users >> >> >> I would like to point out another issue. Even if what I suggested >> prevents disconnects, part of the solution would be only symptomatic >> treatment and doesn't address the root cause of the problem. In most of the >> ping-timer-expiry issues, the root cause is the increased load on bricks >> and the inability of bricks to be responsive under high load. So, the >> actual solution would be doing any or both of the following: >> * identify the source of increased load and if possible throttle it. >> Internal heal processes like self-heal, rebalance, quota heal are known to >> pump traffic into bricks without much throttling (io-threads _might_ do >> some throttling, but my understanding is its not sufficient). >> * identify the reason for bricks to become unresponsive during load. This >> may be fixable issues like not enough event-threads to read from network or >> difficult to fix issues like fsync on backend fs freezing the process or >> semi fixable issues (in code) like lock contention. >> >> So any genuine effort to fix ping-timer-issues (to be honest most of the >> times they are not issues related to rpc/network) would involve performance >> characterization of various subsystems on bricks and clients. Various >> subsystems can include (but not necessarily limited to), underlying >> OS/filesystem, glusterfs processes, CPU consumption etc >> >> regards, >> Raghavendra >> >> On Mon, Mar 4, 2019 at 9:31 PM Mauro Tridici >> wrote: >> >>> Thank you, let?s try! >>> I will inform you about the effects of the change. >>> >>> Regards, >>> Mauro >>> >>> On 4 Mar 2019, at 16:55, Raghavendra Gowdappa >>> wrote: >>> >>> >>> >>> On Mon, Mar 4, 2019 at 8:54 PM Mauro Tridici >>> wrote: >>> >>>> Hi Raghavendra, >>>> >>>> thank you for your reply. >>>> Yes, you are right. It is a problem that seems to happen randomly. >>>> At this moment, server.event-threads value is 4. I will try to increase >>>> this value to 8. Do you think that it could be a valid value ? >>>> >>> >>> Yes. We can try with that. You should see at least frequency of >>> ping-timer related disconnects reduce with this value (even if it doesn't >>> eliminate the problem completely). >>> >>> >>>> Regards, >>>> Mauro >>>> >>>> >>>> On 4 Mar 2019, at 15:36, Raghavendra Gowdappa >>>> wrote: >>>> >>>> >>>> >>>> On Mon, Mar 4, 2019 at 8:01 PM Nithya Balachandran >>>> wrote: >>>> >>>>> Hi Mauro, >>>>> >>>>> It looks like some problem on s06. Are all your other nodes ok? Can >>>>> you send us the gluster logs from this node? >>>>> >>>>> @Raghavendra G , do you have any idea as to >>>>> how this can be debugged? Maybe running top ? Or debug brick logs? >>>>> >>>> >>>> If we can reproduce the problem, collecting tcpdump on both ends of >>>> connection will help. But, one common problem is these bugs are >>>> inconsistently reproducible and hence we may not be able to capture tcpdump >>>> at correct intervals. Other than that, we can try to collect some evidence >>>> that poller threads were busy (waiting on locks). But, not sure what debug >>>> data provides that information. >>>> >>>> From what I know, its difficult to collect evidence for this issue and >>>> we could only reason about it. >>>> >>>> We can try a workaround though - try increasing server.event-threads >>>> and see whether ping-timer expiry issues go away with an optimal value. If >>>> that's the case, it kind of provides proof for our hypothesis. >>>> >>>> >>>>> >>>>> Regards, >>>>> Nithya >>>>> >>>>> On Mon, 4 Mar 2019 at 15:25, Mauro Tridici >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> some minutes ago I received this message from NAGIOS server >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ****** Nagios *****Notification Type: PROBLEMService: Brick - >>>>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: CRITICALDate/Time: Mon >>>>>> Mar 4 10:25:33 CET 2019Additional Info:CHECK_NRPE STATE CRITICAL: Socket >>>>>> timeout after 10 seconds.* >>>>>> >>>>>> I checked the network, RAM and CPUs usage on s06 node and everything >>>>>> seems to be ok. >>>>>> No bricks are in error state. In /var/log/messages, I detected again >>>>>> a crash of ?check_vol_utili? that I think it is a module used by NRPE >>>>>> executable (that is the NAGIOS client). >>>>>> >>>>>> Mar 4 10:15:29 s06 kernel: traps: check_vol_utili[161224] general >>>>>> protection ip:7facffa0a66d sp:7ffe9f4e6fc0 error:0 in >>>>>> libglusterfs.so.0.0.1[7facff9b7000+f7000] >>>>>> Mar 4 10:15:29 s06 abrt-hook-ccpp: Process 161224 (python2.7) of >>>>>> user 0 killed by SIGSEGV - dumping core >>>>>> Mar 4 10:15:29 s06 abrt-server: Generating core_backtrace >>>>>> Mar 4 10:15:29 s06 abrt-server: Error: Unable to open './coredump': >>>>>> No such file or directory >>>>>> Mar 4 10:16:01 s06 systemd: Created slice User Slice of root. >>>>>> Mar 4 10:16:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 4 10:16:01 s06 systemd: Started Session 201010 of user root. >>>>>> Mar 4 10:16:01 s06 systemd: Starting Session 201010 of user root. >>>>>> Mar 4 10:16:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 4 10:16:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 4 10:16:24 s06 abrt-server: Duplicate: UUID >>>>>> Mar 4 10:16:24 s06 abrt-server: DUP_OF_DIR: >>>>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>>> Mar 4 10:16:24 s06 abrt-server: Deleting problem directory >>>>>> ccpp-2019-03-04-10:15:29-161224 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>>> Mar 4 10:16:24 s06 abrt-server: Generating core_backtrace >>>>>> Mar 4 10:16:24 s06 abrt-server: Error: Unable to open './coredump': >>>>>> No such file or directory >>>>>> Mar 4 10:16:24 s06 abrt-server: Cannot notify >>>>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>>>> 'report_uReport' exited with 1 >>>>>> Mar 4 10:16:24 s06 abrt-hook-ccpp: Process 161391 (python2.7) of >>>>>> user 0 killed by SIGABRT - dumping core >>>>>> Mar 4 10:16:25 s06 abrt-server: Generating core_backtrace >>>>>> Mar 4 10:16:25 s06 abrt-server: Error: Unable to open './coredump': >>>>>> No such file or directory >>>>>> Mar 4 10:17:01 s06 systemd: Created slice User Slice of root. >>>>>> >>>>>> Also, I noticed the following errors that I think are very critical: >>>>>> >>>>>> Mar 4 10:21:12 s06 glustershd[20355]: [2019-03-04 09:21:12.954798] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server >>>>>> 192.168.0.55:49158 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:22:01 s06 systemd: Created slice User Slice of root. >>>>>> Mar 4 10:22:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 4 10:22:01 s06 systemd: Started Session 201017 of user root. >>>>>> Mar 4 10:22:01 s06 systemd: Starting Session 201017 of user root. >>>>>> Mar 4 10:22:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 4 10:22:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 4 10:22:03 s06 glustershd[20355]: [2019-03-04 09:22:03.964120] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server >>>>>> 192.168.0.54:49165 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:23:01 s06 systemd: Created slice User Slice of root. >>>>>> Mar 4 10:23:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 4 10:23:01 s06 systemd: Started Session 201018 of user root. >>>>>> Mar 4 10:23:01 s06 systemd: Starting Session 201018 of user root. >>>>>> Mar 4 10:23:02 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 4 10:23:02 s06 systemd: Stopping User Slice of root. >>>>>> Mar 4 10:24:01 s06 systemd: Created slice User Slice of root. >>>>>> Mar 4 10:24:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 4 10:24:01 s06 systemd: Started Session 201019 of user root. >>>>>> Mar 4 10:24:01 s06 systemd: Starting Session 201019 of user root. >>>>>> Mar 4 10:24:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 4 10:24:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 4 10:24:03 s06 glustershd[20355]: [2019-03-04 09:24:03.982502] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-16: server >>>>>> 192.168.0.52:49158 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746109] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-3: server >>>>>> 192.168.0.51:49153 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746215] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server >>>>>> 192.168.0.52:49156 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746260] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-21: server >>>>>> 192.168.0.51:49159 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746296] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server >>>>>> 192.168.0.52:49161 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746413] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server >>>>>> 192.168.0.54:49165 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:24:07 s06 glustershd[20355]: [2019-03-04 09:24:07.982952] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-45: server >>>>>> 192.168.0.54:49155 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:24:18 s06 glustershd[20355]: [2019-03-04 09:24:18.990929] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server >>>>>> 192.168.0.52:49161 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:24:31 s06 glustershd[20355]: [2019-03-04 09:24:31.995781] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-20: server >>>>>> 192.168.0.53:49159 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:25:01 s06 systemd: Created slice User Slice of root. >>>>>> Mar 4 10:25:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 4 10:25:01 s06 systemd: Started Session 201020 of user root. >>>>>> Mar 4 10:25:01 s06 systemd: Starting Session 201020 of user root. >>>>>> Mar 4 10:25:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 4 10:25:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 4 10:25:57 s06 systemd: Created slice User Slice of root. >>>>>> Mar 4 10:25:57 s06 systemd: Starting User Slice of root. >>>>>> Mar 4 10:25:57 s06 systemd-logind: New session 201021 of user root. >>>>>> Mar 4 10:25:57 s06 systemd: Started Session 201021 of user root. >>>>>> Mar 4 10:25:57 s06 systemd: Starting Session 201021 of user root. >>>>>> Mar 4 10:26:01 s06 systemd: Started Session 201022 of user root. >>>>>> Mar 4 10:26:01 s06 systemd: Starting Session 201022 of user root. >>>>>> Mar 4 10:26:21 s06 nrpe[162388]: Error: Could not complete SSL >>>>>> handshake with 192.168.1.56: 5 >>>>>> Mar 4 10:27:01 s06 systemd: Started Session 201023 of user root. >>>>>> Mar 4 10:27:01 s06 systemd: Starting Session 201023 of user root. >>>>>> Mar 4 10:28:01 s06 systemd: Started Session 201024 of user root. >>>>>> Mar 4 10:28:01 s06 systemd: Starting Session 201024 of user root. >>>>>> Mar 4 10:29:01 s06 systemd: Started Session 201025 of user root. >>>>>> Mar 4 10:29:01 s06 systemd: Starting Session 201025 of user root. >>>>>> >>>>>> But, unfortunately, I don?t understand why it is happening. >>>>>> Now, NAGIOS server shows that s06 status is ok: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ****** Nagios *****Notification Type: RECOVERYService: Brick - >>>>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: OKDate/Time: Mon Mar 4 >>>>>> 10:35:23 CET 2019Additional Info:OK: Brick /gluster/mnt2/brick is up* >>>>>> >>>>>> Nothing is changed from RAM, CPUs, and NETWORK point of view. >>>>>> /var/log/message file has been updated: >>>>>> >>>>>> Mar 4 10:32:01 s06 systemd: Starting Session 201029 of user root. >>>>>> Mar 4 10:32:30 s06 glustershd[20355]: [2019-03-04 09:32:30.069082] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server >>>>>> 192.168.0.52:49156 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:32:55 s06 glustershd[20355]: [2019-03-04 09:32:55.074689] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-66: server >>>>>> 192.168.0.54:49167 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 4 10:33:01 s06 systemd: Started Session 201030 of user root. >>>>>> Mar 4 10:33:01 s06 systemd: Starting Session 201030 of user root. >>>>>> Mar 4 10:34:01 s06 systemd: Started Session 201031 of user root. >>>>>> Mar 4 10:34:01 s06 systemd: Starting Session 201031 of user root. >>>>>> Mar 4 10:35:01 s06 nrpe[162562]: Could not read request from client >>>>>> 192.168.1.56, bailing out... >>>>>> Mar 4 10:35:01 s06 nrpe[162562]: INFO: SSL Socket Shutdown. >>>>>> Mar 4 10:35:01 s06 systemd: Started Session 201032 of user root. >>>>>> Mar 4 10:35:01 s06 systemd: Starting Session 201032 of user root. >>>>>> >>>>>> Could you please help me to understand what it?s happening ? >>>>>> Thank you in advance. >>>>>> >>>>>> Rergards, >>>>>> Mauro >>>>>> >>>>>> >>>>>> On 1 Mar 2019, at 12:17, Mauro Tridici wrote: >>>>>> >>>>>> >>>>>> Thank you, Milind. >>>>>> I executed the instructions you suggested: >>>>>> >>>>>> - grep ?blocked for? /var/log/messages on s06 returns no output (no >>>>>> ?blocked? word is detected in messages file); >>>>>> - in /var/log/messages file I can see this kind of error repeated for >>>>>> a lot of times: >>>>>> >>>>>> Mar 1 08:43:01 s06 systemd: Starting Session 196071 of user root. >>>>>> Mar 1 08:43:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 1 08:43:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 1 08:43:02 s06 kernel: traps: check_vol_utili[57091] general >>>>>> protection ip:7f88e76ee66d sp:7ffe5a5bcc30 error:0 in >>>>>> libglusterfs.so.0.0.1[7f88e769b000+f7000] >>>>>> Mar 1 08:43:02 s06 abrt-hook-ccpp: Process 57091 (python2.7) of user >>>>>> 0 killed by SIGSEGV - dumping core >>>>>> Mar 1 08:43:02 s06 abrt-server: Generating core_backtrace >>>>>> Mar 1 08:43:02 s06 abrt-server: Error: Unable to open './coredump': >>>>>> No such file or directory >>>>>> Mar 1 08:43:58 s06 abrt-server: Duplicate: UUID >>>>>> Mar 1 08:43:58 s06 abrt-server: DUP_OF_DIR: >>>>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>>> Mar 1 08:43:58 s06 abrt-server: Deleting problem directory >>>>>> ccpp-2019-03-01-08:43:02-57091 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Activating service >>>>>> name='org.freedesktop.problems' (using servicehelper) >>>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Successfully activated >>>>>> service 'org.freedesktop.problems' >>>>>> Mar 1 08:43:58 s06 abrt-server: Generating core_backtrace >>>>>> Mar 1 08:43:58 s06 abrt-server: Error: Unable to open './coredump': >>>>>> No such file or directory >>>>>> Mar 1 08:43:58 s06 abrt-server: Cannot notify >>>>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>>>> 'report_uReport' exited with 1 >>>>>> Mar 1 08:44:01 s06 systemd: Created slice User Slice of root. >>>>>> Mar 1 08:44:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 1 08:44:01 s06 systemd: Started Session 196072 of user root. >>>>>> Mar 1 08:44:01 s06 systemd: Starting Session 196072 of user root. >>>>>> Mar 1 08:44:01 s06 systemd: Removed slice User Slice of root. >>>>>> >>>>>> - in /var/log/messages file I can see also 4 errors related to other >>>>>> cluster servers: >>>>>> >>>>>> Mar 1 11:05:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 1 11:05:01 s06 systemd: Started Session 196230 of user root. >>>>>> Mar 1 11:05:01 s06 systemd: Starting Session 196230 of user root. >>>>>> Mar 1 11:05:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 1 11:05:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 1 11:05:59 s06 glustershd[70117]: [2019-03-01 10:05:59.347094] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-33: server >>>>>> 192.168.0.51:49163 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 1 11:06:01 s06 systemd: Created slice User Slice of root. >>>>>> Mar 1 11:06:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 1 11:06:01 s06 systemd: Started Session 196231 of user root. >>>>>> Mar 1 11:06:01 s06 systemd: Starting Session 196231 of user root. >>>>>> Mar 1 11:06:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 1 11:06:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 1 11:06:12 s06 glustershd[70117]: [2019-03-01 10:06:12.351319] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-1: server >>>>>> 192.168.0.52:49153 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 1 11:06:38 s06 glustershd[70117]: [2019-03-01 10:06:38.356920] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-7: server >>>>>> 192.168.0.52:49155 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 1 11:07:01 s06 systemd: Created slice User Slice of root. >>>>>> Mar 1 11:07:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 1 11:07:01 s06 systemd: Started Session 196232 of user root. >>>>>> Mar 1 11:07:01 s06 systemd: Starting Session 196232 of user root. >>>>>> Mar 1 11:07:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 1 11:07:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 1 11:07:36 s06 glustershd[70117]: [2019-03-01 10:07:36.366259] C >>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-0: server >>>>>> 192.168.0.51:49152 has not responded in the last 42 seconds, >>>>>> disconnecting. >>>>>> Mar 1 11:08:01 s06 systemd: Created slice User Slice of root. >>>>>> >>>>>> No ?blocked? word is in /var/log/messages files on other cluster >>>>>> servers. >>>>>> In attachment, the /var/log/messages file from s06 server. >>>>>> >>>>>> Thank you in advance, >>>>>> Mauro >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 1 Mar 2019, at 11:47, Milind Changire wrote: >>>>>> >>>>>> The traces of very high disk activity on the servers are often found >>>>>> in /var/log/messages >>>>>> You might want to grep for "blocked for" in /var/log/messages on s06 >>>>>> and correlate the timestamps to confirm the unresponsiveness as reported in >>>>>> gluster client logs. >>>>>> In cases of high disk activity, although the operating system >>>>>> continues to respond to ICMP pings, the processes writing to disks often >>>>>> get blocked to a large flush to the disk which could span beyond 42 seconds >>>>>> and hence result in ping-timer-expiry logs. >>>>>> >>>>>> As a side note: >>>>>> If you indeed find gluster processes being blocked in >>>>>> /var/log/messages, you might want to tweak sysctl tunables called >>>>>> vm.dirty_background_ratio or vm.dirty_background_bytes to a smaller value >>>>>> than the existing. Please read up more on those tunables before touching >>>>>> the settings. >>>>>> >>>>>> >>>>>> On Fri, Mar 1, 2019 at 4:06 PM Mauro Tridici >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> in attachment the client log captured after changing >>>>>>> network.ping-timeout option. >>>>>>> I noticed this error involving server 192.168.0.56 (s06) >>>>>>> >>>>>>> [2019-03-01 09:23:36.077287] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >>>>>>> 0-tier2-client-71: changing ping timeout to 42 (from 0) >>>>>>> [2019-03-01 09:23:36.078213] I >>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>> volfile,continuing >>>>>>> [2019-03-01 09:23:36.078432] I >>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>> volfile,continuing >>>>>>> [2019-03-01 09:23:36.092357] I >>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>> volfile,continuing >>>>>>> [2019-03-01 09:23:36.094146] I >>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>> volfile,continuing >>>>>>> [2019-03-01 10:06:24.708082] C >>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-50: server >>>>>>> 192.168.0.56:49156 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> >>>>>>> I don?t know why it happens, s06 server seems to be reachable. >>>>>>> >>>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>>> Trying 192.168.0.56... >>>>>>> Connected to 192.168.0.56. >>>>>>> Escape character is '^]'. >>>>>>> ^CConnection closed by foreign host. >>>>>>> [athena_login2][/users/home/sysm02/]> ping 192.168.0.56 >>>>>>> PING 192.168.0.56 (192.168.0.56) 56(84) bytes of data. >>>>>>> 64 bytes from 192.168.0.56: icmp_seq=1 ttl=64 time=0.116 ms >>>>>>> 64 bytes from 192.168.0.56: icmp_seq=2 ttl=64 time=0.101 ms >>>>>>> >>>>>>> --- 192.168.0.56 ping statistics --- >>>>>>> 2 packets transmitted, 2 received, 0% packet loss, time 1528ms >>>>>>> rtt min/avg/max/mdev = 0.101/0.108/0.116/0.012 ms >>>>>>> >>>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>>> Trying 192.168.0.56... >>>>>>> Connected to 192.168.0.56. >>>>>>> Escape character is '^]'. >>>>>>> >>>>>>> Thank you for your help, >>>>>>> Mauro >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 1 Mar 2019, at 10:29, Mauro Tridici >>>>>>> wrote: >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> thank you for the explanation. >>>>>>> I just changed network.ping-timeout option to default value >>>>>>> (network.ping-timeout=42). >>>>>>> >>>>>>> I will check the logs to see if the errors will appear again. >>>>>>> >>>>>>> Regards, >>>>>>> Mauro >>>>>>> >>>>>>> On 1 Mar 2019, at 04:43, Milind Changire >>>>>>> wrote: >>>>>>> >>>>>>> network.ping-timeout should not be set to zero for non-glusterd >>>>>>> clients. >>>>>>> glusterd is a special case for which ping-timeout is set to zero via >>>>>>> /etc/glusterfs/glusterd.vol >>>>>>> >>>>>>> Setting network.ping-timeout to zero disables arming of the ping >>>>>>> timer for connections. This disables testing the connection for >>>>>>> responsiveness and hence avoids proactive fail-over. >>>>>>> >>>>>>> Please reset network.ping-timeout to a non-zero positive value, eg. >>>>>>> 42 >>>>>>> >>>>>>> >>>>>>> On Thu, Feb 28, 2019 at 5:07 PM Nithya Balachandran < >>>>>>> nbalacha at redhat.com> wrote: >>>>>>> >>>>>>>> Adding Raghavendra and Milind to comment on this. >>>>>>>> >>>>>>>> What is the effect of setting network.ping-timeout to 0 and should >>>>>>>> it be set back to 42? >>>>>>>> Regards, >>>>>>>> Nithya >>>>>>>> >>>>>>>> On Thu, 28 Feb 2019 at 16:01, Mauro Tridici >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Nithya, >>>>>>>>> >>>>>>>>> sorry for the late. >>>>>>>>> network.ping-timeout has been set to 0 in order to try to solve >>>>>>>>> some timeout problems, but it didn?t help. >>>>>>>>> I can set it to the default value. >>>>>>>>> >>>>>>>>> Can I proceed with the change? >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Mauro >>>>>>>>> >>>>>>>>> >>>>>>>>> On 28 Feb 2019, at 04:41, Nithya Balachandran >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Mauro, >>>>>>>>> >>>>>>>>> Is network.ping-timeout still set to 0. The default value is 42. >>>>>>>>> Is there a particular reason why this was changed? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Nithya >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, 27 Feb 2019 at 21:32, Mauro Tridici >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi Xavi, >>>>>>>>>> >>>>>>>>>> thank you for the detailed explanation and suggestions. >>>>>>>>>> Yes, transport.listen-backlog option is still set to 1024. >>>>>>>>>> >>>>>>>>>> I will check the network and connectivity status using ?ping? and >>>>>>>>>> ?telnet? as soon as the errors will come back again. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Mauro >>>>>>>>>> >>>>>>>>>> Il giorno 27 feb 2019, alle ore 16:42, Xavi Hernandez < >>>>>>>>>> jahernan at redhat.com> ha scritto: >>>>>>>>>> >>>>>>>>>> Hi Mauro, >>>>>>>>>> >>>>>>>>>> those errors say that the mount point is not connected to some of >>>>>>>>>> the bricks while executing operations. I see references to 3rd and 6th >>>>>>>>>> bricks of several disperse sets, which seem to map to server s06. For some >>>>>>>>>> reason, gluster is having troubles connecting from the client machine to >>>>>>>>>> that particular server. At the end of the log I see that after long time a >>>>>>>>>> reconnect is done to both of them. However little after, other bricks from >>>>>>>>>> the s05 get disconnected and a reconnect times out. >>>>>>>>>> >>>>>>>>>> That's really odd. It seems like if server/communication is cut >>>>>>>>>> to s06 for some time, then restored, and then the same happens to the next >>>>>>>>>> server. >>>>>>>>>> >>>>>>>>>> If the servers are really online and it's only a communication >>>>>>>>>> issue, it explains why server memory and network has increased: if the >>>>>>>>>> problem only exists between the client and servers, any write made by the >>>>>>>>>> client will automatically mark the file as damaged, since some of the >>>>>>>>>> servers have not been updated. Since self-heal runs from the server nodes, >>>>>>>>>> they will probably be correctly connected to all bricks, which allows them >>>>>>>>>> to heal the just damaged file, which increases memory and network usage. >>>>>>>>>> >>>>>>>>>> I guess you still have transport.listen-backlog set to 1024, >>>>>>>>>> right ? >>>>>>>>>> >>>>>>>>>> Just to try to identify if the problem really comes from network, >>>>>>>>>> can you check if you lose some pings from the client to all of the servers >>>>>>>>>> while you are seeing those errors in the log file ? >>>>>>>>>> >>>>>>>>>> You can also check if during those errors, you can telnet to the >>>>>>>>>> port of the brick from the client. >>>>>>>>>> >>>>>>>>>> Xavi >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Feb 26, 2019 at 10:17 AM Mauro Tridici < >>>>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Nithya, >>>>>>>>>>> >>>>>>>>>>> ?df -h? operation is not still slow, but no users are using the >>>>>>>>>>> volume, RAM and NETWORK usage is ok on the client node. >>>>>>>>>>> >>>>>>>>>>> I was worried about this kind of warnings/errors: >>>>>>>>>>> >>>>>>>>>>> [2019-02-25 10:59:00.664323] W [MSGID: 122035] >>>>>>>>>>> [ec-common.c:571:ec_child_select] 0-tier2-disperse-6: Executing operation >>>>>>>>>>> with some subvolumes unavailable (20) >>>>>>>>>>> >>>>>>>>>>> [2019-02-26 03:11:35.212603] E >>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>>> called at 2019-02-26 03:10:56.549903 (xid=0x106f1c5) >>>>>>>>>>> >>>>>>>>>>> [2019-02-26 03:13:03.313831] E >>>>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-50: connection to >>>>>>>>>>> 192.168.0.56:49156 failed (Timeout della connessione); >>>>>>>>>>> disconnecting socket >>>>>>>>>>> >>>>>>>>>>> It seems that some subvolumes are not available and 192.168.0.56 >>>>>>>>>>> server (s06) is not reachable. >>>>>>>>>>> But gluster servers are up&running and bricks are ok. >>>>>>>>>>> >>>>>>>>>>> In attachment the updated tier2.log file. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thank you. >>>>>>>>>>> Regards, >>>>>>>>>>> Mauro >>>>>>>>>>> >>>>>>>>>>> Il giorno 26 feb 2019, alle ore 04:03, Nithya Balachandran < >>>>>>>>>>> nbalacha at redhat.com> ha scritto: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I see a lot of EC messages in the log but they don't seem very >>>>>>>>>>> serious. Xavi, can you take a look? >>>>>>>>>>> >>>>>>>>>>> The only errors I see are: >>>>>>>>>>> [2019-02-25 10:58:45.519871] E >>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>>> called at 2019-02-25 10:57:47.429969 (xid=0xd26fe7) >>>>>>>>>>> [2019-02-25 10:58:51.461493] E >>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>>> 0-tier2-client-41: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>>> called at 2019-02-25 10:57:47.499174 (xid=0xf47d6a) >>>>>>>>>>> [2019-02-25 11:07:57.152874] E >>>>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-70: connection to >>>>>>>>>>> 192.168.0.55:49163 failed (Timeout della connessione); >>>>>>>>>>> disconnecting socket >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Is the df -h operation still slow? If yes, can you take a >>>>>>>>>>> tcpdump of the client while running df -h and send that across? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Nithya >>>>>>>>>>> >>>>>>>>>>> On Mon, 25 Feb 2019 at 17:27, Mauro Tridici < >>>>>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Sorry, some minutes after my last mail message, I noticed that >>>>>>>>>>>> ?df -h? command hanged for a while before returns the prompt. >>>>>>>>>>>> Yesterday, everything was ok in the gluster client log, but, >>>>>>>>>>>> today, I see a lot of errors (please, take a look to the attached file). >>>>>>>>>>>> >>>>>>>>>>>> On the client node, I detected an important RAM e NETWORK usage. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Do you think that the errors have been caused by the client >>>>>>>>>>>> resources usage? >>>>>>>>>>>> >>>>>>>>>>>> Thank you in advance, >>>>>>>>>>>> Mauro >>>>>>>>>>>> >>>>>>>>>>>> >>>>>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From atumball at redhat.com Thu Mar 14 05:44:51 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Thu, 14 Mar 2019 11:14:51 +0530 Subject: [Gluster-devel] Github#268 Compatibility with Alpine Linux In-Reply-To: References: Message-ID: I tried this recently, and the issue of rpcgen is real, and is not straight-forward is what I felt. Would like to pick this up after glusterfs-6.0 release. -Amar On Tue, Mar 12, 2019 at 8:17 AM Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > Saw some recent activity on > - is there a plan to > address this or, should the interested users be informed about other > plans? > > /s > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > > > -- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Thu Mar 14 12:31:44 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Thu, 14 Mar 2019 18:01:44 +0530 Subject: [Gluster-devel] [Gluster-users] "rpc_clnt_ping_timer_expired" errors In-Reply-To: References: <96B07283-D8AB-4F06-909D-E00424625528@cmcc.it> <42758A0E-8BE9-497D-BDE3-55D7DC633BC7@cmcc.it> <6A8CF4A4-98EA-48C3-A059-D60D1B2721C7@cmcc.it> <2CF49168-9C1B-4931-8C34-8157262A137A@cmcc.it> <7A151CC9-A0AE-4A45-B450-A4063D216D9E@cmcc.it> <32D53ECE-3F49-4415-A6EE-241B351AC2BA@cmcc.it> <4685A75B-5978-4338-9C9F-4A02FB40B9BC@cmcc.it> <4D2E6B04-C2E8-4FD5-B72D-E7C05931C6F9@cmcc.it> <4E332A56-B318-4BC1-9F44-84AB4392A5DE@cmcc.it> <832FD362-3B14-40D8-8530-604419300476@cmcc.it> <8D926643-1093-48ED-823F-D8F117F702CF@cmcc.it> <9D0BE438-DF11-4D0A-AB85-B44357032F29@cmcc.it> <5F0AC378-8170-4342-8473-9C17159CAC1D@cmcc.it> <7A50E86D-9E27-4EA7-883B-45E9F973991A@cmcc.it> <58B5DB7F-DCB4-4FBF-B1F7-681315B1613A@cmcc.it> <6327B44F-4E7E-46BB-A74C-70F4457DD1EB@cmcc.it> <0167DF4A-8329-4A1A-B439-857DFA6F78BB@cmcc.it> <763F334E-35B8-4729-B8E1-D30866754EEE@cmcc.it> <91DFC9AC-4805-41EB-AC6F-5722E018DD6E@cmcc.it> <8A9752B8-B231-4570-8FF4-8D3D781E7D42@cmcc.it> <47A24804-F975-4EE6-9FA5-67FCDA18D707@cmcc.it> <637FF5D2-D1F4-4686-9D48-646A96F67B96@cmcc.it> <4A87495F-3755-48F7-8507-085869069C64@cmcc.it> <3854BBF6-5B98-4AB3-A67E-E7DE59E69A63@cmcc.it> <313DA021-9173-4899-96B0-831B10B00B61@cmcc.it> <17996AFD-DFC8-40F3-9D09-DB6DDAD5B7D6@cmcc.it> <7074B5D8-955A-4802-A9F3-606C99734417@cmcc.it> <83B84BF9-8334-4230-B6F8-0BC4BFBEFF15@cmcc.it> <133B9AE4-9792-4F72-AD91-D36E7B9EC711@cmcc.it> <6611C4B0-57FD-4390-88B5-BD373555D4C5@cmcc.it> Message-ID: Thanks Mauro. On Thu, Mar 14, 2019 at 3:38 PM Mauro Tridici wrote: > Hi Raghavendra, > > I just changed the client option value to 8. > I will check the volume behaviour during the next hours. > > The GlusterFS version is 3.12.14. > > I will provide you the logs as soon as the activity load will be high. > Thank you, > Mauro > > On 14 Mar 2019, at 04:57, Raghavendra Gowdappa > wrote: > > > > On Wed, Mar 13, 2019 at 3:55 PM Mauro Tridici > wrote: > >> Hi Raghavendra, >> >> Yes, server.event-thread has been changed from 4 to 8. >> > > Was client.event-thread value too changed to 8? If not, I would like to > know the results of including this tuning too. Also, if possible, can you > get the output of following command from problematic clients and bricks > (during the duration when load tends to be high and ping-timer-expiry is > seen)? > > # top -bHd 3 > > This will help us to know CPU utilization of event-threads. > > And I forgot to ask, what version of Glusterfs are you using? > > During last days, I noticed that the error events are still here although >> they have been considerably reduced. >> >> So, I used grep command against the log files in order to provide you a >> global vision about the warning, error and critical events appeared today >> at 06:xx (may be useful I hope). >> I collected the info from s06 gluster server, but the behaviour is the >> the almost the same on the other gluster servers. >> >> *ERRORS: * >> *CWD: /var/log/glusterfs * >> *COMMAND: grep " E " *.log |grep "2019-03-13 06:"* >> >> (I can see a lot of this kind of message in the same period but I'm >> notifying you only one record for each type of error) >> >> glusterd.log:[2019-03-13 06:12:35.982863] E [MSGID: 101042] >> [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of >> /var/run/gluster/tier2_quota_list/ >> >> glustershd.log:[2019-03-13 06:14:28.666562] E >> [rpc-clnt.c:350:saved_frames_unwind] (--> >> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a71ddcebb] (--> >> /lib64/libgfr >> pc.so.0(saved_frames_unwind+0x1de)[0x7f4a71ba1d9e] (--> >> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f4a71ba1ebe] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup >> +0x90)[0x7f4a71ba3640] (--> >> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7f4a71ba4130] ))))) >> 0-tier2-client-55: forced unwinding frame type(GlusterFS 3.3) >> op(INODELK(29)) >> called at 2019-03-13 06:14:14.858441 (xid=0x17fddb50) >> >> glustershd.log:[2019-03-13 06:17:48.883825] E >> [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to >> 192.168.0.55:49158 failed (Connection timed out); disco >> nnecting socket >> glustershd.log:[2019-03-13 06:19:58.931798] E >> [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to >> 192.168.0.55:49158 failed (Connection timed out); disco >> nnecting socket >> glustershd.log:[2019-03-13 06:22:08.979829] E >> [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to >> 192.168.0.55:49158 failed (Connection timed out); disco >> nnecting socket >> glustershd.log:[2019-03-13 06:22:36.226847] E [MSGID: 114031] >> [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote >> operation failed [Transport endpoint >> is not connected] >> glustershd.log:[2019-03-13 06:22:36.306669] E [MSGID: 114031] >> [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote >> operation failed [Transport endpoint >> is not connected] >> glustershd.log:[2019-03-13 06:22:36.385257] E [MSGID: 114031] >> [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote >> operation failed [Transport endpoint >> is not connected] >> >> *WARNINGS:* >> *CWD: /var/log/glusterfs * >> *COMMAND: grep " W " *.log |grep "2019-03-13 06:"* >> >> (I can see a lot of this kind of message in the same period but I'm >> notifying you only one record for each type of warnings) >> >> glustershd.log:[2019-03-13 06:14:28.666772] W [MSGID: 114031] >> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-tier2-client-55: remote >> operation failed. Path: > 0f-f34d-4c25-bbe8-74bde0248d7e> (b6b35d0f-f34d-4c25-bbe8-74bde0248d7e). >> Key: (null) [Transport endpoint is not connected] >> >> glustershd.log:[2019-03-13 06:14:31.421576] W [MSGID: 122035] >> [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation >> with some subvolumes unavailable (2) >> >> glustershd.log:[2019-03-13 06:15:31.547417] W [MSGID: 122032] >> [ec-heald.c:266:ec_shd_index_sweep] 0-tier2-disperse-9: unable to get >> index-dir on tier2-client-55 [Operation >> now in progress] >> >> quota-mount-tier2.log:[2019-03-13 06:12:36.116277] W [MSGID: 101002] >> [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is >> deprecated, preferred is 'trans >> port.address-family', continuing with correction >> quota-mount-tier2.log:[2019-03-13 06:12:36.198430] W [MSGID: 101174] >> [graph.c:363:_log_if_unknown_option] 0-tier2-readdir-ahead: option >> 'parallel-readdir' is not recognized >> quota-mount-tier2.log:[2019-03-13 06:12:37.945007] W >> [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) >> [0x7f340892be25] -->/usr/sbin/glusterfs(gluste >> rfs_sigwaiter+0xe5) [0x55ef010164b5] >> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55ef0101632b] ) 0-: >> received signum (15), shutting down >> >> *CRITICALS:* >> *CWD: /var/log/glusterfs * >> *COMMAND: grep " C " *.log |grep "2019-03-13 06:"* >> >> no critical errors at 06:xx >> only one critical error during the day >> >> *[root at s06 glusterfs]# grep " C " *.log |grep "2019-03-13"* >> glustershd.log:[2019-03-13 02:21:29.126279] C >> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server >> 192.168.0.55:49158 has not responded in the last 42 seconds, >> disconnecting. >> >> >> Thank you very much for your help. >> Regards, >> Mauro >> >> On 12 Mar 2019, at 05:17, Raghavendra Gowdappa >> wrote: >> >> Was the suggestion to increase server.event-thread values tried? If yes, >> what were the results? >> >> On Mon, Mar 11, 2019 at 2:40 PM Mauro Tridici >> wrote: >> >>> Dear All, >>> >>> do you have any suggestions about the right way to "debug" this issue? >>> In attachment, the updated logs of ?s06" gluster server. >>> >>> I noticed a lot of intermittent warning and error messages. >>> >>> Thank you in advance, >>> Mauro >>> >>> >>> >>> On 4 Mar 2019, at 18:45, Raghavendra Gowdappa >>> wrote: >>> >>> >>> +Gluster Devel , +Gluster-users >>> >>> >>> I would like to point out another issue. Even if what I suggested >>> prevents disconnects, part of the solution would be only symptomatic >>> treatment and doesn't address the root cause of the problem. In most of the >>> ping-timer-expiry issues, the root cause is the increased load on bricks >>> and the inability of bricks to be responsive under high load. So, the >>> actual solution would be doing any or both of the following: >>> * identify the source of increased load and if possible throttle it. >>> Internal heal processes like self-heal, rebalance, quota heal are known to >>> pump traffic into bricks without much throttling (io-threads _might_ do >>> some throttling, but my understanding is its not sufficient). >>> * identify the reason for bricks to become unresponsive during load. >>> This may be fixable issues like not enough event-threads to read from >>> network or difficult to fix issues like fsync on backend fs freezing the >>> process or semi fixable issues (in code) like lock contention. >>> >>> So any genuine effort to fix ping-timer-issues (to be honest most of the >>> times they are not issues related to rpc/network) would involve performance >>> characterization of various subsystems on bricks and clients. Various >>> subsystems can include (but not necessarily limited to), underlying >>> OS/filesystem, glusterfs processes, CPU consumption etc >>> >>> regards, >>> Raghavendra >>> >>> On Mon, Mar 4, 2019 at 9:31 PM Mauro Tridici >>> wrote: >>> >>>> Thank you, let?s try! >>>> I will inform you about the effects of the change. >>>> >>>> Regards, >>>> Mauro >>>> >>>> On 4 Mar 2019, at 16:55, Raghavendra Gowdappa >>>> wrote: >>>> >>>> >>>> >>>> On Mon, Mar 4, 2019 at 8:54 PM Mauro Tridici >>>> wrote: >>>> >>>>> Hi Raghavendra, >>>>> >>>>> thank you for your reply. >>>>> Yes, you are right. It is a problem that seems to happen randomly. >>>>> At this moment, server.event-threads value is 4. I will try to >>>>> increase this value to 8. Do you think that it could be a valid value ? >>>>> >>>> >>>> Yes. We can try with that. You should see at least frequency of >>>> ping-timer related disconnects reduce with this value (even if it doesn't >>>> eliminate the problem completely). >>>> >>>> >>>>> Regards, >>>>> Mauro >>>>> >>>>> >>>>> On 4 Mar 2019, at 15:36, Raghavendra Gowdappa >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On Mon, Mar 4, 2019 at 8:01 PM Nithya Balachandran < >>>>> nbalacha at redhat.com> wrote: >>>>> >>>>>> Hi Mauro, >>>>>> >>>>>> It looks like some problem on s06. Are all your other nodes ok? Can >>>>>> you send us the gluster logs from this node? >>>>>> >>>>>> @Raghavendra G , do you have any idea as >>>>>> to how this can be debugged? Maybe running top ? Or debug brick logs? >>>>>> >>>>> >>>>> If we can reproduce the problem, collecting tcpdump on both ends of >>>>> connection will help. But, one common problem is these bugs are >>>>> inconsistently reproducible and hence we may not be able to capture tcpdump >>>>> at correct intervals. Other than that, we can try to collect some evidence >>>>> that poller threads were busy (waiting on locks). But, not sure what debug >>>>> data provides that information. >>>>> >>>>> From what I know, its difficult to collect evidence for this issue and >>>>> we could only reason about it. >>>>> >>>>> We can try a workaround though - try increasing server.event-threads >>>>> and see whether ping-timer expiry issues go away with an optimal value. If >>>>> that's the case, it kind of provides proof for our hypothesis. >>>>> >>>>> >>>>>> >>>>>> Regards, >>>>>> Nithya >>>>>> >>>>>> On Mon, 4 Mar 2019 at 15:25, Mauro Tridici >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> some minutes ago I received this message from NAGIOS server >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ****** Nagios *****Notification Type: PROBLEMService: Brick - >>>>>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: CRITICALDate/Time: Mon >>>>>>> Mar 4 10:25:33 CET 2019Additional Info:CHECK_NRPE STATE CRITICAL: Socket >>>>>>> timeout after 10 seconds.* >>>>>>> >>>>>>> I checked the network, RAM and CPUs usage on s06 node and everything >>>>>>> seems to be ok. >>>>>>> No bricks are in error state. In /var/log/messages, I detected again >>>>>>> a crash of ?check_vol_utili? that I think it is a module used by NRPE >>>>>>> executable (that is the NAGIOS client). >>>>>>> >>>>>>> Mar 4 10:15:29 s06 kernel: traps: check_vol_utili[161224] general >>>>>>> protection ip:7facffa0a66d sp:7ffe9f4e6fc0 error:0 in >>>>>>> libglusterfs.so.0.0.1[7facff9b7000+f7000] >>>>>>> Mar 4 10:15:29 s06 abrt-hook-ccpp: Process 161224 (python2.7) of >>>>>>> user 0 killed by SIGSEGV - dumping core >>>>>>> Mar 4 10:15:29 s06 abrt-server: Generating core_backtrace >>>>>>> Mar 4 10:15:29 s06 abrt-server: Error: Unable to open './coredump': >>>>>>> No such file or directory >>>>>>> Mar 4 10:16:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:16:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:16:01 s06 systemd: Started Session 201010 of user root. >>>>>>> Mar 4 10:16:01 s06 systemd: Starting Session 201010 of user root. >>>>>>> Mar 4 10:16:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 4 10:16:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 4 10:16:24 s06 abrt-server: Duplicate: UUID >>>>>>> Mar 4 10:16:24 s06 abrt-server: DUP_OF_DIR: >>>>>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>>>> Mar 4 10:16:24 s06 abrt-server: Deleting problem directory >>>>>>> ccpp-2019-03-04-10:15:29-161224 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>>>> Mar 4 10:16:24 s06 abrt-server: Generating core_backtrace >>>>>>> Mar 4 10:16:24 s06 abrt-server: Error: Unable to open './coredump': >>>>>>> No such file or directory >>>>>>> Mar 4 10:16:24 s06 abrt-server: Cannot notify >>>>>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>>>>> 'report_uReport' exited with 1 >>>>>>> Mar 4 10:16:24 s06 abrt-hook-ccpp: Process 161391 (python2.7) of >>>>>>> user 0 killed by SIGABRT - dumping core >>>>>>> Mar 4 10:16:25 s06 abrt-server: Generating core_backtrace >>>>>>> Mar 4 10:16:25 s06 abrt-server: Error: Unable to open './coredump': >>>>>>> No such file or directory >>>>>>> Mar 4 10:17:01 s06 systemd: Created slice User Slice of root. >>>>>>> >>>>>>> Also, I noticed the following errors that I think are very critical: >>>>>>> >>>>>>> Mar 4 10:21:12 s06 glustershd[20355]: [2019-03-04 09:21:12.954798] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: >>>>>>> server 192.168.0.55:49158 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:22:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:22:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:22:01 s06 systemd: Started Session 201017 of user root. >>>>>>> Mar 4 10:22:01 s06 systemd: Starting Session 201017 of user root. >>>>>>> Mar 4 10:22:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 4 10:22:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 4 10:22:03 s06 glustershd[20355]: [2019-03-04 09:22:03.964120] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: >>>>>>> server 192.168.0.54:49165 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:23:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:23:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:23:01 s06 systemd: Started Session 201018 of user root. >>>>>>> Mar 4 10:23:01 s06 systemd: Starting Session 201018 of user root. >>>>>>> Mar 4 10:23:02 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 4 10:23:02 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 4 10:24:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:24:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:24:01 s06 systemd: Started Session 201019 of user root. >>>>>>> Mar 4 10:24:01 s06 systemd: Starting Session 201019 of user root. >>>>>>> Mar 4 10:24:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 4 10:24:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 4 10:24:03 s06 glustershd[20355]: [2019-03-04 09:24:03.982502] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-16: >>>>>>> server 192.168.0.52:49158 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746109] C >>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-3: server >>>>>>> 192.168.0.51:49153 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746215] C >>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server >>>>>>> 192.168.0.52:49156 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746260] C >>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-21: server >>>>>>> 192.168.0.51:49159 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746296] C >>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server >>>>>>> 192.168.0.52:49161 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746413] C >>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server >>>>>>> 192.168.0.54:49165 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:07 s06 glustershd[20355]: [2019-03-04 09:24:07.982952] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-45: >>>>>>> server 192.168.0.54:49155 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:18 s06 glustershd[20355]: [2019-03-04 09:24:18.990929] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: >>>>>>> server 192.168.0.52:49161 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:24:31 s06 glustershd[20355]: [2019-03-04 09:24:31.995781] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-20: >>>>>>> server 192.168.0.53:49159 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:25:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:25:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:25:01 s06 systemd: Started Session 201020 of user root. >>>>>>> Mar 4 10:25:01 s06 systemd: Starting Session 201020 of user root. >>>>>>> Mar 4 10:25:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 4 10:25:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 4 10:25:57 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 4 10:25:57 s06 systemd: Starting User Slice of root. >>>>>>> Mar 4 10:25:57 s06 systemd-logind: New session 201021 of user root. >>>>>>> Mar 4 10:25:57 s06 systemd: Started Session 201021 of user root. >>>>>>> Mar 4 10:25:57 s06 systemd: Starting Session 201021 of user root. >>>>>>> Mar 4 10:26:01 s06 systemd: Started Session 201022 of user root. >>>>>>> Mar 4 10:26:01 s06 systemd: Starting Session 201022 of user root. >>>>>>> Mar 4 10:26:21 s06 nrpe[162388]: Error: Could not complete SSL >>>>>>> handshake with 192.168.1.56: 5 >>>>>>> Mar 4 10:27:01 s06 systemd: Started Session 201023 of user root. >>>>>>> Mar 4 10:27:01 s06 systemd: Starting Session 201023 of user root. >>>>>>> Mar 4 10:28:01 s06 systemd: Started Session 201024 of user root. >>>>>>> Mar 4 10:28:01 s06 systemd: Starting Session 201024 of user root. >>>>>>> Mar 4 10:29:01 s06 systemd: Started Session 201025 of user root. >>>>>>> Mar 4 10:29:01 s06 systemd: Starting Session 201025 of user root. >>>>>>> >>>>>>> But, unfortunately, I don?t understand why it is happening. >>>>>>> Now, NAGIOS server shows that s06 status is ok: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ****** Nagios *****Notification Type: RECOVERYService: Brick - >>>>>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: OKDate/Time: Mon Mar 4 >>>>>>> 10:35:23 CET 2019Additional Info:OK: Brick /gluster/mnt2/brick is up* >>>>>>> >>>>>>> Nothing is changed from RAM, CPUs, and NETWORK point of view. >>>>>>> /var/log/message file has been updated: >>>>>>> >>>>>>> Mar 4 10:32:01 s06 systemd: Starting Session 201029 of user root. >>>>>>> Mar 4 10:32:30 s06 glustershd[20355]: [2019-03-04 09:32:30.069082] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: >>>>>>> server 192.168.0.52:49156 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:32:55 s06 glustershd[20355]: [2019-03-04 09:32:55.074689] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-66: >>>>>>> server 192.168.0.54:49167 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 4 10:33:01 s06 systemd: Started Session 201030 of user root. >>>>>>> Mar 4 10:33:01 s06 systemd: Starting Session 201030 of user root. >>>>>>> Mar 4 10:34:01 s06 systemd: Started Session 201031 of user root. >>>>>>> Mar 4 10:34:01 s06 systemd: Starting Session 201031 of user root. >>>>>>> Mar 4 10:35:01 s06 nrpe[162562]: Could not read request from client >>>>>>> 192.168.1.56, bailing out... >>>>>>> Mar 4 10:35:01 s06 nrpe[162562]: INFO: SSL Socket Shutdown. >>>>>>> Mar 4 10:35:01 s06 systemd: Started Session 201032 of user root. >>>>>>> Mar 4 10:35:01 s06 systemd: Starting Session 201032 of user root. >>>>>>> >>>>>>> Could you please help me to understand what it?s happening ? >>>>>>> Thank you in advance. >>>>>>> >>>>>>> Rergards, >>>>>>> Mauro >>>>>>> >>>>>>> >>>>>>> On 1 Mar 2019, at 12:17, Mauro Tridici >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> Thank you, Milind. >>>>>>> I executed the instructions you suggested: >>>>>>> >>>>>>> - grep ?blocked for? /var/log/messages on s06 returns no output (no >>>>>>> ?blocked? word is detected in messages file); >>>>>>> - in /var/log/messages file I can see this kind of error repeated >>>>>>> for a lot of times: >>>>>>> >>>>>>> Mar 1 08:43:01 s06 systemd: Starting Session 196071 of user root. >>>>>>> Mar 1 08:43:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 1 08:43:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 1 08:43:02 s06 kernel: traps: check_vol_utili[57091] general >>>>>>> protection ip:7f88e76ee66d sp:7ffe5a5bcc30 error:0 in >>>>>>> libglusterfs.so.0.0.1[7f88e769b000+f7000] >>>>>>> Mar 1 08:43:02 s06 abrt-hook-ccpp: Process 57091 (python2.7) of >>>>>>> user 0 killed by SIGSEGV - dumping core >>>>>>> Mar 1 08:43:02 s06 abrt-server: Generating core_backtrace >>>>>>> Mar 1 08:43:02 s06 abrt-server: Error: Unable to open './coredump': >>>>>>> No such file or directory >>>>>>> Mar 1 08:43:58 s06 abrt-server: Duplicate: UUID >>>>>>> Mar 1 08:43:58 s06 abrt-server: DUP_OF_DIR: >>>>>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>>>> Mar 1 08:43:58 s06 abrt-server: Deleting problem directory >>>>>>> ccpp-2019-03-01-08:43:02-57091 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Activating service >>>>>>> name='org.freedesktop.problems' (using servicehelper) >>>>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Successfully activated >>>>>>> service 'org.freedesktop.problems' >>>>>>> Mar 1 08:43:58 s06 abrt-server: Generating core_backtrace >>>>>>> Mar 1 08:43:58 s06 abrt-server: Error: Unable to open './coredump': >>>>>>> No such file or directory >>>>>>> Mar 1 08:43:58 s06 abrt-server: Cannot notify >>>>>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>>>>> 'report_uReport' exited with 1 >>>>>>> Mar 1 08:44:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 1 08:44:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 1 08:44:01 s06 systemd: Started Session 196072 of user root. >>>>>>> Mar 1 08:44:01 s06 systemd: Starting Session 196072 of user root. >>>>>>> Mar 1 08:44:01 s06 systemd: Removed slice User Slice of root. >>>>>>> >>>>>>> - in /var/log/messages file I can see also 4 errors related to other >>>>>>> cluster servers: >>>>>>> >>>>>>> Mar 1 11:05:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 1 11:05:01 s06 systemd: Started Session 196230 of user root. >>>>>>> Mar 1 11:05:01 s06 systemd: Starting Session 196230 of user root. >>>>>>> Mar 1 11:05:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 1 11:05:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 1 11:05:59 s06 glustershd[70117]: [2019-03-01 10:05:59.347094] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-33: >>>>>>> server 192.168.0.51:49163 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 1 11:06:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 1 11:06:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 1 11:06:01 s06 systemd: Started Session 196231 of user root. >>>>>>> Mar 1 11:06:01 s06 systemd: Starting Session 196231 of user root. >>>>>>> Mar 1 11:06:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 1 11:06:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 1 11:06:12 s06 glustershd[70117]: [2019-03-01 10:06:12.351319] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-1: >>>>>>> server 192.168.0.52:49153 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 1 11:06:38 s06 glustershd[70117]: [2019-03-01 10:06:38.356920] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-7: >>>>>>> server 192.168.0.52:49155 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 1 11:07:01 s06 systemd: Created slice User Slice of root. >>>>>>> Mar 1 11:07:01 s06 systemd: Starting User Slice of root. >>>>>>> Mar 1 11:07:01 s06 systemd: Started Session 196232 of user root. >>>>>>> Mar 1 11:07:01 s06 systemd: Starting Session 196232 of user root. >>>>>>> Mar 1 11:07:01 s06 systemd: Removed slice User Slice of root. >>>>>>> Mar 1 11:07:01 s06 systemd: Stopping User Slice of root. >>>>>>> Mar 1 11:07:36 s06 glustershd[70117]: [2019-03-01 10:07:36.366259] >>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-0: >>>>>>> server 192.168.0.51:49152 has not responded in the last 42 seconds, >>>>>>> disconnecting. >>>>>>> Mar 1 11:08:01 s06 systemd: Created slice User Slice of root. >>>>>>> >>>>>>> No ?blocked? word is in /var/log/messages files on other cluster >>>>>>> servers. >>>>>>> In attachment, the /var/log/messages file from s06 server. >>>>>>> >>>>>>> Thank you in advance, >>>>>>> Mauro >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 1 Mar 2019, at 11:47, Milind Changire >>>>>>> wrote: >>>>>>> >>>>>>> The traces of very high disk activity on the servers are often found >>>>>>> in /var/log/messages >>>>>>> You might want to grep for "blocked for" in /var/log/messages on s06 >>>>>>> and correlate the timestamps to confirm the unresponsiveness as reported in >>>>>>> gluster client logs. >>>>>>> In cases of high disk activity, although the operating system >>>>>>> continues to respond to ICMP pings, the processes writing to disks often >>>>>>> get blocked to a large flush to the disk which could span beyond 42 seconds >>>>>>> and hence result in ping-timer-expiry logs. >>>>>>> >>>>>>> As a side note: >>>>>>> If you indeed find gluster processes being blocked in >>>>>>> /var/log/messages, you might want to tweak sysctl tunables called >>>>>>> vm.dirty_background_ratio or vm.dirty_background_bytes to a smaller value >>>>>>> than the existing. Please read up more on those tunables before touching >>>>>>> the settings. >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 1, 2019 at 4:06 PM Mauro Tridici >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> in attachment the client log captured after changing >>>>>>>> network.ping-timeout option. >>>>>>>> I noticed this error involving server 192.168.0.56 (s06) >>>>>>>> >>>>>>>> [2019-03-01 09:23:36.077287] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >>>>>>>> 0-tier2-client-71: changing ping timeout to 42 (from 0) >>>>>>>> [2019-03-01 09:23:36.078213] I >>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>> volfile,continuing >>>>>>>> [2019-03-01 09:23:36.078432] I >>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>> volfile,continuing >>>>>>>> [2019-03-01 09:23:36.092357] I >>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>> volfile,continuing >>>>>>>> [2019-03-01 09:23:36.094146] I >>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>> volfile,continuing >>>>>>>> [2019-03-01 10:06:24.708082] C >>>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-50: server >>>>>>>> 192.168.0.56:49156 has not responded in the last 42 seconds, >>>>>>>> disconnecting. >>>>>>>> >>>>>>>> I don?t know why it happens, s06 server seems to be reachable. >>>>>>>> >>>>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>>>> Trying 192.168.0.56... >>>>>>>> Connected to 192.168.0.56. >>>>>>>> Escape character is '^]'. >>>>>>>> ^CConnection closed by foreign host. >>>>>>>> [athena_login2][/users/home/sysm02/]> ping 192.168.0.56 >>>>>>>> PING 192.168.0.56 (192.168.0.56) 56(84) bytes of data. >>>>>>>> 64 bytes from 192.168.0.56: icmp_seq=1 ttl=64 time=0.116 ms >>>>>>>> 64 bytes from 192.168.0.56: icmp_seq=2 ttl=64 time=0.101 ms >>>>>>>> >>>>>>>> --- 192.168.0.56 ping statistics --- >>>>>>>> 2 packets transmitted, 2 received, 0% packet loss, time 1528ms >>>>>>>> rtt min/avg/max/mdev = 0.101/0.108/0.116/0.012 ms >>>>>>>> >>>>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>>>> Trying 192.168.0.56... >>>>>>>> Connected to 192.168.0.56. >>>>>>>> Escape character is '^]'. >>>>>>>> >>>>>>>> Thank you for your help, >>>>>>>> Mauro >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 1 Mar 2019, at 10:29, Mauro Tridici >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> thank you for the explanation. >>>>>>>> I just changed network.ping-timeout option to default value >>>>>>>> (network.ping-timeout=42). >>>>>>>> >>>>>>>> I will check the logs to see if the errors will appear again. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Mauro >>>>>>>> >>>>>>>> On 1 Mar 2019, at 04:43, Milind Changire >>>>>>>> wrote: >>>>>>>> >>>>>>>> network.ping-timeout should not be set to zero for non-glusterd >>>>>>>> clients. >>>>>>>> glusterd is a special case for which ping-timeout is set to zero >>>>>>>> via /etc/glusterfs/glusterd.vol >>>>>>>> >>>>>>>> Setting network.ping-timeout to zero disables arming of the ping >>>>>>>> timer for connections. This disables testing the connection for >>>>>>>> responsiveness and hence avoids proactive fail-over. >>>>>>>> >>>>>>>> Please reset network.ping-timeout to a non-zero positive value, eg. >>>>>>>> 42 >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Feb 28, 2019 at 5:07 PM Nithya Balachandran < >>>>>>>> nbalacha at redhat.com> wrote: >>>>>>>> >>>>>>>>> Adding Raghavendra and Milind to comment on this. >>>>>>>>> >>>>>>>>> What is the effect of setting network.ping-timeout to 0 and should >>>>>>>>> it be set back to 42? >>>>>>>>> Regards, >>>>>>>>> Nithya >>>>>>>>> >>>>>>>>> On Thu, 28 Feb 2019 at 16:01, Mauro Tridici >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Nithya, >>>>>>>>>> >>>>>>>>>> sorry for the late. >>>>>>>>>> network.ping-timeout has been set to 0 in order to try to solve >>>>>>>>>> some timeout problems, but it didn?t help. >>>>>>>>>> I can set it to the default value. >>>>>>>>>> >>>>>>>>>> Can I proceed with the change? >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> Mauro >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 28 Feb 2019, at 04:41, Nithya Balachandran < >>>>>>>>>> nbalacha at redhat.com> wrote: >>>>>>>>>> >>>>>>>>>> Hi Mauro, >>>>>>>>>> >>>>>>>>>> Is network.ping-timeout still set to 0. The default value is 42. >>>>>>>>>> Is there a particular reason why this was changed? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Nithya >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, 27 Feb 2019 at 21:32, Mauro Tridici < >>>>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi Xavi, >>>>>>>>>>> >>>>>>>>>>> thank you for the detailed explanation and suggestions. >>>>>>>>>>> Yes, transport.listen-backlog option is still set to 1024. >>>>>>>>>>> >>>>>>>>>>> I will check the network and connectivity status using ?ping? >>>>>>>>>>> and ?telnet? as soon as the errors will come back again. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Mauro >>>>>>>>>>> >>>>>>>>>>> Il giorno 27 feb 2019, alle ore 16:42, Xavi Hernandez < >>>>>>>>>>> jahernan at redhat.com> ha scritto: >>>>>>>>>>> >>>>>>>>>>> Hi Mauro, >>>>>>>>>>> >>>>>>>>>>> those errors say that the mount point is not connected to some >>>>>>>>>>> of the bricks while executing operations. I see references to 3rd and 6th >>>>>>>>>>> bricks of several disperse sets, which seem to map to server s06. For some >>>>>>>>>>> reason, gluster is having troubles connecting from the client machine to >>>>>>>>>>> that particular server. At the end of the log I see that after long time a >>>>>>>>>>> reconnect is done to both of them. However little after, other bricks from >>>>>>>>>>> the s05 get disconnected and a reconnect times out. >>>>>>>>>>> >>>>>>>>>>> That's really odd. It seems like if server/communication is cut >>>>>>>>>>> to s06 for some time, then restored, and then the same happens to the next >>>>>>>>>>> server. >>>>>>>>>>> >>>>>>>>>>> If the servers are really online and it's only a communication >>>>>>>>>>> issue, it explains why server memory and network has increased: if the >>>>>>>>>>> problem only exists between the client and servers, any write made by the >>>>>>>>>>> client will automatically mark the file as damaged, since some of the >>>>>>>>>>> servers have not been updated. Since self-heal runs from the server nodes, >>>>>>>>>>> they will probably be correctly connected to all bricks, which allows them >>>>>>>>>>> to heal the just damaged file, which increases memory and network usage. >>>>>>>>>>> >>>>>>>>>>> I guess you still have transport.listen-backlog set to 1024, >>>>>>>>>>> right ? >>>>>>>>>>> >>>>>>>>>>> Just to try to identify if the problem really comes from >>>>>>>>>>> network, can you check if you lose some pings from the client to all of the >>>>>>>>>>> servers while you are seeing those errors in the log file ? >>>>>>>>>>> >>>>>>>>>>> You can also check if during those errors, you can telnet to the >>>>>>>>>>> port of the brick from the client. >>>>>>>>>>> >>>>>>>>>>> Xavi >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Feb 26, 2019 at 10:17 AM Mauro Tridici < >>>>>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Nithya, >>>>>>>>>>>> >>>>>>>>>>>> ?df -h? operation is not still slow, but no users are using the >>>>>>>>>>>> volume, RAM and NETWORK usage is ok on the client node. >>>>>>>>>>>> >>>>>>>>>>>> I was worried about this kind of warnings/errors: >>>>>>>>>>>> >>>>>>>>>>>> [2019-02-25 10:59:00.664323] W [MSGID: 122035] >>>>>>>>>>>> [ec-common.c:571:ec_child_select] 0-tier2-disperse-6: Executing operation >>>>>>>>>>>> with some subvolumes unavailable (20) >>>>>>>>>>>> >>>>>>>>>>>> [2019-02-26 03:11:35.212603] E >>>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>>>> called at 2019-02-26 03:10:56.549903 (xid=0x106f1c5) >>>>>>>>>>>> >>>>>>>>>>>> [2019-02-26 03:13:03.313831] E >>>>>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-50: connection to >>>>>>>>>>>> 192.168.0.56:49156 failed (Timeout della connessione); >>>>>>>>>>>> disconnecting socket >>>>>>>>>>>> >>>>>>>>>>>> It seems that some subvolumes are not available and >>>>>>>>>>>> 192.168.0.56 server (s06) is not reachable. >>>>>>>>>>>> But gluster servers are up&running and bricks are ok. >>>>>>>>>>>> >>>>>>>>>>>> In attachment the updated tier2.log file. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thank you. >>>>>>>>>>>> Regards, >>>>>>>>>>>> Mauro >>>>>>>>>>>> >>>>>>>>>>>> Il giorno 26 feb 2019, alle ore 04:03, Nithya Balachandran < >>>>>>>>>>>> nbalacha at redhat.com> ha scritto: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I see a lot of EC messages in the log but they don't seem very >>>>>>>>>>>> serious. Xavi, can you take a look? >>>>>>>>>>>> >>>>>>>>>>>> The only errors I see are: >>>>>>>>>>>> [2019-02-25 10:58:45.519871] E >>>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>>>> called at 2019-02-25 10:57:47.429969 (xid=0xd26fe7) >>>>>>>>>>>> [2019-02-25 10:58:51.461493] E >>>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>>>> 0-tier2-client-41: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>>>> called at 2019-02-25 10:57:47.499174 (xid=0xf47d6a) >>>>>>>>>>>> [2019-02-25 11:07:57.152874] E >>>>>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-70: connection to >>>>>>>>>>>> 192.168.0.55:49163 failed (Timeout della connessione); >>>>>>>>>>>> disconnecting socket >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Is the df -h operation still slow? If yes, can you take a >>>>>>>>>>>> tcpdump of the client while running df -h and send that across? >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Nithya >>>>>>>>>>>> >>>>>>>>>>>> On Mon, 25 Feb 2019 at 17:27, Mauro Tridici < >>>>>>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry, some minutes after my last mail message, I noticed that >>>>>>>>>>>>> ?df -h? command hanged for a while before returns the prompt. >>>>>>>>>>>>> Yesterday, everything was ok in the gluster client log, but, >>>>>>>>>>>>> today, I see a lot of errors (please, take a look to the attached file). >>>>>>>>>>>>> >>>>>>>>>>>>> On the client node, I detected an important RAM e NETWORK >>>>>>>>>>>>> usage. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Do you think that the errors have been caused by the client >>>>>>>>>>>>> resources usage? >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you in advance, >>>>>>>>>>>>> Mauro >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>> >>>> >>>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkeithle at redhat.com Thu Mar 14 13:59:04 2019 From: kkeithle at redhat.com (Kaleb Keithley) Date: Thu, 14 Mar 2019 09:59:04 -0400 Subject: [Gluster-devel] Github#268 Compatibility with Alpine Linux In-Reply-To: References: Message-ID: On Thu, Mar 14, 2019 at 1:47 AM Amar Tumballi Suryanarayan < atumball at redhat.com> wrote: > I tried this recently, and the issue of rpcgen is real, and is not > straight-forward is what I felt. Would like to pick this up after > glusterfs-6.0 release. > rpcgen is in its own separate package. `apk update && apk add rpcgen` (Alpine 3.9.2) (Not unlike Fedora28+ and RHEL8 where rpcgen has been removed from glibc. I expect other Linux distributions to eventually follow suit and remove rpcgen from their glibc.) But alpine ships rpcgen-2.3.2 (versus Fedora's 1.4.x and rhel8's 1.3.1) and it doesn't like the "hyper" data type used in glusterfs4-xdr.x (and presumably in changelog-xdr.x but I didn't get that far.) According to the Fedora rpm .spec Source:, Fedora's rpcgen comes from https://github.com/thkukuk/rpcsvc-proto/ `apk info rpcgen` says it comes from http://linux-nfs.org, I guess from http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=tree;f=tools/rpcgen;h=64ebfa4a7ddd0cf1b4b363f27aba426d571c422e;hb=HEAD Both appear to be originally derived from BSD's rpcgen. It'd be nice if the Linux community could settle on a single version of rpcgen. Maybe someone out in the community that has a vested interest in Alpine Linux would like to dig into it further! -- Kaleb > -Amar > > On Tue, Mar 12, 2019 at 8:17 AM Sankarshan Mukhopadhyay < > sankarshan.mukhopadhyay at gmail.com> wrote: > >> Saw some recent activity on >> - is there a plan to >> address this or, should the interested users be informed about other >> plans? >> >> /s >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-devel >> >> >> > > -- > Amar Tumballi (amarts) > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From jenkins at build.gluster.org Mon Mar 18 01:45:02 2019 From: jenkins at build.gluster.org (jenkins at build.gluster.org) Date: Mon, 18 Mar 2019 01:45:02 +0000 (UTC) Subject: [Gluster-devel] Weekly Untriaged Bugs Message-ID: <2093911699.11.1552873503350.JavaMail.jenkins@jenkins-el7.rht.gluster.org> [...truncated 6 lines...] https://bugzilla.redhat.com/1688226 / core: Brick Still Died After Restart Glusterd & Glusterfsd Services https://bugzilla.redhat.com/1679904 / core: client log flooding with intentional socket shutdown message when a brick is down https://bugzilla.redhat.com/1685023 / core: FD processes for larger files are not closed soon after FOP finished https://bugzilla.redhat.com/1686396 / core: ls and rm run on contents of same directory from a single mount point results in ENOENT errors https://bugzilla.redhat.com/1683815 / core: Memory leak when peer detach fails https://bugzilla.redhat.com/1679744 / core: Minio gateway nas does not work with 2 + 1 dispersed volumes https://bugzilla.redhat.com/1678640 / core: Running 'control-cpu-load.sh' prevents CTDB starting https://bugzilla.redhat.com/1683526 / glusterd: rebalance start command doesn't throw up error message if the command fails https://bugzilla.redhat.com/1686353 / libgfapi: flooding of "dict is NULL" logging https://bugzilla.redhat.com/1687063 / locks: glusterd :symbol lookup error: undefined symbol :use_spinlocks https://bugzilla.redhat.com/1679169 / md-cache: Integer Overflow possible in md-cache.c due to data type inconsistency https://bugzilla.redhat.com/1679170 / md-cache: Integer Overflow possible in md-cache.c due to data type inconsistency https://bugzilla.redhat.com/1678378 / project-infrastructure: Add a nightly build verification job in Jenkins for release-6 https://bugzilla.redhat.com/1685576 / project-infrastructure: DNS delegation record for rhhi-dev.gluster.org https://bugzilla.redhat.com/1685813 / project-infrastructure: Not able to run centos-regression getting exception error https://bugzilla.redhat.com/1686461 / quota: Quotad.log filled with 0-dict is not sent on wire [Invalid argument] messages https://bugzilla.redhat.com/1682925 / replicate: Gluster volumes never heal during oVirt 4.2->4.3 upgrade https://bugzilla.redhat.com/1683317 / tests: ./tests/bugs/glusterfs/bug-866459.t failing on s390x [...truncated 2 lines...] -------------- next part -------------- A non-text attachment was scrubbed... Name: build.log Type: application/octet-stream Size: 2376 bytes Desc: not available URL: From amudhan83 at gmail.com Fri Mar 1 07:58:08 2019 From: amudhan83 at gmail.com (Amudhan P) Date: Fri, 01 Mar 2019 07:58:08 -0000 Subject: [Gluster-devel] [Gluster-users] Bitrot: Time of signing depending on the file size??? In-Reply-To: References: Message-ID: Hi David, I have also tested the bitrot signature process by default it takes < 250 KB/s. regards Amudhan P On Fri, Mar 1, 2019 at 1:19 PM David Spisla wrote: > Hello folks, > > I did some observations concerning the bitrot daemon. It seems to be that > the bitrot signer is signing files depending on file size. I copied files > with different sizes into a volume and I was wonderung because the files > get their signature not the same time (I keep the expiry time default with > 120). Here are some examples: > > 300 KB file ~2-3 m > 70 MB file ~ 40 m > 115 MB file ~ 1 Sh > 800 MB file ~ 4,5 h > > What is the expected behaviour here? > Why does it take so long to sign a 800MB file? > What about 500GB or 1TB? > Is there a way to speed up the sign process? > > My ambition is to understand this observation > > Regards > David Spisla > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From amudhan83 at gmail.com Fri Mar 1 13:56:32 2019 From: amudhan83 at gmail.com (Amudhan P) Date: Fri, 01 Mar 2019 13:56:32 -0000 Subject: [Gluster-devel] [Gluster-users] Bitrot: Time of signing depending on the file size??? In-Reply-To: References: Message-ID: Hi David, Once after file write completes (fd closed) bitrot process will wait for 120 seconds and if there no fd is opened for the file it will trigger the signer process. considering the signer, process start and end time file read speed was < 250KB/s. To increase bitrot signer read speed need to modify the bitrot source file. regards Amudhan On Fri, Mar 1, 2019 at 5:49 PM David Spisla wrote: > Hello Amudhan, > > What does exactly mean "it takes < 250KB/s"? > I figured out this discussion between you and Kotresh: > https://lists.gluster.org/pipermail/gluster-users/2016-September/028354.html > Kotresh mentioned there that the problem is because for some files fd > process are still up in the brick process list. Bitrot signer can only sign > a file if the fd is closed. And according to my observations it seems to be > that as bigger a file is as longer the fd is still up. I could verify this > with a 500MiB file and some smaller files. After a specific time only the > fd for the 500MiB was up and the file still had no signature, for the > smaller files there were no fds and they already had a signature. > > Does anybody know what is the reason for this? For me it looks loke a bug. > > Regards > David > > Am Fr., 1. M?rz 2019 um 08:58 Uhr schrieb Amudhan P : > >> Hi David, >> >> I have also tested the bitrot signature process by default it takes < 250 >> KB/s. >> >> regards >> Amudhan P >> >> >> On Fri, Mar 1, 2019 at 1:19 PM David Spisla wrote: >> >>> Hello folks, >>> >>> I did some observations concerning the bitrot daemon. It seems to be >>> that the bitrot signer is signing files depending on file size. I copied >>> files with different sizes into a volume and I was wonderung because the >>> files get their signature not the same time (I keep the expiry time default >>> with 120). Here are some examples: >>> >>> 300 KB file ~2-3 m >>> 70 MB file ~ 40 m >>> 115 MB file ~ 1 Sh >>> 800 MB file ~ 4,5 h >>> >>> What is the expected behaviour here? >>> Why does it take so long to sign a 800MB file? >>> What about 500GB or 1TB? >>> Is there a way to speed up the sign process? >>> >>> My ambition is to understand this observation >>> >>> Regards >>> David Spisla >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Wed Mar 6 16:51:26 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Wed, 06 Mar 2019 18:51:26 +0200 Subject: [Gluster-devel] [Gluster-users] Gluster : Improvements on "heal info" command Message-ID: <1arqcq3vmp99pobus3sb2kaw.1551891086158@email.android.com> Hi , This sounds nice. I would like to ask if the order is starting from the local node's bricks first ? (I am talking about --brick=one) Best Regards, Strahil NikolovOn Mar 5, 2019 10:51, Ashish Pandey wrote: > > Hi All, > > We have observed and heard from gluster users about the long time "heal info" command takes. > Even when we all want to know if a gluster volume is healthy or not, it takes time to list down all the files from all the bricks after which we can be > sure if the volume is healthy or not. > Here, we have come up with some options for "heal info" command which provide report quickly and reliably. > > gluster v heal vol info --subvol=[number of the subvol] --brick=[one,all] > -------- > > Problem: "gluster v heal info" command picks each subvolume and checks the .glusterfs/indices/xattrop folder of? every brick of that subvolume to find out if there is any entry > which needs to be healed. It picks the entry and takes a lock? on that entry to check xattrs to find out if that entry actually needs heal or not. > This LOCK->CHECK-XATTR->UNLOCK cycle takes lot of time for each file. > > Let's consider two most often seen cases for which we use "heal info" and try to understand the improvements. > > Case -1 : Consider 4+2 EC volume and all the bricks on 6 different nodes. > A brick of the volume is down and client has written 10000 files on one of the mount point of this volume. Entries for these 10K files will be created on ".glusterfs/indices/xattrop" > on all the rest of 5 bricks. Now, brick is UP and when we use "heal info" command for this volume, it goes to all the bricks and picks these 10K file entries and > goes through LOCK->CHECK-XATTR->UNLOCK cycle for all the files. This happens for all the bricks, that means, we check 50K files and perform the?LOCK->CHECK-XATTR->UNLOCK cycle 50K times, > while only 10K entries were sufficient to check. It is a very time consuming operation. If IO"s are happening one some of the new files, we check these files also which will add the time. > Here, all we wanted to know if our volume has been healed and healthy. > > Solution : Whenever a brick goes down and comes up and when we use "heal info" command, our *main intention* is to find out if the volume is *healthy* or *unhealthy*. A volume is unhealthy even if one > file is not healthy. So, we should scan bricks one by one and as soon as we find that one brick is having some entries which require to be healed, we can come out and list the files and say the volume is not > healthy. No need to scan rest of the bricks. That's where "--brick=[one,all]" option has been introduced. > > "gluster v heal vol info --brick=[one,all]" > "one" - It will scan the brick sequentially and as soon as it will find any unhealthy entries, it will list it out and stop scanning other bricks. > "all" - It will act just like current behavior and provide all the files from all the bricks. If we do not provide this option, default (current) behavior will be applicable. > > Case -2 : Consider 24 X (4+2) EC volume. Let's say one brick from *only one* of the sub volume has been replaced and a heal has been triggered. > To know if the volume is in healthy state, we go to each brick of *each and every sub volume* and check if there are any entries in ".glusterfs/indices/xattrop" folder which need heal or not. > If we know which sub volume participated in brick replacement, we just need to check health of that sub volume and not query/check other sub volumes. > > If several clients are writing number of files on this volume, an entry for each of these files will be created in? .glusterfs/indices/xattrop and "heal info' > command will go through LOCK->CHECK-XATTR->UNLOCK cycle to find out if these entries need heal or not which takes lot of time. > In addition to this a client will also see performance drop as it will have to release and take lock again. > > Solution: Provide an option to mention number of sub volume for which we want to check heal info. > > "gluster v heal vol info --subvol=? " > Here, --subvol will be given number of the subvolume we want to check. > Example: > "gluster v heal vol info --subvol=1 " > > > =================================== > Performance Data - > A quick performance test done on standalone system. > > Type: Distributed-Disperse > Volume ID: ea40eb13-d42c-431c-9c89-0153e834e67e > Status: Started > Snapshot Count: 0 > Number of Bricks: 2 x (4 + 2) = 12 > Transport-type: tcp > Bricks: > Brick1: apandey:/home/apandey/bricks/gluster/vol-1 > Brick2: apandey:/home/apandey/bricks/gluster/vol-2 > Brick3: apandey:/home/apandey/bricks/gluster/vol-3 > Brick4: apandey:/home/apandey/bricks/gluster/vol-4 > Brick5: apandey:/home/apandey/bricks/gluster/vol-5 > Brick6: apandey:/home/apandey/bricks/gluster/vol-6 > Brick7: apandey:/home/apandey/bricks/gluster/new-1 > Brick8: apandey:/home/apandey/bricks/gluster/new-2 > Brick9: apandey:/home/apandey/bricks/gluster/new-3 > Brick10: apandey:/home/apandey/bricks/gluster/new-4 > Brick11: apandey:/home/apandey/bricks/gluster/new-5 > Brick12: apandey:/home/apandey/bricks/gluster/new-6 > > Just disabled the shd to get the data - > > Killed one brick each from two subvolumes and wrote 2000 files on mount point. > [root at apandey vol]# for i in {1..2000};do echo abc >> file-$i; done > > Start the volume using force option and get the heal info. Following is the data - > > [root at apandey glusterfs]# time gluster v heal vol info --brick=one >> /dev/null ????????????<<<<<<<< This will scan brick one by one and come out as soon as we find volume is unhealthy. > > real?? ?0m8.316s > user?? ?0m2.241s > sys?? ?0m1.278s > [root at apandey glusterfs]# > > [root at apandey glusterfs]# time gluster v heal vol info >> /dev/null????????????????????????? ????????<<<<<<<< This is current behavior. > > real?? ?0m26.097s > user?? ?0m10.868s > sys?? ?0m6.198s > [root at apandey glusterfs]# > =================================== > > I would like your comments/suggestions on this improvements. > Specially, would like to hear on the new syntax of the command - > > gluster v heal vol info --subvol=[number of the subvol] --brick=[one,all] > > Note that if we do not provide new options, command will behave just like it does right now. > Also, this improvement is valid for AFR and EC. > > --- > Ashish > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Thu Mar 7 05:04:42 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Thu, 07 Mar 2019 07:04:42 +0200 Subject: [Gluster-devel] [Gluster-users] Experiences with FUSE in real world - Presentationat Vault 2019 Message-ID: Thanks a lot. Is there a recording of that ? Best Regards, Strahil NikolovOn Mar 5, 2019 11:13, Raghavendra Gowdappa wrote: > > All, > > Recently me, Manoj and Csaba presented on positives and negatives of implementing File systems in userspace using FUSE [1]. We had based the talk on our experiences with Glusterfs having FUSE as the native interface. The slides can also be found at [1]. > > [1] https://www.usenix.org/conference/vault19/presentation/pillai > > regards, > Raghavendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From hunter86_bg at yahoo.com Thu Mar 7 11:22:38 2019 From: hunter86_bg at yahoo.com (Strahil) Date: Thu, 07 Mar 2019 13:22:38 +0200 Subject: [Gluster-devel] [Gluster-users] Experiences with FUSE in real world - Presentationat Vault 2019 Message-ID: <0qhas4o39y8l9my09wxfritk.1551957758236@email.android.com> Thanks, I have nothing in mind - but I know from experience that live sessions are much more interesting and going in deep. Best Regards, Strahil Nikolov On Mar 7, 2019 08:54, Raghavendra Gowdappa wrote: > > Unfortunately, there is no recording. However, we are willing to discuss our findings if you've specific questions. We can do that in this thread. > > On Thu, Mar 7, 2019 at 10:33 AM Strahil wrote: >> >> Thanks a lot. >> Is there a recording of that ? >> >> Best Regards, >> Strahil Nikolov >> >> On Mar 5, 2019 11:13, Raghavendra Gowdappa wrote: >>> >>> All, >>> >>> Recently me, Manoj and Csaba presented on positives and negatives of implementing File systems in userspace using FUSE [1]. We had based the talk on our experiences with Glusterfs having FUSE as the native interface. The slides can also be found at [1]. >>> >>> [1] https://www.usenix.org/conference/vault19/presentation/pillai >>> >>> regards, >>> Raghavendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From mauro.tridici at cmcc.it Mon Mar 11 09:09:41 2019 From: mauro.tridici at cmcc.it (Mauro Tridici) Date: Mon, 11 Mar 2019 10:09:41 +0100 Subject: [Gluster-devel] [Gluster-users] "rpc_clnt_ping_timer_expired" errors In-Reply-To: References: <96B07283-D8AB-4F06-909D-E00424625528@cmcc.it> <42758A0E-8BE9-497D-BDE3-55D7DC633BC7@cmcc.it> <6A8CF4A4-98EA-48C3-A059-D60D1B2721C7@cmcc.it> <2CF49168-9C1B-4931-8C34-8157262A137A@cmcc.it> <7A151CC9-A0AE-4A45-B450-A4063D216D9E@cmcc.it> <32D53ECE-3F49-4415-A6EE-241B351AC2BA@cmcc.it> <4685A75B-5978-4338-9C9F-4A02FB40B9BC@cmcc.it> <4D2E6B04-C2E8-4FD5-B72D-E7C05931C6F9@cmcc.it> <4E332A56-B318-4BC1-9F44-84AB4392A5DE@cmcc.it> <832FD362-3B14-40D8-8530-604419300476@cmcc.it> <8D926643-1093-48ED-823F-D8F117F702CF@cmcc.it> <9D0BE438-DF11-4D0A-AB85-B44357032F29@cmcc.it> <5F0AC378-8170-4342-8473-9C17159CAC1D@cmcc.it> <7A50E86D-9E27-4EA7-883B-45E9F973991A@cmcc.it> <58B5DB7F-DCB4-4FBF-B1F7-681315B1613A@cmcc.it> <6327B44F-4E7E-46BB-A74C-70F4457DD1EB@cmcc.it> <0167DF4A-8329-4A1A-B439-857DFA6F78BB@cmcc.it> <763F334E-35B8-4729-B8E1-D30866754EEE@cmcc.it> <91DFC9AC-4805-41EB-AC6F-5722E018DD6E@cmcc.it> <8A9752B8-B231-4570-8FF4-8D3D781E7D42@cmcc.it> <47A24804-F975-4EE6-9FA5-67FCDA18D707@cmcc.it> <637FF5D2-D1F4-4686-9D48-646A96F67B96@cmcc.it> <4A87495F-3755-48F7-8507-085869069C64@cmcc.it> <3854BBF6-5B98-4AB3-A67E-E7DE59E69A63@cmcc.it> <313DA021-9173-4899-96B0-831B10B00B61@cmcc.it> <17996AFD-DFC8-40F3-9D09-DB6DDAD5B7D6@cmcc.it> <7074B5D8-955A-4802-A9F3-606C99734417@cmcc.it> <83B84BF9-8334-4230-B6F8-0BC4BFBEFF15@cmcc.it> <133B9AE4-9792-4F72-AD91-D36E7B9EC711@cmcc.it> <6611C4B0-57FD-4390-88B5-BD373555D4C5@cmcc.it> Message-ID: Dear All, do you have any suggestions about the right way to "debug" this issue? In attachment, the updated logs of ?s06" gluster server. I noticed a lot of intermittent warning and error messages. Thank you in advance, Mauro > On 4 Mar 2019, at 18:45, Raghavendra Gowdappa wrote: > > > +Gluster Devel , +Gluster-users > > I would like to point out another issue. Even if what I suggested prevents disconnects, part of the solution would be only symptomatic treatment and doesn't address the root cause of the problem. In most of the ping-timer-expiry issues, the root cause is the increased load on bricks and the inability of bricks to be responsive under high load. So, the actual solution would be doing any or both of the following: > * identify the source of increased load and if possible throttle it. Internal heal processes like self-heal, rebalance, quota heal are known to pump traffic into bricks without much throttling (io-threads _might_ do some throttling, but my understanding is its not sufficient). > * identify the reason for bricks to become unresponsive during load. This may be fixable issues like not enough event-threads to read from network or difficult to fix issues like fsync on backend fs freezing the process or semi fixable issues (in code) like lock contention. > > So any genuine effort to fix ping-timer-issues (to be honest most of the times they are not issues related to rpc/network) would involve performance characterization of various subsystems on bricks and clients. Various subsystems can include (but not necessarily limited to), underlying OS/filesystem, glusterfs processes, CPU consumption etc > > regards, > Raghavendra > > On Mon, Mar 4, 2019 at 9:31 PM Mauro Tridici > wrote: > Thank you, let?s try! > I will inform you about the effects of the change. > > Regards, > Mauro > >> On 4 Mar 2019, at 16:55, Raghavendra Gowdappa > wrote: >> >> >> >> On Mon, Mar 4, 2019 at 8:54 PM Mauro Tridici > wrote: >> Hi Raghavendra, >> >> thank you for your reply. >> Yes, you are right. It is a problem that seems to happen randomly. >> At this moment, server.event-threads value is 4. I will try to increase this value to 8. Do you think that it could be a valid value ? >> >> Yes. We can try with that. You should see at least frequency of ping-timer related disconnects reduce with this value (even if it doesn't eliminate the problem completely). >> >> >> Regards, >> Mauro >> >> >>> On 4 Mar 2019, at 15:36, Raghavendra Gowdappa > wrote: >>> >>> >>> >>> On Mon, Mar 4, 2019 at 8:01 PM Nithya Balachandran > wrote: >>> Hi Mauro, >>> >>> It looks like some problem on s06. Are all your other nodes ok? Can you send us the gluster logs from this node? >>> >>> @Raghavendra G , do you have any idea as to how this can be debugged? Maybe running top ? Or debug brick logs? >>> >>> If we can reproduce the problem, collecting tcpdump on both ends of connection will help. But, one common problem is these bugs are inconsistently reproducible and hence we may not be able to capture tcpdump at correct intervals. Other than that, we can try to collect some evidence that poller threads were busy (waiting on locks). But, not sure what debug data provides that information. >>> >>> From what I know, its difficult to collect evidence for this issue and we could only reason about it. >>> >>> We can try a workaround though - try increasing server.event-threads and see whether ping-timer expiry issues go away with an optimal value. If that's the case, it kind of provides proof for our hypothesis. >>> >>> >>> >>> Regards, >>> Nithya >>> >>> On Mon, 4 Mar 2019 at 15:25, Mauro Tridici > wrote: >>> Hi All, >>> >>> some minutes ago I received this message from NAGIOS server >>> >>> ***** Nagios ***** >>> >>> Notification Type: PROBLEM >>> >>> Service: Brick - /gluster/mnt2/brick >>> Host: s06 >>> Address: s06-stg >>> State: CRITICAL >>> >>> Date/Time: Mon Mar 4 10:25:33 CET 2019 >>> >>> Additional Info: >>> CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. >>> >>> I checked the network, RAM and CPUs usage on s06 node and everything seems to be ok. >>> No bricks are in error state. In /var/log/messages, I detected again a crash of ?check_vol_utili? that I think it is a module used by NRPE executable (that is the NAGIOS client). >>> >>> Mar 4 10:15:29 s06 kernel: traps: check_vol_utili[161224] general protection ip:7facffa0a66d sp:7ffe9f4e6fc0 error:0 in libglusterfs.so.0.0.1[7facff9b7000+f7000] >>> Mar 4 10:15:29 s06 abrt-hook-ccpp: Process 161224 (python2.7) of user 0 killed by SIGSEGV - dumping core >>> Mar 4 10:15:29 s06 abrt-server: Generating core_backtrace >>> Mar 4 10:15:29 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>> Mar 4 10:16:01 s06 systemd: Created slice User Slice of root. >>> Mar 4 10:16:01 s06 systemd: Starting User Slice of root. >>> Mar 4 10:16:01 s06 systemd: Started Session 201010 of user root. >>> Mar 4 10:16:01 s06 systemd: Starting Session 201010 of user root. >>> Mar 4 10:16:01 s06 systemd: Removed slice User Slice of root. >>> Mar 4 10:16:01 s06 systemd: Stopping User Slice of root. >>> Mar 4 10:16:24 s06 abrt-server: Duplicate: UUID >>> Mar 4 10:16:24 s06 abrt-server: DUP_OF_DIR: /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>> Mar 4 10:16:24 s06 abrt-server: Deleting problem directory ccpp-2019-03-04-10:15:29-161224 (dup of ccpp-2018-09-25-12:27:42-13041) >>> Mar 4 10:16:24 s06 abrt-server: Generating core_backtrace >>> Mar 4 10:16:24 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>> Mar 4 10:16:24 s06 abrt-server: Cannot notify '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event 'report_uReport' exited with 1 >>> Mar 4 10:16:24 s06 abrt-hook-ccpp: Process 161391 (python2.7) of user 0 killed by SIGABRT - dumping core >>> Mar 4 10:16:25 s06 abrt-server: Generating core_backtrace >>> Mar 4 10:16:25 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>> Mar 4 10:17:01 s06 systemd: Created slice User Slice of root. >>> >>> Also, I noticed the following errors that I think are very critical: >>> >>> Mar 4 10:21:12 s06 glustershd[20355]: [2019-03-04 09:21:12.954798] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server 192.168.0.55:49158 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:22:01 s06 systemd: Created slice User Slice of root. >>> Mar 4 10:22:01 s06 systemd: Starting User Slice of root. >>> Mar 4 10:22:01 s06 systemd: Started Session 201017 of user root. >>> Mar 4 10:22:01 s06 systemd: Starting Session 201017 of user root. >>> Mar 4 10:22:01 s06 systemd: Removed slice User Slice of root. >>> Mar 4 10:22:01 s06 systemd: Stopping User Slice of root. >>> Mar 4 10:22:03 s06 glustershd[20355]: [2019-03-04 09:22:03.964120] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server 192.168.0.54:49165 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:23:01 s06 systemd: Created slice User Slice of root. >>> Mar 4 10:23:01 s06 systemd: Starting User Slice of root. >>> Mar 4 10:23:01 s06 systemd: Started Session 201018 of user root. >>> Mar 4 10:23:01 s06 systemd: Starting Session 201018 of user root. >>> Mar 4 10:23:02 s06 systemd: Removed slice User Slice of root. >>> Mar 4 10:23:02 s06 systemd: Stopping User Slice of root. >>> Mar 4 10:24:01 s06 systemd: Created slice User Slice of root. >>> Mar 4 10:24:01 s06 systemd: Starting User Slice of root. >>> Mar 4 10:24:01 s06 systemd: Started Session 201019 of user root. >>> Mar 4 10:24:01 s06 systemd: Starting Session 201019 of user root. >>> Mar 4 10:24:01 s06 systemd: Removed slice User Slice of root. >>> Mar 4 10:24:01 s06 systemd: Stopping User Slice of root. >>> Mar 4 10:24:03 s06 glustershd[20355]: [2019-03-04 09:24:03.982502] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-16: server 192.168.0.52:49158 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746109] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-3: server 192.168.0.51:49153 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746215] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server 192.168.0.52:49156 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746260] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-21: server 192.168.0.51:49159 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746296] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server 192.168.0.52:49161 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746413] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server 192.168.0.54:49165 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:24:07 s06 glustershd[20355]: [2019-03-04 09:24:07.982952] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-45: server 192.168.0.54:49155 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:24:18 s06 glustershd[20355]: [2019-03-04 09:24:18.990929] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server 192.168.0.52:49161 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:24:31 s06 glustershd[20355]: [2019-03-04 09:24:31.995781] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-20: server 192.168.0.53:49159 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:25:01 s06 systemd: Created slice User Slice of root. >>> Mar 4 10:25:01 s06 systemd: Starting User Slice of root. >>> Mar 4 10:25:01 s06 systemd: Started Session 201020 of user root. >>> Mar 4 10:25:01 s06 systemd: Starting Session 201020 of user root. >>> Mar 4 10:25:01 s06 systemd: Removed slice User Slice of root. >>> Mar 4 10:25:01 s06 systemd: Stopping User Slice of root. >>> Mar 4 10:25:57 s06 systemd: Created slice User Slice of root. >>> Mar 4 10:25:57 s06 systemd: Starting User Slice of root. >>> Mar 4 10:25:57 s06 systemd-logind: New session 201021 of user root. >>> Mar 4 10:25:57 s06 systemd: Started Session 201021 of user root. >>> Mar 4 10:25:57 s06 systemd: Starting Session 201021 of user root. >>> Mar 4 10:26:01 s06 systemd: Started Session 201022 of user root. >>> Mar 4 10:26:01 s06 systemd: Starting Session 201022 of user root. >>> Mar 4 10:26:21 s06 nrpe[162388]: Error: Could not complete SSL handshake with 192.168.1.56 : 5 >>> Mar 4 10:27:01 s06 systemd: Started Session 201023 of user root. >>> Mar 4 10:27:01 s06 systemd: Starting Session 201023 of user root. >>> Mar 4 10:28:01 s06 systemd: Started Session 201024 of user root. >>> Mar 4 10:28:01 s06 systemd: Starting Session 201024 of user root. >>> Mar 4 10:29:01 s06 systemd: Started Session 201025 of user root. >>> Mar 4 10:29:01 s06 systemd: Starting Session 201025 of user root. >>> >>> But, unfortunately, I don?t understand why it is happening. >>> Now, NAGIOS server shows that s06 status is ok: >>> >>> ***** Nagios ***** >>> >>> Notification Type: RECOVERY >>> >>> Service: Brick - /gluster/mnt2/brick >>> Host: s06 >>> Address: s06-stg >>> State: OK >>> >>> Date/Time: Mon Mar 4 10:35:23 CET 2019 >>> >>> Additional Info: >>> OK: Brick /gluster/mnt2/brick is up >>> >>> Nothing is changed from RAM, CPUs, and NETWORK point of view. >>> /var/log/message file has been updated: >>> >>> Mar 4 10:32:01 s06 systemd: Starting Session 201029 of user root. >>> Mar 4 10:32:30 s06 glustershd[20355]: [2019-03-04 09:32:30.069082] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server 192.168.0.52:49156 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:32:55 s06 glustershd[20355]: [2019-03-04 09:32:55.074689] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-66: server 192.168.0.54:49167 has not responded in the last 42 seconds, disconnecting. >>> Mar 4 10:33:01 s06 systemd: Started Session 201030 of user root. >>> Mar 4 10:33:01 s06 systemd: Starting Session 201030 of user root. >>> Mar 4 10:34:01 s06 systemd: Started Session 201031 of user root. >>> Mar 4 10:34:01 s06 systemd: Starting Session 201031 of user root. >>> Mar 4 10:35:01 s06 nrpe[162562]: Could not read request from client 192.168.1.56, bailing out... >>> Mar 4 10:35:01 s06 nrpe[162562]: INFO: SSL Socket Shutdown. >>> Mar 4 10:35:01 s06 systemd: Started Session 201032 of user root. >>> Mar 4 10:35:01 s06 systemd: Starting Session 201032 of user root. >>> >>> Could you please help me to understand what it?s happening ? >>> Thank you in advance. >>> >>> Rergards, >>> Mauro >>> >>> >>>> On 1 Mar 2019, at 12:17, Mauro Tridici > wrote: >>>> >>>> >>>> Thank you, Milind. >>>> I executed the instructions you suggested: >>>> >>>> - grep ?blocked for? /var/log/messages on s06 returns no output (no ?blocked? word is detected in messages file); >>>> - in /var/log/messages file I can see this kind of error repeated for a lot of times: >>>> >>>> Mar 1 08:43:01 s06 systemd: Starting Session 196071 of user root. >>>> Mar 1 08:43:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 1 08:43:01 s06 systemd: Stopping User Slice of root. >>>> Mar 1 08:43:02 s06 kernel: traps: check_vol_utili[57091] general protection ip:7f88e76ee66d sp:7ffe5a5bcc30 error:0 in libglusterfs.so.0.0.1[7f88e769b000+f7000] >>>> Mar 1 08:43:02 s06 abrt-hook-ccpp: Process 57091 (python2.7) of user 0 killed by SIGSEGV - dumping core >>>> Mar 1 08:43:02 s06 abrt-server: Generating core_backtrace >>>> Mar 1 08:43:02 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>> Mar 1 08:43:58 s06 abrt-server: Duplicate: UUID >>>> Mar 1 08:43:58 s06 abrt-server: DUP_OF_DIR: /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>> Mar 1 08:43:58 s06 abrt-server: Deleting problem directory ccpp-2019-03-01-08:43:02-57091 (dup of ccpp-2018-09-25-12:27:42-13041) >>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Activating service name='org.freedesktop.problems' (using servicehelper) >>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Successfully activated service 'org.freedesktop.problems' >>>> Mar 1 08:43:58 s06 abrt-server: Generating core_backtrace >>>> Mar 1 08:43:58 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>> Mar 1 08:43:58 s06 abrt-server: Cannot notify '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event 'report_uReport' exited with 1 >>>> Mar 1 08:44:01 s06 systemd: Created slice User Slice of root. >>>> Mar 1 08:44:01 s06 systemd: Starting User Slice of root. >>>> Mar 1 08:44:01 s06 systemd: Started Session 196072 of user root. >>>> Mar 1 08:44:01 s06 systemd: Starting Session 196072 of user root. >>>> Mar 1 08:44:01 s06 systemd: Removed slice User Slice of root. >>>> >>>> - in /var/log/messages file I can see also 4 errors related to other cluster servers: >>>> >>>> Mar 1 11:05:01 s06 systemd: Starting User Slice of root. >>>> Mar 1 11:05:01 s06 systemd: Started Session 196230 of user root. >>>> Mar 1 11:05:01 s06 systemd: Starting Session 196230 of user root. >>>> Mar 1 11:05:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 1 11:05:01 s06 systemd: Stopping User Slice of root. >>>> Mar 1 11:05:59 s06 glustershd[70117]: [2019-03-01 10:05:59.347094] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-33: server 192.168.0.51:49163 has not responded in the last 42 seconds, disconnecting. >>>> Mar 1 11:06:01 s06 systemd: Created slice User Slice of root. >>>> Mar 1 11:06:01 s06 systemd: Starting User Slice of root. >>>> Mar 1 11:06:01 s06 systemd: Started Session 196231 of user root. >>>> Mar 1 11:06:01 s06 systemd: Starting Session 196231 of user root. >>>> Mar 1 11:06:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 1 11:06:01 s06 systemd: Stopping User Slice of root. >>>> Mar 1 11:06:12 s06 glustershd[70117]: [2019-03-01 10:06:12.351319] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-1: server 192.168.0.52:49153 has not responded in the last 42 seconds, disconnecting. >>>> Mar 1 11:06:38 s06 glustershd[70117]: [2019-03-01 10:06:38.356920] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-7: server 192.168.0.52:49155 has not responded in the last 42 seconds, disconnecting. >>>> Mar 1 11:07:01 s06 systemd: Created slice User Slice of root. >>>> Mar 1 11:07:01 s06 systemd: Starting User Slice of root. >>>> Mar 1 11:07:01 s06 systemd: Started Session 196232 of user root. >>>> Mar 1 11:07:01 s06 systemd: Starting Session 196232 of user root. >>>> Mar 1 11:07:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 1 11:07:01 s06 systemd: Stopping User Slice of root. >>>> Mar 1 11:07:36 s06 glustershd[70117]: [2019-03-01 10:07:36.366259] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-0: server 192.168.0.51:49152 has not responded in the last 42 seconds, disconnecting. >>>> Mar 1 11:08:01 s06 systemd: Created slice User Slice of root. >>>> >>>> No ?blocked? word is in /var/log/messages files on other cluster servers. >>>> In attachment, the /var/log/messages file from s06 server. >>>> >>>> Thank you in advance, >>>> Mauro >>>> >>>> >>>> >>>> >>>>> On 1 Mar 2019, at 11:47, Milind Changire > wrote: >>>>> >>>>> The traces of very high disk activity on the servers are often found in /var/log/messages >>>>> You might want to grep for "blocked for" in /var/log/messages on s06 and correlate the timestamps to confirm the unresponsiveness as reported in gluster client logs. >>>>> In cases of high disk activity, although the operating system continues to respond to ICMP pings, the processes writing to disks often get blocked to a large flush to the disk which could span beyond 42 seconds and hence result in ping-timer-expiry logs. >>>>> >>>>> As a side note: >>>>> If you indeed find gluster processes being blocked in /var/log/messages, you might want to tweak sysctl tunables called vm.dirty_background_ratio or vm.dirty_background_bytes to a smaller value than the existing. Please read up more on those tunables before touching the settings. >>>>> >>>>> >>>>> On Fri, Mar 1, 2019 at 4:06 PM Mauro Tridici > wrote: >>>>> >>>>> Hi all, >>>>> >>>>> in attachment the client log captured after changing network.ping-timeout option. >>>>> I noticed this error involving server 192.168.0.56 (s06) >>>>> >>>>> [2019-03-01 09:23:36.077287] I [rpc-clnt.c:1962:rpc_clnt_reconfig] 0-tier2-client-71: changing ping timeout to 42 (from 0) >>>>> [2019-03-01 09:23:36.078213] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>> [2019-03-01 09:23:36.078432] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>> [2019-03-01 09:23:36.092357] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>> [2019-03-01 09:23:36.094146] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>> [2019-03-01 10:06:24.708082] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-50: server 192.168.0.56:49156 has not responded in the last 42 seconds, disconnecting. >>>>> >>>>> I don?t know why it happens, s06 server seems to be reachable. >>>>> >>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>> Trying 192.168.0.56... >>>>> Connected to 192.168.0.56. >>>>> Escape character is '^]'. >>>>> ^CConnection closed by foreign host. >>>>> [athena_login2][/users/home/sysm02/]> ping 192.168.0.56 >>>>> PING 192.168.0.56 (192.168.0.56) 56(84) bytes of data. >>>>> 64 bytes from 192.168.0.56 : icmp_seq=1 ttl=64 time=0.116 ms >>>>> 64 bytes from 192.168.0.56 : icmp_seq=2 ttl=64 time=0.101 ms >>>>> >>>>> --- 192.168.0.56 ping statistics --- >>>>> 2 packets transmitted, 2 received, 0% packet loss, time 1528ms >>>>> rtt min/avg/max/mdev = 0.101/0.108/0.116/0.012 ms >>>>> >>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>> Trying 192.168.0.56... >>>>> Connected to 192.168.0.56. >>>>> Escape character is '^]'. >>>>> >>>>> Thank you for your help, >>>>> Mauro >>>>> >>>>> >>>>> >>>>>> On 1 Mar 2019, at 10:29, Mauro Tridici > wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> thank you for the explanation. >>>>>> I just changed network.ping-timeout option to default value (network.ping-timeout=42). >>>>>> >>>>>> I will check the logs to see if the errors will appear again. >>>>>> >>>>>> Regards, >>>>>> Mauro >>>>>> >>>>>>> On 1 Mar 2019, at 04:43, Milind Changire > wrote: >>>>>>> >>>>>>> network.ping-timeout should not be set to zero for non-glusterd clients. >>>>>>> glusterd is a special case for which ping-timeout is set to zero via /etc/glusterfs/glusterd.vol >>>>>>> >>>>>>> Setting network.ping-timeout to zero disables arming of the ping timer for connections. This disables testing the connection for responsiveness and hence avoids proactive fail-over. >>>>>>> >>>>>>> Please reset network.ping-timeout to a non-zero positive value, eg. 42 >>>>>>> >>>>>>> >>>>>>> On Thu, Feb 28, 2019 at 5:07 PM Nithya Balachandran > wrote: >>>>>>> Adding Raghavendra and Milind to comment on this. >>>>>>> >>>>>>> What is the effect of setting network.ping-timeout to 0 and should it be set back to 42? >>>>>>> Regards, >>>>>>> Nithya >>>>>>> >>>>>>> On Thu, 28 Feb 2019 at 16:01, Mauro Tridici > wrote: >>>>>>> Hi Nithya, >>>>>>> >>>>>>> sorry for the late. >>>>>>> network.ping-timeout has been set to 0 in order to try to solve some timeout problems, but it didn?t help. >>>>>>> I can set it to the default value. >>>>>>> >>>>>>> Can I proceed with the change? >>>>>>> >>>>>>> Thank you, >>>>>>> Mauro >>>>>>> >>>>>>> >>>>>>>> On 28 Feb 2019, at 04:41, Nithya Balachandran > wrote: >>>>>>>> >>>>>>>> Hi Mauro, >>>>>>>> >>>>>>>> Is network.ping-timeout still set to 0. The default value is 42. Is there a particular reason why this was changed? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Nithya >>>>>>>> >>>>>>>> >>>>>>>> On Wed, 27 Feb 2019 at 21:32, Mauro Tridici > wrote: >>>>>>>> >>>>>>>> Hi Xavi, >>>>>>>> >>>>>>>> thank you for the detailed explanation and suggestions. >>>>>>>> Yes, transport.listen-backlog option is still set to 1024. >>>>>>>> >>>>>>>> I will check the network and connectivity status using ?ping? and ?telnet? as soon as the errors will come back again. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Mauro >>>>>>>> >>>>>>>>> Il giorno 27 feb 2019, alle ore 16:42, Xavi Hernandez > ha scritto: >>>>>>>>> >>>>>>>>> Hi Mauro, >>>>>>>>> >>>>>>>>> those errors say that the mount point is not connected to some of the bricks while executing operations. I see references to 3rd and 6th bricks of several disperse sets, which seem to map to server s06. For some reason, gluster is having troubles connecting from the client machine to that particular server. At the end of the log I see that after long time a reconnect is done to both of them. However little after, other bricks from the s05 get disconnected and a reconnect times out. >>>>>>>>> >>>>>>>>> That's really odd. It seems like if server/communication is cut to s06 for some time, then restored, and then the same happens to the next server. >>>>>>>>> >>>>>>>>> If the servers are really online and it's only a communication issue, it explains why server memory and network has increased: if the problem only exists between the client and servers, any write made by the client will automatically mark the file as damaged, since some of the servers have not been updated. Since self-heal runs from the server nodes, they will probably be correctly connected to all bricks, which allows them to heal the just damaged file, which increases memory and network usage. >>>>>>>>> >>>>>>>>> I guess you still have transport.listen-backlog set to 1024, right ? >>>>>>>>> >>>>>>>>> Just to try to identify if the problem really comes from network, can you check if you lose some pings from the client to all of the servers while you are seeing those errors in the log file ? >>>>>>>>> >>>>>>>>> You can also check if during those errors, you can telnet to the port of the brick from the client. >>>>>>>>> >>>>>>>>> Xavi >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Feb 26, 2019 at 10:17 AM Mauro Tridici > wrote: >>>>>>>>> Hi Nithya, >>>>>>>>> >>>>>>>>> ?df -h? operation is not still slow, but no users are using the volume, RAM and NETWORK usage is ok on the client node. >>>>>>>>> >>>>>>>>> I was worried about this kind of warnings/errors: >>>>>>>>> >>>>>>>>> [2019-02-25 10:59:00.664323] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-6: Executing operation with some subvolumes unavailable (20) >>>>>>>>> >>>>>>>>> [2019-02-26 03:11:35.212603] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-26 03:10:56.549903 (xid=0x106f1c5) >>>>>>>>> >>>>>>>>> [2019-02-26 03:13:03.313831] E [socket.c:2376:socket_connect_finish] 0-tier2-client-50: connection to 192.168.0.56:49156 failed (Timeout della connessione); disconnecting socket >>>>>>>>> >>>>>>>>> It seems that some subvolumes are not available and 192.168.0.56 server (s06) is not reachable. >>>>>>>>> But gluster servers are up&running and bricks are ok. >>>>>>>>> >>>>>>>>> In attachment the updated tier2.log file. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> Regards, >>>>>>>>> Mauro >>>>>>>>> >>>>>>>>>> Il giorno 26 feb 2019, alle ore 04:03, Nithya Balachandran > ha scritto: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I see a lot of EC messages in the log but they don't seem very serious. Xavi, can you take a look? >>>>>>>>>> >>>>>>>>>> The only errors I see are: >>>>>>>>>> [2019-02-25 10:58:45.519871] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-25 10:57:47.429969 (xid=0xd26fe7) >>>>>>>>>> [2019-02-25 10:58:51.461493] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-41: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-25 10:57:47.499174 (xid=0xf47d6a) >>>>>>>>>> [2019-02-25 11:07:57.152874] E [socket.c:2376:socket_connect_finish] 0-tier2-client-70: connection to 192.168.0.55:49163 failed (Timeout della connessione); disconnecting socket >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Is the df -h operation still slow? If yes, can you take a tcpdump of the client while running df -h and send that across? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Nithya >>>>>>>>>> >>>>>>>>>> On Mon, 25 Feb 2019 at 17:27, Mauro Tridici > wrote: >>>>>>>>>> >>>>>>>>>> Sorry, some minutes after my last mail message, I noticed that ?df -h? command hanged for a while before returns the prompt. >>>>>>>>>> Yesterday, everything was ok in the gluster client log, but, today, I see a lot of errors (please, take a look to the attached file). >>>>>>>>>> >>>>>>>>>> On the client node, I detected an important RAM e NETWORK usage. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Do you think that the errors have been caused by the client resources usage? >>>>>>>>>> >>>>>>>>>> Thank you in advance, >>>>>>>>>> Mauro >>>>>>>>>> >>> > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: s06-20190311.logs.tar.gz Type: application/x-gzip Size: 2338627 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: From mauro.tridici at cmcc.it Wed Mar 13 10:25:40 2019 From: mauro.tridici at cmcc.it (Mauro Tridici) Date: Wed, 13 Mar 2019 11:25:40 +0100 Subject: [Gluster-devel] [Gluster-users] "rpc_clnt_ping_timer_expired" errors In-Reply-To: References: <96B07283-D8AB-4F06-909D-E00424625528@cmcc.it> <42758A0E-8BE9-497D-BDE3-55D7DC633BC7@cmcc.it> <6A8CF4A4-98EA-48C3-A059-D60D1B2721C7@cmcc.it> <2CF49168-9C1B-4931-8C34-8157262A137A@cmcc.it> <7A151CC9-A0AE-4A45-B450-A4063D216D9E@cmcc.it> <32D53ECE-3F49-4415-A6EE-241B351AC2BA@cmcc.it> <4685A75B-5978-4338-9C9F-4A02FB40B9BC@cmcc.it> <4D2E6B04-C2E8-4FD5-B72D-E7C05931C6F9@cmcc.it> <4E332A56-B318-4BC1-9F44-84AB4392A5DE@cmcc.it> <832FD362-3B14-40D8-8530-604419300476@cmcc.it> <8D926643-1093-48ED-823F-D8F117F702CF@cmcc.it> <9D0BE438-DF11-4D0A-AB85-B44357032F29@cmcc.it> <5F0AC378-8170-4342-8473-9C17159CAC1D@cmcc.it> <7A50E86D-9E27-4EA7-883B-45E9F973991A@cmcc.it> <58B5DB7F-DCB4-4FBF-B1F7-681315B1613A@cmcc.it> <6327B44F-4E7E-46BB-A74C-70F4457DD1EB@cmcc.it> <0167DF4A-8329-4A1A-B439-857DFA6F78BB@cmcc.it> <763F334E-35B8-4729-B8E1-D30866754EEE@cmcc.it> <91DFC9AC-4805-41EB-AC6F-5722E018DD6E@cmcc.it> <8A9752B8-B231-4570-8FF4-8D3D781E7D42@cmcc.it> <47A24804-F975-4EE6-9FA5-67FCDA18D707@cmcc.it> <637FF5D2-D1F4-4686-9D48-646A96F67B96@cmcc.it> <4A87495F-3755-48F7-8507-085869069C64@cmcc.it> <3854BBF6-5B98-4AB3-A67E-E7DE59E69A63@cmcc.it> <313DA021-9173-4899-96B0-831B10B00B61@cmcc.it> <17996AFD-DFC8-40F3-9D09-DB6DDAD5B7D6@cmcc.it> <7074B5D8-955A-4802-A9F3-606C99734417@cmcc.it> <83B84BF9-8334-4230-B6F8-0BC4BFBEFF15@cmcc.it> <133B9AE4-9792-4F72-AD91-D36E7B9EC711@cmcc.it> <6611C4B0-57FD-4390-88B5-BD373555D4C5@cmcc.it> Message-ID: Hi Raghavendra, Yes, server.event-thread has been changed from 4 to 8. During last days, I noticed that the error events are still here although they have been considerably reduced. So, I used grep command against the log files in order to provide you a global vision about the warning, error and critical events appeared today at 06:xx (may be useful I hope). I collected the info from s06 gluster server, but the behaviour is the the almost the same on the other gluster servers. ERRORS: CWD: /var/log/glusterfs COMMAND: grep " E " *.log |grep "2019-03-13 06:" (I can see a lot of this kind of message in the same period but I'm notifying you only one record for each type of error) glusterd.log:[2019-03-13 06:12:35.982863] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of /var/run/gluster/tier2_quota_list/ glustershd.log:[2019-03-13 06:14:28.666562] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a71ddcebb] (--> /lib64/libgfr pc.so.0(saved_frames_unwind+0x1de)[0x7f4a71ba1d9e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f4a71ba1ebe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup +0x90)[0x7f4a71ba3640] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7f4a71ba4130] ))))) 0-tier2-client-55: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2019-03-13 06:14:14.858441 (xid=0x17fddb50) glustershd.log:[2019-03-13 06:17:48.883825] E [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to 192.168.0.55:49158 failed (Connection timed out); disco nnecting socket glustershd.log:[2019-03-13 06:19:58.931798] E [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to 192.168.0.55:49158 failed (Connection timed out); disco nnecting socket glustershd.log:[2019-03-13 06:22:08.979829] E [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to 192.168.0.55:49158 failed (Connection timed out); disco nnecting socket glustershd.log:[2019-03-13 06:22:36.226847] E [MSGID: 114031] [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote operation failed [Transport endpoint is not connected] glustershd.log:[2019-03-13 06:22:36.306669] E [MSGID: 114031] [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote operation failed [Transport endpoint is not connected] glustershd.log:[2019-03-13 06:22:36.385257] E [MSGID: 114031] [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote operation failed [Transport endpoint is not connected] WARNINGS: CWD: /var/log/glusterfs COMMAND: grep " W " *.log |grep "2019-03-13 06:" (I can see a lot of this kind of message in the same period but I'm notifying you only one record for each type of warnings) glustershd.log:[2019-03-13 06:14:28.666772] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-tier2-client-55: remote operation failed. Path: (b6b35d0f-f34d-4c25-bbe8-74bde0248d7e). Key: (null) [Transport endpoint is not connected] glustershd.log:[2019-03-13 06:14:31.421576] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (2) glustershd.log:[2019-03-13 06:15:31.547417] W [MSGID: 122032] [ec-heald.c:266:ec_shd_index_sweep] 0-tier2-disperse-9: unable to get index-dir on tier2-client-55 [Operation now in progress] quota-mount-tier2.log:[2019-03-13 06:12:36.116277] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'trans port.address-family', continuing with correction quota-mount-tier2.log:[2019-03-13 06:12:36.198430] W [MSGID: 101174] [graph.c:363:_log_if_unknown_option] 0-tier2-readdir-ahead: option 'parallel-readdir' is not recognized quota-mount-tier2.log:[2019-03-13 06:12:37.945007] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f340892be25] -->/usr/sbin/glusterfs(gluste rfs_sigwaiter+0xe5) [0x55ef010164b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55ef0101632b] ) 0-: received signum (15), shutting down CRITICALS: CWD: /var/log/glusterfs COMMAND: grep " C " *.log |grep "2019-03-13 06:" no critical errors at 06:xx only one critical error during the day [root at s06 glusterfs]# grep " C " *.log |grep "2019-03-13" glustershd.log:[2019-03-13 02:21:29.126279] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server 192.168.0.55:49158 has not responded in the last 42 seconds, disconnecting. Thank you very much for your help. Regards, Mauro > On 12 Mar 2019, at 05:17, Raghavendra Gowdappa wrote: > > Was the suggestion to increase server.event-thread values tried? If yes, what were the results? > > On Mon, Mar 11, 2019 at 2:40 PM Mauro Tridici > wrote: > Dear All, > > do you have any suggestions about the right way to "debug" this issue? > In attachment, the updated logs of ?s06" gluster server. > > I noticed a lot of intermittent warning and error messages. > > Thank you in advance, > Mauro > > > >> On 4 Mar 2019, at 18:45, Raghavendra Gowdappa > wrote: >> >> >> +Gluster Devel , +Gluster-users >> >> I would like to point out another issue. Even if what I suggested prevents disconnects, part of the solution would be only symptomatic treatment and doesn't address the root cause of the problem. In most of the ping-timer-expiry issues, the root cause is the increased load on bricks and the inability of bricks to be responsive under high load. So, the actual solution would be doing any or both of the following: >> * identify the source of increased load and if possible throttle it. Internal heal processes like self-heal, rebalance, quota heal are known to pump traffic into bricks without much throttling (io-threads _might_ do some throttling, but my understanding is its not sufficient). >> * identify the reason for bricks to become unresponsive during load. This may be fixable issues like not enough event-threads to read from network or difficult to fix issues like fsync on backend fs freezing the process or semi fixable issues (in code) like lock contention. >> >> So any genuine effort to fix ping-timer-issues (to be honest most of the times they are not issues related to rpc/network) would involve performance characterization of various subsystems on bricks and clients. Various subsystems can include (but not necessarily limited to), underlying OS/filesystem, glusterfs processes, CPU consumption etc >> >> regards, >> Raghavendra >> >> On Mon, Mar 4, 2019 at 9:31 PM Mauro Tridici > wrote: >> Thank you, let?s try! >> I will inform you about the effects of the change. >> >> Regards, >> Mauro >> >>> On 4 Mar 2019, at 16:55, Raghavendra Gowdappa > wrote: >>> >>> >>> >>> On Mon, Mar 4, 2019 at 8:54 PM Mauro Tridici > wrote: >>> Hi Raghavendra, >>> >>> thank you for your reply. >>> Yes, you are right. It is a problem that seems to happen randomly. >>> At this moment, server.event-threads value is 4. I will try to increase this value to 8. Do you think that it could be a valid value ? >>> >>> Yes. We can try with that. You should see at least frequency of ping-timer related disconnects reduce with this value (even if it doesn't eliminate the problem completely). >>> >>> >>> Regards, >>> Mauro >>> >>> >>>> On 4 Mar 2019, at 15:36, Raghavendra Gowdappa > wrote: >>>> >>>> >>>> >>>> On Mon, Mar 4, 2019 at 8:01 PM Nithya Balachandran > wrote: >>>> Hi Mauro, >>>> >>>> It looks like some problem on s06. Are all your other nodes ok? Can you send us the gluster logs from this node? >>>> >>>> @Raghavendra G , do you have any idea as to how this can be debugged? Maybe running top ? Or debug brick logs? >>>> >>>> If we can reproduce the problem, collecting tcpdump on both ends of connection will help. But, one common problem is these bugs are inconsistently reproducible and hence we may not be able to capture tcpdump at correct intervals. Other than that, we can try to collect some evidence that poller threads were busy (waiting on locks). But, not sure what debug data provides that information. >>>> >>>> From what I know, its difficult to collect evidence for this issue and we could only reason about it. >>>> >>>> We can try a workaround though - try increasing server.event-threads and see whether ping-timer expiry issues go away with an optimal value. If that's the case, it kind of provides proof for our hypothesis. >>>> >>>> >>>> >>>> Regards, >>>> Nithya >>>> >>>> On Mon, 4 Mar 2019 at 15:25, Mauro Tridici > wrote: >>>> Hi All, >>>> >>>> some minutes ago I received this message from NAGIOS server >>>> >>>> ***** Nagios ***** >>>> >>>> Notification Type: PROBLEM >>>> >>>> Service: Brick - /gluster/mnt2/brick >>>> Host: s06 >>>> Address: s06-stg >>>> State: CRITICAL >>>> >>>> Date/Time: Mon Mar 4 10:25:33 CET 2019 >>>> >>>> Additional Info: >>>> CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. >>>> >>>> I checked the network, RAM and CPUs usage on s06 node and everything seems to be ok. >>>> No bricks are in error state. In /var/log/messages, I detected again a crash of ?check_vol_utili? that I think it is a module used by NRPE executable (that is the NAGIOS client). >>>> >>>> Mar 4 10:15:29 s06 kernel: traps: check_vol_utili[161224] general protection ip:7facffa0a66d sp:7ffe9f4e6fc0 error:0 in libglusterfs.so.0.0.1[7facff9b7000+f7000] >>>> Mar 4 10:15:29 s06 abrt-hook-ccpp: Process 161224 (python2.7) of user 0 killed by SIGSEGV - dumping core >>>> Mar 4 10:15:29 s06 abrt-server: Generating core_backtrace >>>> Mar 4 10:15:29 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>> Mar 4 10:16:01 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:16:01 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:16:01 s06 systemd: Started Session 201010 of user root. >>>> Mar 4 10:16:01 s06 systemd: Starting Session 201010 of user root. >>>> Mar 4 10:16:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 4 10:16:01 s06 systemd: Stopping User Slice of root. >>>> Mar 4 10:16:24 s06 abrt-server: Duplicate: UUID >>>> Mar 4 10:16:24 s06 abrt-server: DUP_OF_DIR: /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>> Mar 4 10:16:24 s06 abrt-server: Deleting problem directory ccpp-2019-03-04-10:15:29-161224 (dup of ccpp-2018-09-25-12:27:42-13041) >>>> Mar 4 10:16:24 s06 abrt-server: Generating core_backtrace >>>> Mar 4 10:16:24 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>> Mar 4 10:16:24 s06 abrt-server: Cannot notify '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event 'report_uReport' exited with 1 >>>> Mar 4 10:16:24 s06 abrt-hook-ccpp: Process 161391 (python2.7) of user 0 killed by SIGABRT - dumping core >>>> Mar 4 10:16:25 s06 abrt-server: Generating core_backtrace >>>> Mar 4 10:16:25 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>> Mar 4 10:17:01 s06 systemd: Created slice User Slice of root. >>>> >>>> Also, I noticed the following errors that I think are very critical: >>>> >>>> Mar 4 10:21:12 s06 glustershd[20355]: [2019-03-04 09:21:12.954798] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server 192.168.0.55:49158 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:22:01 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:22:01 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:22:01 s06 systemd: Started Session 201017 of user root. >>>> Mar 4 10:22:01 s06 systemd: Starting Session 201017 of user root. >>>> Mar 4 10:22:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 4 10:22:01 s06 systemd: Stopping User Slice of root. >>>> Mar 4 10:22:03 s06 glustershd[20355]: [2019-03-04 09:22:03.964120] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server 192.168.0.54:49165 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:23:01 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:23:01 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:23:01 s06 systemd: Started Session 201018 of user root. >>>> Mar 4 10:23:01 s06 systemd: Starting Session 201018 of user root. >>>> Mar 4 10:23:02 s06 systemd: Removed slice User Slice of root. >>>> Mar 4 10:23:02 s06 systemd: Stopping User Slice of root. >>>> Mar 4 10:24:01 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:24:01 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:24:01 s06 systemd: Started Session 201019 of user root. >>>> Mar 4 10:24:01 s06 systemd: Starting Session 201019 of user root. >>>> Mar 4 10:24:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 4 10:24:01 s06 systemd: Stopping User Slice of root. >>>> Mar 4 10:24:03 s06 glustershd[20355]: [2019-03-04 09:24:03.982502] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-16: server 192.168.0.52:49158 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746109] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-3: server 192.168.0.51:49153 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746215] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server 192.168.0.52:49156 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746260] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-21: server 192.168.0.51:49159 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746296] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server 192.168.0.52:49161 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746413] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server 192.168.0.54:49165 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:24:07 s06 glustershd[20355]: [2019-03-04 09:24:07.982952] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-45: server 192.168.0.54:49155 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:24:18 s06 glustershd[20355]: [2019-03-04 09:24:18.990929] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server 192.168.0.52:49161 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:24:31 s06 glustershd[20355]: [2019-03-04 09:24:31.995781] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-20: server 192.168.0.53:49159 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:25:01 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:25:01 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:25:01 s06 systemd: Started Session 201020 of user root. >>>> Mar 4 10:25:01 s06 systemd: Starting Session 201020 of user root. >>>> Mar 4 10:25:01 s06 systemd: Removed slice User Slice of root. >>>> Mar 4 10:25:01 s06 systemd: Stopping User Slice of root. >>>> Mar 4 10:25:57 s06 systemd: Created slice User Slice of root. >>>> Mar 4 10:25:57 s06 systemd: Starting User Slice of root. >>>> Mar 4 10:25:57 s06 systemd-logind: New session 201021 of user root. >>>> Mar 4 10:25:57 s06 systemd: Started Session 201021 of user root. >>>> Mar 4 10:25:57 s06 systemd: Starting Session 201021 of user root. >>>> Mar 4 10:26:01 s06 systemd: Started Session 201022 of user root. >>>> Mar 4 10:26:01 s06 systemd: Starting Session 201022 of user root. >>>> Mar 4 10:26:21 s06 nrpe[162388]: Error: Could not complete SSL handshake with 192.168.1.56 : 5 >>>> Mar 4 10:27:01 s06 systemd: Started Session 201023 of user root. >>>> Mar 4 10:27:01 s06 systemd: Starting Session 201023 of user root. >>>> Mar 4 10:28:01 s06 systemd: Started Session 201024 of user root. >>>> Mar 4 10:28:01 s06 systemd: Starting Session 201024 of user root. >>>> Mar 4 10:29:01 s06 systemd: Started Session 201025 of user root. >>>> Mar 4 10:29:01 s06 systemd: Starting Session 201025 of user root. >>>> >>>> But, unfortunately, I don?t understand why it is happening. >>>> Now, NAGIOS server shows that s06 status is ok: >>>> >>>> ***** Nagios ***** >>>> >>>> Notification Type: RECOVERY >>>> >>>> Service: Brick - /gluster/mnt2/brick >>>> Host: s06 >>>> Address: s06-stg >>>> State: OK >>>> >>>> Date/Time: Mon Mar 4 10:35:23 CET 2019 >>>> >>>> Additional Info: >>>> OK: Brick /gluster/mnt2/brick is up >>>> >>>> Nothing is changed from RAM, CPUs, and NETWORK point of view. >>>> /var/log/message file has been updated: >>>> >>>> Mar 4 10:32:01 s06 systemd: Starting Session 201029 of user root. >>>> Mar 4 10:32:30 s06 glustershd[20355]: [2019-03-04 09:32:30.069082] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server 192.168.0.52:49156 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:32:55 s06 glustershd[20355]: [2019-03-04 09:32:55.074689] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-66: server 192.168.0.54:49167 has not responded in the last 42 seconds, disconnecting. >>>> Mar 4 10:33:01 s06 systemd: Started Session 201030 of user root. >>>> Mar 4 10:33:01 s06 systemd: Starting Session 201030 of user root. >>>> Mar 4 10:34:01 s06 systemd: Started Session 201031 of user root. >>>> Mar 4 10:34:01 s06 systemd: Starting Session 201031 of user root. >>>> Mar 4 10:35:01 s06 nrpe[162562]: Could not read request from client 192.168.1.56, bailing out... >>>> Mar 4 10:35:01 s06 nrpe[162562]: INFO: SSL Socket Shutdown. >>>> Mar 4 10:35:01 s06 systemd: Started Session 201032 of user root. >>>> Mar 4 10:35:01 s06 systemd: Starting Session 201032 of user root. >>>> >>>> Could you please help me to understand what it?s happening ? >>>> Thank you in advance. >>>> >>>> Rergards, >>>> Mauro >>>> >>>> >>>>> On 1 Mar 2019, at 12:17, Mauro Tridici > wrote: >>>>> >>>>> >>>>> Thank you, Milind. >>>>> I executed the instructions you suggested: >>>>> >>>>> - grep ?blocked for? /var/log/messages on s06 returns no output (no ?blocked? word is detected in messages file); >>>>> - in /var/log/messages file I can see this kind of error repeated for a lot of times: >>>>> >>>>> Mar 1 08:43:01 s06 systemd: Starting Session 196071 of user root. >>>>> Mar 1 08:43:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 1 08:43:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 1 08:43:02 s06 kernel: traps: check_vol_utili[57091] general protection ip:7f88e76ee66d sp:7ffe5a5bcc30 error:0 in libglusterfs.so.0.0.1[7f88e769b000+f7000] >>>>> Mar 1 08:43:02 s06 abrt-hook-ccpp: Process 57091 (python2.7) of user 0 killed by SIGSEGV - dumping core >>>>> Mar 1 08:43:02 s06 abrt-server: Generating core_backtrace >>>>> Mar 1 08:43:02 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>>> Mar 1 08:43:58 s06 abrt-server: Duplicate: UUID >>>>> Mar 1 08:43:58 s06 abrt-server: DUP_OF_DIR: /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>> Mar 1 08:43:58 s06 abrt-server: Deleting problem directory ccpp-2019-03-01-08:43:02-57091 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Activating service name='org.freedesktop.problems' (using servicehelper) >>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Successfully activated service 'org.freedesktop.problems' >>>>> Mar 1 08:43:58 s06 abrt-server: Generating core_backtrace >>>>> Mar 1 08:43:58 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>>> Mar 1 08:43:58 s06 abrt-server: Cannot notify '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event 'report_uReport' exited with 1 >>>>> Mar 1 08:44:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 1 08:44:01 s06 systemd: Starting User Slice of root. >>>>> Mar 1 08:44:01 s06 systemd: Started Session 196072 of user root. >>>>> Mar 1 08:44:01 s06 systemd: Starting Session 196072 of user root. >>>>> Mar 1 08:44:01 s06 systemd: Removed slice User Slice of root. >>>>> >>>>> - in /var/log/messages file I can see also 4 errors related to other cluster servers: >>>>> >>>>> Mar 1 11:05:01 s06 systemd: Starting User Slice of root. >>>>> Mar 1 11:05:01 s06 systemd: Started Session 196230 of user root. >>>>> Mar 1 11:05:01 s06 systemd: Starting Session 196230 of user root. >>>>> Mar 1 11:05:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 1 11:05:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 1 11:05:59 s06 glustershd[70117]: [2019-03-01 10:05:59.347094] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-33: server 192.168.0.51:49163 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 1 11:06:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 1 11:06:01 s06 systemd: Starting User Slice of root. >>>>> Mar 1 11:06:01 s06 systemd: Started Session 196231 of user root. >>>>> Mar 1 11:06:01 s06 systemd: Starting Session 196231 of user root. >>>>> Mar 1 11:06:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 1 11:06:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 1 11:06:12 s06 glustershd[70117]: [2019-03-01 10:06:12.351319] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-1: server 192.168.0.52:49153 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 1 11:06:38 s06 glustershd[70117]: [2019-03-01 10:06:38.356920] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-7: server 192.168.0.52:49155 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 1 11:07:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 1 11:07:01 s06 systemd: Starting User Slice of root. >>>>> Mar 1 11:07:01 s06 systemd: Started Session 196232 of user root. >>>>> Mar 1 11:07:01 s06 systemd: Starting Session 196232 of user root. >>>>> Mar 1 11:07:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 1 11:07:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 1 11:07:36 s06 glustershd[70117]: [2019-03-01 10:07:36.366259] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-0: server 192.168.0.51:49152 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 1 11:08:01 s06 systemd: Created slice User Slice of root. >>>>> >>>>> No ?blocked? word is in /var/log/messages files on other cluster servers. >>>>> In attachment, the /var/log/messages file from s06 server. >>>>> >>>>> Thank you in advance, >>>>> Mauro >>>>> >>>>> >>>>> >>>>> >>>>>> On 1 Mar 2019, at 11:47, Milind Changire > wrote: >>>>>> >>>>>> The traces of very high disk activity on the servers are often found in /var/log/messages >>>>>> You might want to grep for "blocked for" in /var/log/messages on s06 and correlate the timestamps to confirm the unresponsiveness as reported in gluster client logs. >>>>>> In cases of high disk activity, although the operating system continues to respond to ICMP pings, the processes writing to disks often get blocked to a large flush to the disk which could span beyond 42 seconds and hence result in ping-timer-expiry logs. >>>>>> >>>>>> As a side note: >>>>>> If you indeed find gluster processes being blocked in /var/log/messages, you might want to tweak sysctl tunables called vm.dirty_background_ratio or vm.dirty_background_bytes to a smaller value than the existing. Please read up more on those tunables before touching the settings. >>>>>> >>>>>> >>>>>> On Fri, Mar 1, 2019 at 4:06 PM Mauro Tridici > wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> in attachment the client log captured after changing network.ping-timeout option. >>>>>> I noticed this error involving server 192.168.0.56 (s06) >>>>>> >>>>>> [2019-03-01 09:23:36.077287] I [rpc-clnt.c:1962:rpc_clnt_reconfig] 0-tier2-client-71: changing ping timeout to 42 (from 0) >>>>>> [2019-03-01 09:23:36.078213] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>>> [2019-03-01 09:23:36.078432] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>>> [2019-03-01 09:23:36.092357] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>>> [2019-03-01 09:23:36.094146] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>>> [2019-03-01 10:06:24.708082] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-50: server 192.168.0.56:49156 has not responded in the last 42 seconds, disconnecting. >>>>>> >>>>>> I don?t know why it happens, s06 server seems to be reachable. >>>>>> >>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>> Trying 192.168.0.56... >>>>>> Connected to 192.168.0.56. >>>>>> Escape character is '^]'. >>>>>> ^CConnection closed by foreign host. >>>>>> [athena_login2][/users/home/sysm02/]> ping 192.168.0.56 >>>>>> PING 192.168.0.56 (192.168.0.56) 56(84) bytes of data. >>>>>> 64 bytes from 192.168.0.56 : icmp_seq=1 ttl=64 time=0.116 ms >>>>>> 64 bytes from 192.168.0.56 : icmp_seq=2 ttl=64 time=0.101 ms >>>>>> >>>>>> --- 192.168.0.56 ping statistics --- >>>>>> 2 packets transmitted, 2 received, 0% packet loss, time 1528ms >>>>>> rtt min/avg/max/mdev = 0.101/0.108/0.116/0.012 ms >>>>>> >>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>> Trying 192.168.0.56... >>>>>> Connected to 192.168.0.56. >>>>>> Escape character is '^]'. >>>>>> >>>>>> Thank you for your help, >>>>>> Mauro >>>>>> >>>>>> >>>>>> >>>>>>> On 1 Mar 2019, at 10:29, Mauro Tridici > wrote: >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> thank you for the explanation. >>>>>>> I just changed network.ping-timeout option to default value (network.ping-timeout=42). >>>>>>> >>>>>>> I will check the logs to see if the errors will appear again. >>>>>>> >>>>>>> Regards, >>>>>>> Mauro >>>>>>> >>>>>>>> On 1 Mar 2019, at 04:43, Milind Changire > wrote: >>>>>>>> >>>>>>>> network.ping-timeout should not be set to zero for non-glusterd clients. >>>>>>>> glusterd is a special case for which ping-timeout is set to zero via /etc/glusterfs/glusterd.vol >>>>>>>> >>>>>>>> Setting network.ping-timeout to zero disables arming of the ping timer for connections. This disables testing the connection for responsiveness and hence avoids proactive fail-over. >>>>>>>> >>>>>>>> Please reset network.ping-timeout to a non-zero positive value, eg. 42 >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Feb 28, 2019 at 5:07 PM Nithya Balachandran > wrote: >>>>>>>> Adding Raghavendra and Milind to comment on this. >>>>>>>> >>>>>>>> What is the effect of setting network.ping-timeout to 0 and should it be set back to 42? >>>>>>>> Regards, >>>>>>>> Nithya >>>>>>>> >>>>>>>> On Thu, 28 Feb 2019 at 16:01, Mauro Tridici > wrote: >>>>>>>> Hi Nithya, >>>>>>>> >>>>>>>> sorry for the late. >>>>>>>> network.ping-timeout has been set to 0 in order to try to solve some timeout problems, but it didn?t help. >>>>>>>> I can set it to the default value. >>>>>>>> >>>>>>>> Can I proceed with the change? >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Mauro >>>>>>>> >>>>>>>> >>>>>>>>> On 28 Feb 2019, at 04:41, Nithya Balachandran > wrote: >>>>>>>>> >>>>>>>>> Hi Mauro, >>>>>>>>> >>>>>>>>> Is network.ping-timeout still set to 0. The default value is 42. Is there a particular reason why this was changed? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Nithya >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, 27 Feb 2019 at 21:32, Mauro Tridici > wrote: >>>>>>>>> >>>>>>>>> Hi Xavi, >>>>>>>>> >>>>>>>>> thank you for the detailed explanation and suggestions. >>>>>>>>> Yes, transport.listen-backlog option is still set to 1024. >>>>>>>>> >>>>>>>>> I will check the network and connectivity status using ?ping? and ?telnet? as soon as the errors will come back again. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Mauro >>>>>>>>> >>>>>>>>>> Il giorno 27 feb 2019, alle ore 16:42, Xavi Hernandez > ha scritto: >>>>>>>>>> >>>>>>>>>> Hi Mauro, >>>>>>>>>> >>>>>>>>>> those errors say that the mount point is not connected to some of the bricks while executing operations. I see references to 3rd and 6th bricks of several disperse sets, which seem to map to server s06. For some reason, gluster is having troubles connecting from the client machine to that particular server. At the end of the log I see that after long time a reconnect is done to both of them. However little after, other bricks from the s05 get disconnected and a reconnect times out. >>>>>>>>>> >>>>>>>>>> That's really odd. It seems like if server/communication is cut to s06 for some time, then restored, and then the same happens to the next server. >>>>>>>>>> >>>>>>>>>> If the servers are really online and it's only a communication issue, it explains why server memory and network has increased: if the problem only exists between the client and servers, any write made by the client will automatically mark the file as damaged, since some of the servers have not been updated. Since self-heal runs from the server nodes, they will probably be correctly connected to all bricks, which allows them to heal the just damaged file, which increases memory and network usage. >>>>>>>>>> >>>>>>>>>> I guess you still have transport.listen-backlog set to 1024, right ? >>>>>>>>>> >>>>>>>>>> Just to try to identify if the problem really comes from network, can you check if you lose some pings from the client to all of the servers while you are seeing those errors in the log file ? >>>>>>>>>> >>>>>>>>>> You can also check if during those errors, you can telnet to the port of the brick from the client. >>>>>>>>>> >>>>>>>>>> Xavi >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Feb 26, 2019 at 10:17 AM Mauro Tridici > wrote: >>>>>>>>>> Hi Nithya, >>>>>>>>>> >>>>>>>>>> ?df -h? operation is not still slow, but no users are using the volume, RAM and NETWORK usage is ok on the client node. >>>>>>>>>> >>>>>>>>>> I was worried about this kind of warnings/errors: >>>>>>>>>> >>>>>>>>>> [2019-02-25 10:59:00.664323] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-6: Executing operation with some subvolumes unavailable (20) >>>>>>>>>> >>>>>>>>>> [2019-02-26 03:11:35.212603] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-26 03:10:56.549903 (xid=0x106f1c5) >>>>>>>>>> >>>>>>>>>> [2019-02-26 03:13:03.313831] E [socket.c:2376:socket_connect_finish] 0-tier2-client-50: connection to 192.168.0.56:49156 failed (Timeout della connessione); disconnecting socket >>>>>>>>>> >>>>>>>>>> It seems that some subvolumes are not available and 192.168.0.56 server (s06) is not reachable. >>>>>>>>>> But gluster servers are up&running and bricks are ok. >>>>>>>>>> >>>>>>>>>> In attachment the updated tier2.log file. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thank you. >>>>>>>>>> Regards, >>>>>>>>>> Mauro >>>>>>>>>> >>>>>>>>>>> Il giorno 26 feb 2019, alle ore 04:03, Nithya Balachandran > ha scritto: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I see a lot of EC messages in the log but they don't seem very serious. Xavi, can you take a look? >>>>>>>>>>> >>>>>>>>>>> The only errors I see are: >>>>>>>>>>> [2019-02-25 10:58:45.519871] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-25 10:57:47.429969 (xid=0xd26fe7) >>>>>>>>>>> [2019-02-25 10:58:51.461493] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-41: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-25 10:57:47.499174 (xid=0xf47d6a) >>>>>>>>>>> [2019-02-25 11:07:57.152874] E [socket.c:2376:socket_connect_finish] 0-tier2-client-70: connection to 192.168.0.55:49163 failed (Timeout della connessione); disconnecting socket >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Is the df -h operation still slow? If yes, can you take a tcpdump of the client while running df -h and send that across? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Nithya >>>>>>>>>>> >>>>>>>>>>> On Mon, 25 Feb 2019 at 17:27, Mauro Tridici > wrote: >>>>>>>>>>> >>>>>>>>>>> Sorry, some minutes after my last mail message, I noticed that ?df -h? command hanged for a while before returns the prompt. >>>>>>>>>>> Yesterday, everything was ok in the gluster client log, but, today, I see a lot of errors (please, take a look to the attached file). >>>>>>>>>>> >>>>>>>>>>> On the client node, I detected an important RAM e NETWORK usage. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Do you think that the errors have been caused by the client resources usage? >>>>>>>>>>> >>>>>>>>>>> Thank you in advance, >>>>>>>>>>> Mauro >>>>>>>>>>> >>>> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mauro.tridici at cmcc.it Thu Mar 14 10:08:21 2019 From: mauro.tridici at cmcc.it (Mauro Tridici) Date: Thu, 14 Mar 2019 11:08:21 +0100 Subject: [Gluster-devel] [Gluster-users] "rpc_clnt_ping_timer_expired" errors In-Reply-To: References: <96B07283-D8AB-4F06-909D-E00424625528@cmcc.it> <42758A0E-8BE9-497D-BDE3-55D7DC633BC7@cmcc.it> <6A8CF4A4-98EA-48C3-A059-D60D1B2721C7@cmcc.it> <2CF49168-9C1B-4931-8C34-8157262A137A@cmcc.it> <7A151CC9-A0AE-4A45-B450-A4063D216D9E@cmcc.it> <32D53ECE-3F49-4415-A6EE-241B351AC2BA@cmcc.it> <4685A75B-5978-4338-9C9F-4A02FB40B9BC@cmcc.it> <4D2E6B04-C2E8-4FD5-B72D-E7C05931C6F9@cmcc.it> <4E332A56-B318-4BC1-9F44-84AB4392A5DE@cmcc.it> <832FD362-3B14-40D8-8530-604419300476@cmcc.it> <8D926643-1093-48ED-823F-D8F117F702CF@cmcc.it> <9D0BE438-DF11-4D0A-AB85-B44357032F29@cmcc.it> <5F0AC378-8170-4342-8473-9C17159CAC1D@cmcc.it> <7A50E86D-9E27-4EA7-883B-45E9F973991A@cmcc.it> <58B5DB7F-DCB4-4FBF-B1F7-681315B1613A@cmcc.it> <6327B44F-4E7E-46BB-A74C-70F4457DD1EB@cmcc.it> <0167DF4A-8329-4A1A-B439-857DFA6F78BB@cmcc.it> <763F334E-35B8-4729-B8E1-D30866754EEE@cmcc.it> <91DFC9AC-4805-41EB-AC6F-5722E018DD6E@cmcc.it> <8A9752B8-B231-4570-8FF4-8D3D781E7D42@cmcc.it> <47A24804-F975-4EE6-9FA5-67FCDA18D707@cmcc.it> <637FF5D2-D1F4-4686-9D48-646A96F67B96@cmcc.it> <4A87495F-3755-48F7-8507-085869069C64@cmcc.it> <3854BBF6-5B98-4AB3-A67E-E7DE59E69A63@cmcc.it> <313DA021-9173-4899-96B0-831B10B00B61@cmcc.it> <17996AFD-DFC8-40F3-9D09-DB6DDAD5B7D6@cmcc.it> <7074B5D8-955A-4802-A9F3-606C99734417@cmcc.it> <83B84BF9-8334-4230-B6F8-0BC4BFBEFF15@cmcc.it> <133B9AE4-9792-4F72-AD91-D36E7B9EC711@cmcc.it> <6611C4B0-57FD-4390-88B5-BD373555D4C5@cmcc.it> Message-ID: Hi Raghavendra, I just changed the client option value to 8. I will check the volume behaviour during the next hours. The GlusterFS version is 3.12.14. I will provide you the logs as soon as the activity load will be high. Thank you, Mauro > On 14 Mar 2019, at 04:57, Raghavendra Gowdappa wrote: > > > > On Wed, Mar 13, 2019 at 3:55 PM Mauro Tridici > wrote: > Hi Raghavendra, > > Yes, server.event-thread has been changed from 4 to 8. > > Was client.event-thread value too changed to 8? If not, I would like to know the results of including this tuning too. Also, if possible, can you get the output of following command from problematic clients and bricks (during the duration when load tends to be high and ping-timer-expiry is seen)? > > # top -bHd 3 > > This will help us to know CPU utilization of event-threads. > > And I forgot to ask, what version of Glusterfs are you using? > > During last days, I noticed that the error events are still here although they have been considerably reduced. > > So, I used grep command against the log files in order to provide you a global vision about the warning, error and critical events appeared today at 06:xx (may be useful I hope). > I collected the info from s06 gluster server, but the behaviour is the the almost the same on the other gluster servers. > > ERRORS: > CWD: /var/log/glusterfs > COMMAND: grep " E " *.log |grep "2019-03-13 06:" > > (I can see a lot of this kind of message in the same period but I'm notifying you only one record for each type of error) > > glusterd.log:[2019-03-13 06:12:35.982863] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of /var/run/gluster/tier2_quota_list/ > > glustershd.log:[2019-03-13 06:14:28.666562] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a71ddcebb] (--> /lib64/libgfr > pc.so.0(saved_frames_unwind+0x1de)[0x7f4a71ba1d9e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f4a71ba1ebe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup > +0x90)[0x7f4a71ba3640] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7f4a71ba4130] ))))) 0-tier2-client-55: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) > called at 2019-03-13 06:14:14.858441 (xid=0x17fddb50) > > glustershd.log:[2019-03-13 06:17:48.883825] E [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to 192.168.0.55:49158 failed (Connection timed out); disco > nnecting socket > glustershd.log:[2019-03-13 06:19:58.931798] E [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to 192.168.0.55:49158 failed (Connection timed out); disco > nnecting socket > glustershd.log:[2019-03-13 06:22:08.979829] E [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to 192.168.0.55:49158 failed (Connection timed out); disco > nnecting socket > glustershd.log:[2019-03-13 06:22:36.226847] E [MSGID: 114031] [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote operation failed [Transport endpoint > is not connected] > glustershd.log:[2019-03-13 06:22:36.306669] E [MSGID: 114031] [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote operation failed [Transport endpoint > is not connected] > glustershd.log:[2019-03-13 06:22:36.385257] E [MSGID: 114031] [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote operation failed [Transport endpoint > is not connected] > > WARNINGS: > CWD: /var/log/glusterfs > COMMAND: grep " W " *.log |grep "2019-03-13 06:" > > (I can see a lot of this kind of message in the same period but I'm notifying you only one record for each type of warnings) > > glustershd.log:[2019-03-13 06:14:28.666772] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-tier2-client-55: remote operation failed. Path: 0f-f34d-4c25-bbe8-74bde0248d7e> (b6b35d0f-f34d-4c25-bbe8-74bde0248d7e). Key: (null) [Transport endpoint is not connected] > > glustershd.log:[2019-03-13 06:14:31.421576] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (2) > > glustershd.log:[2019-03-13 06:15:31.547417] W [MSGID: 122032] [ec-heald.c:266:ec_shd_index_sweep] 0-tier2-disperse-9: unable to get index-dir on tier2-client-55 [Operation > now in progress] > > quota-mount-tier2.log:[2019-03-13 06:12:36.116277] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'trans > port.address-family', continuing with correction > quota-mount-tier2.log:[2019-03-13 06:12:36.198430] W [MSGID: 101174] [graph.c:363:_log_if_unknown_option] 0-tier2-readdir-ahead: option 'parallel-readdir' is not recognized > quota-mount-tier2.log:[2019-03-13 06:12:37.945007] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f340892be25] -->/usr/sbin/glusterfs(gluste > rfs_sigwaiter+0xe5) [0x55ef010164b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55ef0101632b] ) 0-: received signum (15), shutting down > > CRITICALS: > CWD: /var/log/glusterfs > COMMAND: grep " C " *.log |grep "2019-03-13 06:" > > no critical errors at 06:xx > only one critical error during the day > > [root at s06 glusterfs]# grep " C " *.log |grep "2019-03-13" > glustershd.log:[2019-03-13 02:21:29.126279] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server 192.168.0.55:49158 has not responded in the last 42 seconds, disconnecting. > > > Thank you very much for your help. > Regards, > Mauro > >> On 12 Mar 2019, at 05:17, Raghavendra Gowdappa > wrote: >> >> Was the suggestion to increase server.event-thread values tried? If yes, what were the results? >> >> On Mon, Mar 11, 2019 at 2:40 PM Mauro Tridici > wrote: >> Dear All, >> >> do you have any suggestions about the right way to "debug" this issue? >> In attachment, the updated logs of ?s06" gluster server. >> >> I noticed a lot of intermittent warning and error messages. >> >> Thank you in advance, >> Mauro >> >> >> >>> On 4 Mar 2019, at 18:45, Raghavendra Gowdappa > wrote: >>> >>> >>> +Gluster Devel , +Gluster-users >>> >>> I would like to point out another issue. Even if what I suggested prevents disconnects, part of the solution would be only symptomatic treatment and doesn't address the root cause of the problem. In most of the ping-timer-expiry issues, the root cause is the increased load on bricks and the inability of bricks to be responsive under high load. So, the actual solution would be doing any or both of the following: >>> * identify the source of increased load and if possible throttle it. Internal heal processes like self-heal, rebalance, quota heal are known to pump traffic into bricks without much throttling (io-threads _might_ do some throttling, but my understanding is its not sufficient). >>> * identify the reason for bricks to become unresponsive during load. This may be fixable issues like not enough event-threads to read from network or difficult to fix issues like fsync on backend fs freezing the process or semi fixable issues (in code) like lock contention. >>> >>> So any genuine effort to fix ping-timer-issues (to be honest most of the times they are not issues related to rpc/network) would involve performance characterization of various subsystems on bricks and clients. Various subsystems can include (but not necessarily limited to), underlying OS/filesystem, glusterfs processes, CPU consumption etc >>> >>> regards, >>> Raghavendra >>> >>> On Mon, Mar 4, 2019 at 9:31 PM Mauro Tridici > wrote: >>> Thank you, let?s try! >>> I will inform you about the effects of the change. >>> >>> Regards, >>> Mauro >>> >>>> On 4 Mar 2019, at 16:55, Raghavendra Gowdappa > wrote: >>>> >>>> >>>> >>>> On Mon, Mar 4, 2019 at 8:54 PM Mauro Tridici > wrote: >>>> Hi Raghavendra, >>>> >>>> thank you for your reply. >>>> Yes, you are right. It is a problem that seems to happen randomly. >>>> At this moment, server.event-threads value is 4. I will try to increase this value to 8. Do you think that it could be a valid value ? >>>> >>>> Yes. We can try with that. You should see at least frequency of ping-timer related disconnects reduce with this value (even if it doesn't eliminate the problem completely). >>>> >>>> >>>> Regards, >>>> Mauro >>>> >>>> >>>>> On 4 Mar 2019, at 15:36, Raghavendra Gowdappa > wrote: >>>>> >>>>> >>>>> >>>>> On Mon, Mar 4, 2019 at 8:01 PM Nithya Balachandran > wrote: >>>>> Hi Mauro, >>>>> >>>>> It looks like some problem on s06. Are all your other nodes ok? Can you send us the gluster logs from this node? >>>>> >>>>> @Raghavendra G , do you have any idea as to how this can be debugged? Maybe running top ? Or debug brick logs? >>>>> >>>>> If we can reproduce the problem, collecting tcpdump on both ends of connection will help. But, one common problem is these bugs are inconsistently reproducible and hence we may not be able to capture tcpdump at correct intervals. Other than that, we can try to collect some evidence that poller threads were busy (waiting on locks). But, not sure what debug data provides that information. >>>>> >>>>> From what I know, its difficult to collect evidence for this issue and we could only reason about it. >>>>> >>>>> We can try a workaround though - try increasing server.event-threads and see whether ping-timer expiry issues go away with an optimal value. If that's the case, it kind of provides proof for our hypothesis. >>>>> >>>>> >>>>> >>>>> Regards, >>>>> Nithya >>>>> >>>>> On Mon, 4 Mar 2019 at 15:25, Mauro Tridici > wrote: >>>>> Hi All, >>>>> >>>>> some minutes ago I received this message from NAGIOS server >>>>> >>>>> ***** Nagios ***** >>>>> >>>>> Notification Type: PROBLEM >>>>> >>>>> Service: Brick - /gluster/mnt2/brick >>>>> Host: s06 >>>>> Address: s06-stg >>>>> State: CRITICAL >>>>> >>>>> Date/Time: Mon Mar 4 10:25:33 CET 2019 >>>>> >>>>> Additional Info: >>>>> CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. >>>>> >>>>> I checked the network, RAM and CPUs usage on s06 node and everything seems to be ok. >>>>> No bricks are in error state. In /var/log/messages, I detected again a crash of ?check_vol_utili? that I think it is a module used by NRPE executable (that is the NAGIOS client). >>>>> >>>>> Mar 4 10:15:29 s06 kernel: traps: check_vol_utili[161224] general protection ip:7facffa0a66d sp:7ffe9f4e6fc0 error:0 in libglusterfs.so.0.0.1[7facff9b7000+f7000] >>>>> Mar 4 10:15:29 s06 abrt-hook-ccpp: Process 161224 (python2.7) of user 0 killed by SIGSEGV - dumping core >>>>> Mar 4 10:15:29 s06 abrt-server: Generating core_backtrace >>>>> Mar 4 10:15:29 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>>> Mar 4 10:16:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:16:01 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:16:01 s06 systemd: Started Session 201010 of user root. >>>>> Mar 4 10:16:01 s06 systemd: Starting Session 201010 of user root. >>>>> Mar 4 10:16:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 4 10:16:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 4 10:16:24 s06 abrt-server: Duplicate: UUID >>>>> Mar 4 10:16:24 s06 abrt-server: DUP_OF_DIR: /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>> Mar 4 10:16:24 s06 abrt-server: Deleting problem directory ccpp-2019-03-04-10:15:29-161224 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>> Mar 4 10:16:24 s06 abrt-server: Generating core_backtrace >>>>> Mar 4 10:16:24 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>>> Mar 4 10:16:24 s06 abrt-server: Cannot notify '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event 'report_uReport' exited with 1 >>>>> Mar 4 10:16:24 s06 abrt-hook-ccpp: Process 161391 (python2.7) of user 0 killed by SIGABRT - dumping core >>>>> Mar 4 10:16:25 s06 abrt-server: Generating core_backtrace >>>>> Mar 4 10:16:25 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>>> Mar 4 10:17:01 s06 systemd: Created slice User Slice of root. >>>>> >>>>> Also, I noticed the following errors that I think are very critical: >>>>> >>>>> Mar 4 10:21:12 s06 glustershd[20355]: [2019-03-04 09:21:12.954798] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server 192.168.0.55:49158 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:22:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:22:01 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:22:01 s06 systemd: Started Session 201017 of user root. >>>>> Mar 4 10:22:01 s06 systemd: Starting Session 201017 of user root. >>>>> Mar 4 10:22:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 4 10:22:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 4 10:22:03 s06 glustershd[20355]: [2019-03-04 09:22:03.964120] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server 192.168.0.54:49165 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:23:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:23:01 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:23:01 s06 systemd: Started Session 201018 of user root. >>>>> Mar 4 10:23:01 s06 systemd: Starting Session 201018 of user root. >>>>> Mar 4 10:23:02 s06 systemd: Removed slice User Slice of root. >>>>> Mar 4 10:23:02 s06 systemd: Stopping User Slice of root. >>>>> Mar 4 10:24:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:24:01 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:24:01 s06 systemd: Started Session 201019 of user root. >>>>> Mar 4 10:24:01 s06 systemd: Starting Session 201019 of user root. >>>>> Mar 4 10:24:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 4 10:24:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 4 10:24:03 s06 glustershd[20355]: [2019-03-04 09:24:03.982502] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-16: server 192.168.0.52:49158 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746109] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-3: server 192.168.0.51:49153 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746215] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server 192.168.0.52:49156 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746260] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-21: server 192.168.0.51:49159 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746296] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server 192.168.0.52:49161 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746413] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server 192.168.0.54:49165 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:24:07 s06 glustershd[20355]: [2019-03-04 09:24:07.982952] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-45: server 192.168.0.54:49155 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:24:18 s06 glustershd[20355]: [2019-03-04 09:24:18.990929] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server 192.168.0.52:49161 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:24:31 s06 glustershd[20355]: [2019-03-04 09:24:31.995781] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-20: server 192.168.0.53:49159 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:25:01 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:25:01 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:25:01 s06 systemd: Started Session 201020 of user root. >>>>> Mar 4 10:25:01 s06 systemd: Starting Session 201020 of user root. >>>>> Mar 4 10:25:01 s06 systemd: Removed slice User Slice of root. >>>>> Mar 4 10:25:01 s06 systemd: Stopping User Slice of root. >>>>> Mar 4 10:25:57 s06 systemd: Created slice User Slice of root. >>>>> Mar 4 10:25:57 s06 systemd: Starting User Slice of root. >>>>> Mar 4 10:25:57 s06 systemd-logind: New session 201021 of user root. >>>>> Mar 4 10:25:57 s06 systemd: Started Session 201021 of user root. >>>>> Mar 4 10:25:57 s06 systemd: Starting Session 201021 of user root. >>>>> Mar 4 10:26:01 s06 systemd: Started Session 201022 of user root. >>>>> Mar 4 10:26:01 s06 systemd: Starting Session 201022 of user root. >>>>> Mar 4 10:26:21 s06 nrpe[162388]: Error: Could not complete SSL handshake with 192.168.1.56 : 5 >>>>> Mar 4 10:27:01 s06 systemd: Started Session 201023 of user root. >>>>> Mar 4 10:27:01 s06 systemd: Starting Session 201023 of user root. >>>>> Mar 4 10:28:01 s06 systemd: Started Session 201024 of user root. >>>>> Mar 4 10:28:01 s06 systemd: Starting Session 201024 of user root. >>>>> Mar 4 10:29:01 s06 systemd: Started Session 201025 of user root. >>>>> Mar 4 10:29:01 s06 systemd: Starting Session 201025 of user root. >>>>> >>>>> But, unfortunately, I don?t understand why it is happening. >>>>> Now, NAGIOS server shows that s06 status is ok: >>>>> >>>>> ***** Nagios ***** >>>>> >>>>> Notification Type: RECOVERY >>>>> >>>>> Service: Brick - /gluster/mnt2/brick >>>>> Host: s06 >>>>> Address: s06-stg >>>>> State: OK >>>>> >>>>> Date/Time: Mon Mar 4 10:35:23 CET 2019 >>>>> >>>>> Additional Info: >>>>> OK: Brick /gluster/mnt2/brick is up >>>>> >>>>> Nothing is changed from RAM, CPUs, and NETWORK point of view. >>>>> /var/log/message file has been updated: >>>>> >>>>> Mar 4 10:32:01 s06 systemd: Starting Session 201029 of user root. >>>>> Mar 4 10:32:30 s06 glustershd[20355]: [2019-03-04 09:32:30.069082] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server 192.168.0.52:49156 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:32:55 s06 glustershd[20355]: [2019-03-04 09:32:55.074689] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-66: server 192.168.0.54:49167 has not responded in the last 42 seconds, disconnecting. >>>>> Mar 4 10:33:01 s06 systemd: Started Session 201030 of user root. >>>>> Mar 4 10:33:01 s06 systemd: Starting Session 201030 of user root. >>>>> Mar 4 10:34:01 s06 systemd: Started Session 201031 of user root. >>>>> Mar 4 10:34:01 s06 systemd: Starting Session 201031 of user root. >>>>> Mar 4 10:35:01 s06 nrpe[162562]: Could not read request from client 192.168.1.56, bailing out... >>>>> Mar 4 10:35:01 s06 nrpe[162562]: INFO: SSL Socket Shutdown. >>>>> Mar 4 10:35:01 s06 systemd: Started Session 201032 of user root. >>>>> Mar 4 10:35:01 s06 systemd: Starting Session 201032 of user root. >>>>> >>>>> Could you please help me to understand what it?s happening ? >>>>> Thank you in advance. >>>>> >>>>> Rergards, >>>>> Mauro >>>>> >>>>> >>>>>> On 1 Mar 2019, at 12:17, Mauro Tridici > wrote: >>>>>> >>>>>> >>>>>> Thank you, Milind. >>>>>> I executed the instructions you suggested: >>>>>> >>>>>> - grep ?blocked for? /var/log/messages on s06 returns no output (no ?blocked? word is detected in messages file); >>>>>> - in /var/log/messages file I can see this kind of error repeated for a lot of times: >>>>>> >>>>>> Mar 1 08:43:01 s06 systemd: Starting Session 196071 of user root. >>>>>> Mar 1 08:43:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 1 08:43:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 1 08:43:02 s06 kernel: traps: check_vol_utili[57091] general protection ip:7f88e76ee66d sp:7ffe5a5bcc30 error:0 in libglusterfs.so.0.0.1[7f88e769b000+f7000] >>>>>> Mar 1 08:43:02 s06 abrt-hook-ccpp: Process 57091 (python2.7) of user 0 killed by SIGSEGV - dumping core >>>>>> Mar 1 08:43:02 s06 abrt-server: Generating core_backtrace >>>>>> Mar 1 08:43:02 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>>>> Mar 1 08:43:58 s06 abrt-server: Duplicate: UUID >>>>>> Mar 1 08:43:58 s06 abrt-server: DUP_OF_DIR: /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>>> Mar 1 08:43:58 s06 abrt-server: Deleting problem directory ccpp-2019-03-01-08:43:02-57091 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Activating service name='org.freedesktop.problems' (using servicehelper) >>>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Successfully activated service 'org.freedesktop.problems' >>>>>> Mar 1 08:43:58 s06 abrt-server: Generating core_backtrace >>>>>> Mar 1 08:43:58 s06 abrt-server: Error: Unable to open './coredump': No such file or directory >>>>>> Mar 1 08:43:58 s06 abrt-server: Cannot notify '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event 'report_uReport' exited with 1 >>>>>> Mar 1 08:44:01 s06 systemd: Created slice User Slice of root. >>>>>> Mar 1 08:44:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 1 08:44:01 s06 systemd: Started Session 196072 of user root. >>>>>> Mar 1 08:44:01 s06 systemd: Starting Session 196072 of user root. >>>>>> Mar 1 08:44:01 s06 systemd: Removed slice User Slice of root. >>>>>> >>>>>> - in /var/log/messages file I can see also 4 errors related to other cluster servers: >>>>>> >>>>>> Mar 1 11:05:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 1 11:05:01 s06 systemd: Started Session 196230 of user root. >>>>>> Mar 1 11:05:01 s06 systemd: Starting Session 196230 of user root. >>>>>> Mar 1 11:05:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 1 11:05:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 1 11:05:59 s06 glustershd[70117]: [2019-03-01 10:05:59.347094] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-33: server 192.168.0.51:49163 has not responded in the last 42 seconds, disconnecting. >>>>>> Mar 1 11:06:01 s06 systemd: Created slice User Slice of root. >>>>>> Mar 1 11:06:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 1 11:06:01 s06 systemd: Started Session 196231 of user root. >>>>>> Mar 1 11:06:01 s06 systemd: Starting Session 196231 of user root. >>>>>> Mar 1 11:06:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 1 11:06:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 1 11:06:12 s06 glustershd[70117]: [2019-03-01 10:06:12.351319] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-1: server 192.168.0.52:49153 has not responded in the last 42 seconds, disconnecting. >>>>>> Mar 1 11:06:38 s06 glustershd[70117]: [2019-03-01 10:06:38.356920] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-7: server 192.168.0.52:49155 has not responded in the last 42 seconds, disconnecting. >>>>>> Mar 1 11:07:01 s06 systemd: Created slice User Slice of root. >>>>>> Mar 1 11:07:01 s06 systemd: Starting User Slice of root. >>>>>> Mar 1 11:07:01 s06 systemd: Started Session 196232 of user root. >>>>>> Mar 1 11:07:01 s06 systemd: Starting Session 196232 of user root. >>>>>> Mar 1 11:07:01 s06 systemd: Removed slice User Slice of root. >>>>>> Mar 1 11:07:01 s06 systemd: Stopping User Slice of root. >>>>>> Mar 1 11:07:36 s06 glustershd[70117]: [2019-03-01 10:07:36.366259] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-0: server 192.168.0.51:49152 has not responded in the last 42 seconds, disconnecting. >>>>>> Mar 1 11:08:01 s06 systemd: Created slice User Slice of root. >>>>>> >>>>>> No ?blocked? word is in /var/log/messages files on other cluster servers. >>>>>> In attachment, the /var/log/messages file from s06 server. >>>>>> >>>>>> Thank you in advance, >>>>>> Mauro >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On 1 Mar 2019, at 11:47, Milind Changire > wrote: >>>>>>> >>>>>>> The traces of very high disk activity on the servers are often found in /var/log/messages >>>>>>> You might want to grep for "blocked for" in /var/log/messages on s06 and correlate the timestamps to confirm the unresponsiveness as reported in gluster client logs. >>>>>>> In cases of high disk activity, although the operating system continues to respond to ICMP pings, the processes writing to disks often get blocked to a large flush to the disk which could span beyond 42 seconds and hence result in ping-timer-expiry logs. >>>>>>> >>>>>>> As a side note: >>>>>>> If you indeed find gluster processes being blocked in /var/log/messages, you might want to tweak sysctl tunables called vm.dirty_background_ratio or vm.dirty_background_bytes to a smaller value than the existing. Please read up more on those tunables before touching the settings. >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 1, 2019 at 4:06 PM Mauro Tridici > wrote: >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> in attachment the client log captured after changing network.ping-timeout option. >>>>>>> I noticed this error involving server 192.168.0.56 (s06) >>>>>>> >>>>>>> [2019-03-01 09:23:36.077287] I [rpc-clnt.c:1962:rpc_clnt_reconfig] 0-tier2-client-71: changing ping timeout to 42 (from 0) >>>>>>> [2019-03-01 09:23:36.078213] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>>>> [2019-03-01 09:23:36.078432] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>>>> [2019-03-01 09:23:36.092357] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>>>> [2019-03-01 09:23:36.094146] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing >>>>>>> [2019-03-01 10:06:24.708082] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-50: server 192.168.0.56:49156 has not responded in the last 42 seconds, disconnecting. >>>>>>> >>>>>>> I don?t know why it happens, s06 server seems to be reachable. >>>>>>> >>>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>>> Trying 192.168.0.56... >>>>>>> Connected to 192.168.0.56. >>>>>>> Escape character is '^]'. >>>>>>> ^CConnection closed by foreign host. >>>>>>> [athena_login2][/users/home/sysm02/]> ping 192.168.0.56 >>>>>>> PING 192.168.0.56 (192.168.0.56) 56(84) bytes of data. >>>>>>> 64 bytes from 192.168.0.56 : icmp_seq=1 ttl=64 time=0.116 ms >>>>>>> 64 bytes from 192.168.0.56 : icmp_seq=2 ttl=64 time=0.101 ms >>>>>>> >>>>>>> --- 192.168.0.56 ping statistics --- >>>>>>> 2 packets transmitted, 2 received, 0% packet loss, time 1528ms >>>>>>> rtt min/avg/max/mdev = 0.101/0.108/0.116/0.012 ms >>>>>>> >>>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>>> Trying 192.168.0.56... >>>>>>> Connected to 192.168.0.56. >>>>>>> Escape character is '^]'. >>>>>>> >>>>>>> Thank you for your help, >>>>>>> Mauro >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On 1 Mar 2019, at 10:29, Mauro Tridici > wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> thank you for the explanation. >>>>>>>> I just changed network.ping-timeout option to default value (network.ping-timeout=42). >>>>>>>> >>>>>>>> I will check the logs to see if the errors will appear again. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Mauro >>>>>>>> >>>>>>>>> On 1 Mar 2019, at 04:43, Milind Changire > wrote: >>>>>>>>> >>>>>>>>> network.ping-timeout should not be set to zero for non-glusterd clients. >>>>>>>>> glusterd is a special case for which ping-timeout is set to zero via /etc/glusterfs/glusterd.vol >>>>>>>>> >>>>>>>>> Setting network.ping-timeout to zero disables arming of the ping timer for connections. This disables testing the connection for responsiveness and hence avoids proactive fail-over. >>>>>>>>> >>>>>>>>> Please reset network.ping-timeout to a non-zero positive value, eg. 42 >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Feb 28, 2019 at 5:07 PM Nithya Balachandran > wrote: >>>>>>>>> Adding Raghavendra and Milind to comment on this. >>>>>>>>> >>>>>>>>> What is the effect of setting network.ping-timeout to 0 and should it be set back to 42? >>>>>>>>> Regards, >>>>>>>>> Nithya >>>>>>>>> >>>>>>>>> On Thu, 28 Feb 2019 at 16:01, Mauro Tridici > wrote: >>>>>>>>> Hi Nithya, >>>>>>>>> >>>>>>>>> sorry for the late. >>>>>>>>> network.ping-timeout has been set to 0 in order to try to solve some timeout problems, but it didn?t help. >>>>>>>>> I can set it to the default value. >>>>>>>>> >>>>>>>>> Can I proceed with the change? >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> Mauro >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 28 Feb 2019, at 04:41, Nithya Balachandran > wrote: >>>>>>>>>> >>>>>>>>>> Hi Mauro, >>>>>>>>>> >>>>>>>>>> Is network.ping-timeout still set to 0. The default value is 42. Is there a particular reason why this was changed? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Nithya >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, 27 Feb 2019 at 21:32, Mauro Tridici > wrote: >>>>>>>>>> >>>>>>>>>> Hi Xavi, >>>>>>>>>> >>>>>>>>>> thank you for the detailed explanation and suggestions. >>>>>>>>>> Yes, transport.listen-backlog option is still set to 1024. >>>>>>>>>> >>>>>>>>>> I will check the network and connectivity status using ?ping? and ?telnet? as soon as the errors will come back again. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Mauro >>>>>>>>>> >>>>>>>>>>> Il giorno 27 feb 2019, alle ore 16:42, Xavi Hernandez > ha scritto: >>>>>>>>>>> >>>>>>>>>>> Hi Mauro, >>>>>>>>>>> >>>>>>>>>>> those errors say that the mount point is not connected to some of the bricks while executing operations. I see references to 3rd and 6th bricks of several disperse sets, which seem to map to server s06. For some reason, gluster is having troubles connecting from the client machine to that particular server. At the end of the log I see that after long time a reconnect is done to both of them. However little after, other bricks from the s05 get disconnected and a reconnect times out. >>>>>>>>>>> >>>>>>>>>>> That's really odd. It seems like if server/communication is cut to s06 for some time, then restored, and then the same happens to the next server. >>>>>>>>>>> >>>>>>>>>>> If the servers are really online and it's only a communication issue, it explains why server memory and network has increased: if the problem only exists between the client and servers, any write made by the client will automatically mark the file as damaged, since some of the servers have not been updated. Since self-heal runs from the server nodes, they will probably be correctly connected to all bricks, which allows them to heal the just damaged file, which increases memory and network usage. >>>>>>>>>>> >>>>>>>>>>> I guess you still have transport.listen-backlog set to 1024, right ? >>>>>>>>>>> >>>>>>>>>>> Just to try to identify if the problem really comes from network, can you check if you lose some pings from the client to all of the servers while you are seeing those errors in the log file ? >>>>>>>>>>> >>>>>>>>>>> You can also check if during those errors, you can telnet to the port of the brick from the client. >>>>>>>>>>> >>>>>>>>>>> Xavi >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Feb 26, 2019 at 10:17 AM Mauro Tridici > wrote: >>>>>>>>>>> Hi Nithya, >>>>>>>>>>> >>>>>>>>>>> ?df -h? operation is not still slow, but no users are using the volume, RAM and NETWORK usage is ok on the client node. >>>>>>>>>>> >>>>>>>>>>> I was worried about this kind of warnings/errors: >>>>>>>>>>> >>>>>>>>>>> [2019-02-25 10:59:00.664323] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-6: Executing operation with some subvolumes unavailable (20) >>>>>>>>>>> >>>>>>>>>>> [2019-02-26 03:11:35.212603] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-26 03:10:56.549903 (xid=0x106f1c5) >>>>>>>>>>> >>>>>>>>>>> [2019-02-26 03:13:03.313831] E [socket.c:2376:socket_connect_finish] 0-tier2-client-50: connection to 192.168.0.56:49156 failed (Timeout della connessione); disconnecting socket >>>>>>>>>>> >>>>>>>>>>> It seems that some subvolumes are not available and 192.168.0.56 server (s06) is not reachable. >>>>>>>>>>> But gluster servers are up&running and bricks are ok. >>>>>>>>>>> >>>>>>>>>>> In attachment the updated tier2.log file. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thank you. >>>>>>>>>>> Regards, >>>>>>>>>>> Mauro >>>>>>>>>>> >>>>>>>>>>>> Il giorno 26 feb 2019, alle ore 04:03, Nithya Balachandran > ha scritto: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I see a lot of EC messages in the log but they don't seem very serious. Xavi, can you take a look? >>>>>>>>>>>> >>>>>>>>>>>> The only errors I see are: >>>>>>>>>>>> [2019-02-25 10:58:45.519871] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-25 10:57:47.429969 (xid=0xd26fe7) >>>>>>>>>>>> [2019-02-25 10:58:51.461493] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-41: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-25 10:57:47.499174 (xid=0xf47d6a) >>>>>>>>>>>> [2019-02-25 11:07:57.152874] E [socket.c:2376:socket_connect_finish] 0-tier2-client-70: connection to 192.168.0.55:49163 failed (Timeout della connessione); disconnecting socket >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Is the df -h operation still slow? If yes, can you take a tcpdump of the client while running df -h and send that across? >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Nithya >>>>>>>>>>>> >>>>>>>>>>>> On Mon, 25 Feb 2019 at 17:27, Mauro Tridici > wrote: >>>>>>>>>>>> >>>>>>>>>>>> Sorry, some minutes after my last mail message, I noticed that ?df -h? command hanged for a while before returns the prompt. >>>>>>>>>>>> Yesterday, everything was ok in the gluster client log, but, today, I see a lot of errors (please, take a look to the attached file). >>>>>>>>>>>> >>>>>>>>>>>> On the client node, I detected an important RAM e NETWORK usage. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Do you think that the errors have been caused by the client resources usage? >>>>>>>>>>>> >>>>>>>>>>>> Thank you in advance, >>>>>>>>>>>> Mauro >>>>>>>>>>>> >>>>> >>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rajibcse2k10 at gmail.com Thu Mar 14 16:50:59 2019 From: rajibcse2k10 at gmail.com (Rajib Hossen) Date: Thu, 14 Mar 2019 11:50:59 -0500 Subject: [Gluster-devel] GlusterFS development Environment-Debug Message-ID: Hello All, I am new to development of GlusterFS and have some experience in competitive programming in C. I want to implement systematic code for erasure coding in glusterfs. I would like to setup debugging environment. I can use gdb or any IDE. So far, I tried with CLion and Visual Studio Code but no luck. I also tried gdb. Maybe I am missing something or doing it wrong. Could you please let me know how can I setup my development environment and play with the code? Thank you very much. Sincerely, Md Rajib Hossen PhD Student Department of Computer Science The University of Texas at Arlington -------------- next part -------------- An HTML attachment was scrubbed... URL: From vbellur at redhat.com Mon Mar 18 21:02:00 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Mon, 18 Mar 2019 14:02:00 -0700 Subject: [Gluster-devel] Different glusterfs clients's data not consistent. In-Reply-To: References: Message-ID: On Mon, Mar 18, 2019 at 1:21 PM ?? <994506334 at qq.com> wrote: > Three node? node1? node2, node3 > > Steps: > > 1. gluster volume create volume_test node1:/brick1 > 2. gluster volume set volume_test cluster.server-quorum-ratio 51 > 3. gluster volume set volume_test cluster.server-quorum-type server > 4. On node1, mount -t glusterfs node1:/volume_test /mnt. > 5. On node2, mount -t glusterfs node2:/volume_test /mnt. > 6. On node1, killall glusterd > 7. On node2, gluster volume add-brick volume_test node2:/brick2 > 8. On node2. mkdir /mnt/test > 8. touch /mnt/test/file1 on two nodes. > > On node1, found /brick1/file1. But on node2, also found /brick2/file1. > Can you please check the output of stat on file1 in both the bricks? There's a good likelihood that one of them is a link file [1]. > > I don't want to set cluster.server-quorum-ratio to 100. > Cound you help me to solve this porblem? > If it is a link file, a volume rebalance operation might provide the behavior you are looking for. Regards, Vijay [1] http://pl.atyp.us/hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vbellur at redhat.com Mon Mar 18 21:11:47 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Mon, 18 Mar 2019 14:11:47 -0700 Subject: [Gluster-devel] GlusterFS development Environment-Debug In-Reply-To: References: Message-ID: Hello Rajib, On Mon, Mar 18, 2019 at 1:29 PM Rajib Hossen wrote: > Hello All, > I am new to development of GlusterFS and have some experience in > competitive programming in C. I want to implement systematic code for > erasure coding in glusterfs. I would like to setup debugging environment. I > can use gdb or any IDE. So far, I tried with CLion and Visual Studio Code > but no luck. I also tried gdb. Maybe I am missing something or doing > it wrong. Could you please let me know how can I setup my development > environment and play with the code? Thank you very much. > > Welcome to Gluster development! Can you provide a few more details on the problems you are encountering with gdb? For working with gdb, please ensure that your glusterfs install has been built with symbols. Also, Are you in a position to share your thoughts on implementing systematic erasure coding in GlusterFS? Thanks, Vijay [1] http://pl.atyp.us/hekafs.org/index.php/2011/11/translator-101-lesson-4-debugging-a-translator/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Tue Mar 19 04:19:28 2019 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Tue, 19 Mar 2019 09:49:28 +0530 Subject: [Gluster-devel] requesting review available gluster* plugins in sos Message-ID: is (as might just be widely known) an extensible, portable, support data collection tool primarily aimed at Linux distributions and other UNIX-like operating systems. At present there are 2 plugins and I'd like to request that the maintainers do a quick review that this sufficiently covers topics to help diagnose issues. This is a lead up to requesting more usage of the sos tool to diagnose issues we see reported. From ykaul at redhat.com Tue Mar 19 09:18:12 2019 From: ykaul at redhat.com (Yaniv Kaul) Date: Tue, 19 Mar 2019 11:18:12 +0200 Subject: [Gluster-devel] Coverity scan is back Message-ID: After a long shutdown, the upstream Coverity scan for Gluster is back[1]. We were last time measured (January) @ 61 issues and went up to 93 items. Overall, we are still at a very good 0.16 defect density, but we've regressed a bit. Y. [1] https://scan.coverity.com/projects/gluster-glusterfs?tab=overview -------------- next part -------------- An HTML attachment was scrubbed... URL: From atumball at redhat.com Tue Mar 19 13:10:26 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Tue, 19 Mar 2019 18:40:26 +0530 Subject: [Gluster-devel] [Gluster-users] Proposal to mark few features as Deprecated / SunSet from Version 5.0 In-Reply-To: References: Message-ID: Hi Hans, Thanks for the honest feedback. Appreciate this. On Tue, Mar 19, 2019 at 5:39 PM Hans Henrik Happe wrote: > Hi, > > Looking into something else I fell over this proposal. Being a shop that > are going into "Leaving GlusterFS" mode, I thought I would give my two > cents. > > While being partially an HPC shop with a few Lustre filesystems, we chose > GlusterFS for an archiving solution (2-3 PB), because we could find files > in the underlying ZFS filesystems if GlusterFS went sour. > > We have used the access to the underlying files plenty, because of the > continuous instability of GlusterFS'. Meanwhile, Lustre have been almost > effortless to run and mainly for that reason we are planning to move away > from GlusterFS. > > Reading this proposal kind of underlined that "Leaving GluserFS" is the > right thing to do. While I never understood why GlusterFS has been in > feature crazy mode instead of stabilizing mode, taking away crucial > features I don't get. With RoCE, RDMA is getting mainstream. Quotas are > very useful, even though the current implementation are not perfect. > Tiering also makes so much sense, but, for large files, not on a per-file > level. > > It is a right concern to raise, and removing the existing features is not a good thing most of the times. But, one thing we noticed over the years is, the features which we develop, and not take to completion cause the major heart-burn. People think it is present, and it is already few years since its introduced, but if the developers are not working on it, users would always feel that the product doesn't work, because that one feature didn't work. Other than Quota in the proposal email, for all other features, even though we have *some* users, we are inclined towards deprecating them, considering projects overall goals of stability in the longer run. > To be honest we only use quotas. We got scared of trying out new > performance features that potentially would open up a new back of issues. > > About Quota, we heard enough voices, so we will make sure we keep it. The original email was 'Proposal', and hence these opinions matter for decision. Sorry for being such a buzzkill. I really wanted it to be different. > > We hear you. Please let us know one thing, which were the versions you tried ? We hope in coming months, our recent focus on Stability and Technical debt reduction will help you to re-look at Gluster after sometime. > Cheers, > Hans Henrik > On 19/07/2018 08.56, Amar Tumballi wrote: > > > * Hi all, Over last 12 years of Gluster, we have developed many features, > and continue to support most of it till now. But along the way, we have > figured out better methods of doing things. Also we are not actively > maintaining some of these features. We are now thinking of cleaning up some > of these ?unsupported? features, and mark them as ?SunSet? (i.e., would be > totally taken out of codebase in following releases) in next upcoming > release, v5.0. The release notes will provide options for smoothly > migrating to the supported configurations. If you are using any of these > features, do let us know, so that we can help you with ?migration?.. Also, > we are happy to guide new developers to work on those components which are > not actively being maintained by current set of developers. List of > features hitting sunset: ?cluster/stripe? translator: This translator was > developed very early in the evolution of GlusterFS, and addressed one of > the very common question of Distributed FS, which is ?What happens if one > of my file is bigger than the available brick. Say, I have 2 TB hard drive, > exported in glusterfs, my file is 3 TB?. While it solved the purpose, it > was very hard to handle failure scenarios, and give a real good experience > to our users with this feature. Over the time, Gluster solved the problem > with it?s ?Shard? feature, which solves the problem in much better way, and > provides much better solution with existing well supported stack. Hence the > proposal for Deprecation. If you are using this feature, then do write to > us, as it needs a proper migration from existing volume to a new full > supported volume type before you upgrade. ?storage/bd? translator: This > feature got into the code base 5 years back with this patch > [1]. Plan was to use a block device > directly as a brick, which would help to handle disk-image storage much > easily in glusterfs. As the feature is not getting more contribution, and > we are not seeing any user traction on this, would like to propose for > Deprecation. If you are using the feature, plan to move to a supported > gluster volume configuration, and have your setup ?supported? before > upgrading to your new gluster version. ?RDMA? transport support: Gluster > started supporting RDMA while ib-verbs was still new, and very high-end > infra around that time were using Infiniband. Engineers did work with > Mellanox, and got the technology into GlusterFS for better data migration, > data copy. While current day kernels support very good speed with IPoIB > module itself, and there are no more bandwidth for experts in these area to > maintain the feature, we recommend migrating over to TCP (IP based) network > for your volume. If you are successfully using RDMA transport, do get in > touch with us to prioritize the migration plan for your volume. Plan is to > work on this after the release, so by version 6.0, we will have a cleaner > transport code, which just needs to support one type. ?Tiering? feature > Gluster?s tiering feature which was planned to be providing an option to > keep your ?hot? data in different location than your cold data, so one can > get better performance. While we saw some users for the feature, it needs > much more attention to be completely bug free. At the time, we are not > having any active maintainers for the feature, and hence suggesting to take > it out of the ?supported? tag. If you are willing to take it up, and > maintain it, do let us know, and we are happy to assist you. If you are > already using tiering feature, before upgrading, make sure to do gluster > volume tier detach all the bricks before upgrading to next release. Also, > we recommend you to use features like dmcache on your LVM setup to get best > performance from bricks. ?Quota? This is a call out for ?Quota? feature, to > let you all know that it will be ?no new development? state. While this > feature is ?actively? in use by many people, the challenges we have in > accounting mechanisms involved, has made it hard to achieve good > performance with the feature. Also, the amount of extended attribute > get/set operations while using the feature is not very ideal. Hence we > recommend our users to move towards setting quota on backend bricks > directly (ie, XFS project quota), or to use different volumes for different > directories etc. As the feature wouldn?t be deprecated immediately, the > feature doesn?t need a migration plan when you upgrade to newer version, > but if you are a new user, we wouldn?t recommend setting quota feature. By > the release dates, we will be publishing our best alternatives guide for > gluster?s current quota feature. Note that if you want to contribute to the > feature, we have project quota based issue open > [2] Happy to get > contributions, and help in getting a newer approach to Quota. > ------------------------------ These are our set of initial features which > we propose to take out of ?fully? supported features. While we are in the > process of making the user/developer experience of the project much better > with providing well maintained codebase, we may come up with few more set > of features which we may possibly consider to move out of support, and > hence keep watching this space. [1] - http://review.gluster.org/4809 > [2] - > https://github.com/gluster/glusterfs/issues/184 > Regards, Vijay, Shyam, > Amar * > > > > _______________________________________________ > Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users > > -- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: From atumball at redhat.com Tue Mar 19 13:12:25 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Tue, 19 Mar 2019 18:42:25 +0530 Subject: [Gluster-devel] [Gluster-users] Proposal to mark few features as Deprecated / SunSet from Version 5.0 In-Reply-To: References: Message-ID: Hi Jim, On Tue, Mar 19, 2019 at 6:21 PM Jim Kinney wrote: > > Issues with glusterfs fuse mounts cause issues with python file open for > write. We have to use nfs to avoid this. > > Really want to see better back-end tools to facilitate cleaning up of > glusterfs failures. If system is going to use hard linked ID, need a > mapping of id to file to fix things. That option is now on for all exports. > It should be the default If a host is down and users delete files by the > thousands, gluster _never_ catches up. Finding path names for ids across > even a 40TB mount, much less the 200+TB one, is a slow process. A network > outage of 2 minutes and one system didn't get the call to recursively > delete several dozen directories each with several thousand files. > > Are you talking about some issues in geo-replication module or some other application using native mount? Happy to take the discussion forward about these issues. Are there any bugs open on this? Thanks, Amar > > > nfs > On March 19, 2019 8:09:01 AM EDT, Hans Henrik Happe wrote: >> >> Hi, >> >> Looking into something else I fell over this proposal. Being a shop that >> are going into "Leaving GlusterFS" mode, I thought I would give my two >> cents. >> >> While being partially an HPC shop with a few Lustre filesystems, we >> chose GlusterFS for an archiving solution (2-3 PB), because we could find >> files in the underlying ZFS filesystems if GlusterFS went sour. >> >> We have used the access to the underlying files plenty, because of the >> continuous instability of GlusterFS'. Meanwhile, Lustre have been almost >> effortless to run and mainly for that reason we are planning to move away >> from GlusterFS. >> >> Reading this proposal kind of underlined that "Leaving GluserFS" is the >> right thing to do. While I never understood why GlusterFS has been in >> feature crazy mode instead of stabilizing mode, taking away crucial >> features I don't get. With RoCE, RDMA is getting mainstream. Quotas are >> very useful, even though the current implementation are not perfect. >> Tiering also makes so much sense, but, for large files, not on a per-file >> level. >> >> To be honest we only use quotas. We got scared of trying out new >> performance features that potentially would open up a new back of issues. >> >> Sorry for being such a buzzkill. I really wanted it to be different. >> >> Cheers, >> Hans Henrik >> On 19/07/2018 08.56, Amar Tumballi wrote: >> >> >> * Hi all, Over last 12 years of Gluster, we have developed many features, >> and continue to support most of it till now. But along the way, we have >> figured out better methods of doing things. Also we are not actively >> maintaining some of these features. We are now thinking of cleaning up some >> of these ?unsupported? features, and mark them as ?SunSet? (i.e., would be >> totally taken out of codebase in following releases) in next upcoming >> release, v5.0. The release notes will provide options for smoothly >> migrating to the supported configurations. If you are using any of these >> features, do let us know, so that we can help you with ?migration?.. Also, >> we are happy to guide new developers to work on those components which are >> not actively being maintained by current set of developers. List of >> features hitting sunset: ?cluster/stripe? translator: This translator was >> developed very early in the evolution of GlusterFS, and addressed one of >> the very common question of Distributed FS, which is ?What happens if one >> of my file is bigger than the available brick. Say, I have 2 TB hard drive, >> exported in glusterfs, my file is 3 TB?. While it solved the purpose, it >> was very hard to handle failure scenarios, and give a real good experience >> to our users with this feature. Over the time, Gluster solved the problem >> with it?s ?Shard? feature, which solves the problem in much better way, and >> provides much better solution with existing well supported stack. Hence the >> proposal for Deprecation. If you are using this feature, then do write to >> us, as it needs a proper migration from existing volume to a new full >> supported volume type before you upgrade. ?storage/bd? translator: This >> feature got into the code base 5 years back with this patch >> [1]. Plan was to use a block device >> directly as a brick, which would help to handle disk-image storage much >> easily in glusterfs. As the feature is not getting more contribution, and >> we are not seeing any user traction on this, would like to propose for >> Deprecation. If you are using the feature, plan to move to a supported >> gluster volume configuration, and have your setup ?supported? before >> upgrading to your new gluster version. ?RDMA? transport support: Gluster >> started supporting RDMA while ib-verbs was still new, and very high-end >> infra around that time were using Infiniband. Engineers did work with >> Mellanox, and got the technology into GlusterFS for better data migration, >> data copy. While current day kernels support very good speed with IPoIB >> module itself, and there are no more bandwidth for experts in these area to >> maintain the feature, we recommend migrating over to TCP (IP based) network >> for your volume. If you are successfully using RDMA transport, do get in >> touch with us to prioritize the migration plan for your volume. Plan is to >> work on this after the release, so by version 6.0, we will have a cleaner >> transport code, which just needs to support one type. ?Tiering? feature >> Gluster?s tiering feature which was planned to be providing an option to >> keep your ?hot? data in different location than your cold data, so one can >> get better performance. While we saw some users for the feature, it needs >> much more attention to be completely bug free. At the time, we are not >> having any active maintainers for the feature, and hence suggesting to take >> it out of the ?supported? tag. If you are willing to take it up, and >> maintain it, do let us know, and we are happy to assist you. If you are >> already using tiering feature, before upgrading, make sure to do gluster >> volume tier detach all the bricks before upgrading to next release. Also, >> we recommend you to use features like dmcache on your LVM setup to get best >> performance from bricks. ?Quota? This is a call out for ?Quota? feature, to >> let you all know that it will be ?no new development? state. While this >> feature is ?actively? in use by many people, the challenges we have in >> accounting mechanisms involved, has made it hard to achieve good >> performance with the feature. Also, the amount of extended attribute >> get/set operations while using the feature is not very ideal. Hence we >> recommend our users to move towards setting quota on backend bricks >> directly (ie, XFS project quota), or to use different volumes for different >> directories etc. As the feature wouldn?t be deprecated immediately, the >> feature doesn?t need a migration plan when you upgrade to newer version, >> but if you are a new user, we wouldn?t recommend setting quota feature. By >> the release dates, we will be publishing our best alternatives guide for >> gluster?s current quota feature. Note that if you want to contribute to the >> feature, we have project quota based issue open >> [2] Happy to get >> contributions, and help in getting a newer approach to Quota. >> ------------------------------ These are our set of initial features which >> we propose to take out of ?fully? supported features. While we are in the >> process of making the user/developer experience of the project much better >> with providing well maintained codebase, we may come up with few more set >> of features which we may possibly consider to move out of support, and >> hence keep watching this space. [1] - http://review.gluster.org/4809 >> [2] - >> https://github.com/gluster/glusterfs/issues/184 >> Regards, Vijay, Shyam, >> Amar * >> >> >> >> _______________________________________________ >> Gluster-users mailing listGluster-users at gluster.orghttps://lists.glus >> -- >> Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.ter.org/mailman/listinfo/gluster-users >> >> -- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: From skoduri at redhat.com Tue Mar 19 15:00:23 2019 From: skoduri at redhat.com (Soumya Koduri) Date: Tue, 19 Mar 2019 20:30:23 +0530 Subject: [Gluster-devel] requesting review available gluster* plugins in sos In-Reply-To: References: Message-ID: <9f73a6a1-5736-7414-bc0f-fcbafa03a1e3@redhat.com> On 3/19/19 9:49 AM, Sankarshan Mukhopadhyay wrote: > is (as might just be widely known) > an extensible, portable, support data collection tool primarily aimed > at Linux distributions and other UNIX-like operating systems. > > At present there are 2 plugins > > and > I'd like to request that the maintainers do a quick review that this > sufficiently covers topics to help diagnose issues. There is one plugin available for nfs-ganesha as well - https://github.com/sosreport/sos/blob/master/sos/plugins/nfsganesha.py It needs a minor update. Sent a pull request for the same - https://github.com/sosreport/sos/pull/1593 Kindly review. Thanks, Soumya > > This is a lead up to requesting more usage of the sos tool to diagnose > issues we see reported. > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > From srangana at redhat.com Tue Mar 19 15:11:03 2019 From: srangana at redhat.com (Shyam Ranganathan) Date: Tue, 19 Mar 2019 11:11:03 -0400 Subject: [Gluster-devel] Release 6: Tagged and ready for packaging Message-ID: <924a16b2-8574-0915-b24b-86f88fbbbfa3@redhat.com> Hi, RC1 testing is complete and blockers have been addressed. The release is now tagged for a final round of packaging and package testing before release. Thanks for testing out the RC builds and reporting issues that needed to be addressed. As packaging and final package testing is finishing up, we would be writing the upgrade guide for the release as well, before announcing the release for general consumption. Shyam From vbellur at redhat.com Tue Mar 19 20:59:42 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Tue, 19 Mar 2019 13:59:42 -0700 Subject: [Gluster-devel] [Gluster-users] Proposal to mark few features as Deprecated / SunSet from Version 5.0 In-Reply-To: <2aa722b474de38085772f5513facefa878ff70f3.camel@gmail.com> References: <2aa722b474de38085772f5513facefa878ff70f3.camel@gmail.com> Message-ID: Thank you for the reproducer! Can you please let us know the output of `gluster volume info`? Regards, Vijay On Tue, Mar 19, 2019 at 12:53 PM Jim Kinney wrote: > This python will fail when writing to a file in a glusterfs fuse mounted > directory. > > import mmap > > # write a simple example file > with open("hello.txt", "wb") as f: > f.write("Hello Python!\n") > > with open("hello.txt", "r+b") as f: > # memory-map the file, size 0 means whole file > mm = mmap.mmap(f.fileno(), 0) > # read content via standard file methods > print mm.readline() # prints "Hello Python!" > # read content via slice notation > print mm[:5] # prints "Hello" > # update content using slice notation; > # note that new content must have same size > mm[6:] = " world!\n" > # ... and read again using standard file methods > mm.seek(0) > print mm.readline() # prints "Hello world!" > # close the map > mm.close() > > > > > > > > On Tue, 2019-03-19 at 12:06 -0400, Jim Kinney wrote: > > Native mount issue with multiple clients (centos7 glusterfs 3.12). > > Seems to hit python 2.7 and 3+. User tries to open file(s) for write on > long process and system eventually times out. > > Switching to NFS stops the error. > > No bug notice yet. Too many pans on the fire :-( > > On Tue, 2019-03-19 at 18:42 +0530, Amar Tumballi Suryanarayan wrote: > > Hi Jim, > > On Tue, Mar 19, 2019 at 6:21 PM Jim Kinney wrote: > > > Issues with glusterfs fuse mounts cause issues with python file open for > write. We have to use nfs to avoid this. > > Really want to see better back-end tools to facilitate cleaning up of > glusterfs failures. If system is going to use hard linked ID, need a > mapping of id to file to fix things. That option is now on for all exports. > It should be the default If a host is down and users delete files by the > thousands, gluster _never_ catches up. Finding path names for ids across > even a 40TB mount, much less the 200+TB one, is a slow process. A network > outage of 2 minutes and one system didn't get the call to recursively > delete several dozen directories each with several thousand files. > > > > Are you talking about some issues in geo-replication module or some other > application using native mount? Happy to take the discussion forward about > these issues. > > Are there any bugs open on this? > > Thanks, > Amar > > > > > nfs > On March 19, 2019 8:09:01 AM EDT, Hans Henrik Happe wrote: > > Hi, > > Looking into something else I fell over this proposal. Being a shop that > are going into "Leaving GlusterFS" mode, I thought I would give my two > cents. > > While being partially an HPC shop with a few Lustre filesystems, we chose > GlusterFS for an archiving solution (2-3 PB), because we could find files > in the underlying ZFS filesystems if GlusterFS went sour. > > We have used the access to the underlying files plenty, because of the > continuous instability of GlusterFS'. Meanwhile, Lustre have been almost > effortless to run and mainly for that reason we are planning to move away > from GlusterFS. > > Reading this proposal kind of underlined that "Leaving GluserFS" is the > right thing to do. While I never understood why GlusterFS has been in > feature crazy mode instead of stabilizing mode, taking away crucial > features I don't get. With RoCE, RDMA is getting mainstream. Quotas are > very useful, even though the current implementation are not perfect. > Tiering also makes so much sense, but, for large files, not on a per-file > level. > > To be honest we only use quotas. We got scared of trying out new > performance features that potentially would open up a new back of issues. > > Sorry for being such a buzzkill. I really wanted it to be different. > > Cheers, > Hans Henrik > On 19/07/2018 08.56, Amar Tumballi wrote: > > > * Hi all, Over last 12 years of Gluster, we have developed many features, > and continue to support most of it till now. But along the way, we have > figured out better methods of doing things. Also we are not actively > maintaining some of these features. We are now thinking of cleaning up some > of these ?unsupported? features, and mark them as ?SunSet? (i.e., would be > totally taken out of codebase in following releases) in next upcoming > release, v5.0. The release notes will provide options for smoothly > migrating to the supported configurations. If you are using any of these > features, do let us know, so that we can help you with ?migration?.. Also, > we are happy to guide new developers to work on those components which are > not actively being maintained by current set of developers. List of > features hitting sunset: ?cluster/stripe? translator: This translator was > developed very early in the evolution of GlusterFS, and addressed one of > the very common question of Distributed FS, which is ?What happens if one > of my file is bigger than the available brick. Say, I have 2 TB hard drive, > exported in glusterfs, my file is 3 TB?. While it solved the purpose, it > was very hard to handle failure scenarios, and give a real good experience > to our users with this feature. Over the time, Gluster solved the problem > with it?s ?Shard? feature, which solves the problem in much better way, and > provides much better solution with existing well supported stack. Hence the > proposal for Deprecation. If you are using this feature, then do write to > us, as it needs a proper migration from existing volume to a new full > supported volume type before you upgrade. ?storage/bd? translator: This > feature got into the code base 5 years back with this patch > [1]. Plan was to use a block device > directly as a brick, which would help to handle disk-image storage much > easily in glusterfs. As the feature is not getting more contribution, and > we are not seeing any user traction on this, would like to propose for > Deprecation. If you are using the feature, plan to move to a supported > gluster volume configuration, and have your setup ?supported? before > upgrading to your new gluster version. ?RDMA? transport support: Gluster > started supporting RDMA while ib-verbs was still new, and very high-end > infra around that time were using Infiniband. Engineers did work with > Mellanox, and got the technology into GlusterFS for better data migration, > data copy. While current day kernels support very good speed with IPoIB > module itself, and there are no more bandwidth for experts in these area to > maintain the feature, we recommend migrating over to TCP (IP based) network > for your volume. If you are successfully using RDMA transport, do get in > touch with us to prioritize the migration plan for your volume. Plan is to > work on this after the release, so by version 6.0, we will have a cleaner > transport code, which just needs to support one type. ?Tiering? feature > Gluster?s tiering feature which was planned to be providing an option to > keep your ?hot? data in different location than your cold data, so one can > get better performance. While we saw some users for the feature, it needs > much more attention to be completely bug free. At the time, we are not > having any active maintainers for the feature, and hence suggesting to take > it out of the ?supported? tag. If you are willing to take it up, and > maintain it, do let us know, and we are happy to assist you. If you are > already using tiering feature, before upgrading, make sure to do gluster > volume tier detach all the bricks before upgrading to next release. Also, > we recommend you to use features like dmcache on your LVM setup to get best > performance from bricks. ?Quota? This is a call out for ?Quota? feature, to > let you all know that it will be ?no new development? state. While this > feature is ?actively? in use by many people, the challenges we have in > accounting mechanisms involved, has made it hard to achieve good > performance with the feature. Also, the amount of extended attribute > get/set operations while using the feature is not very ideal. Hence we > recommend our users to move towards setting quota on backend bricks > directly (ie, XFS project quota), or to use different volumes for different > directories etc. As the feature wouldn?t be deprecated immediately, the > feature doesn?t need a migration plan when you upgrade to newer version, > but if you are a new user, we wouldn?t recommend setting quota feature. By > the release dates, we will be publishing our best alternatives guide for > gluster?s current quota feature. Note that if you want to contribute to the > feature, we have project quota based issue open > [2] Happy to get > contributions, and help in getting a newer approach to Quota. > ------------------------------ These are our set of initial features which > we propose to take out of ?fully? supported features. While we are in the > process of making the user/developer experience of the project much better > with providing well maintained codebase, we may come up with few more set > of features which we may possibly consider to move out of support, and > hence keep watching this space. [1] - http://review.gluster.org/4809 > [2] - > https://github.com/gluster/glusterfs/issues/184 > Regards, Vijay, Shyam, > Amar * > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > > https://lists.glus > > > -- > > > Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.ter.org/mailman/listinfo/gluster-users > > > > > > -- > > > James P. Kinney III > > > Every time you stop a school, you will have to build a jail. What you > > gain at one end you lose at the other. It's like feeding a dog on his > > own tail. It won't fatten the dog. > > - Speech 11/23/1900 Mark Twain > > > http://heretothereideas.blogspot.com/ > > -- > > James P. Kinney III Every time you stop a school, you will have to build a > jail. What you gain at one end you lose at the other. It's like feeding a > dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark > Twain http://heretothereideas.blogspot.com/ > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Wed Mar 20 01:54:42 2019 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Wed, 20 Mar 2019 07:24:42 +0530 Subject: [Gluster-devel] requesting review available gluster* plugins in sos In-Reply-To: <9f73a6a1-5736-7414-bc0f-fcbafa03a1e3@redhat.com> References: <9f73a6a1-5736-7414-bc0f-fcbafa03a1e3@redhat.com> Message-ID: On Tue, Mar 19, 2019 at 8:30 PM Soumya Koduri wrote: > On 3/19/19 9:49 AM, Sankarshan Mukhopadhyay wrote: > > is (as might just be widely known) > > an extensible, portable, support data collection tool primarily aimed > > at Linux distributions and other UNIX-like operating systems. > > > > At present there are 2 plugins > > > > and > > I'd like to request that the maintainers do a quick review that this > > sufficiently covers topics to help diagnose issues. > > There is one plugin available for nfs-ganesha as well - > https://github.com/sosreport/sos/blob/master/sos/plugins/nfsganesha.py > > It needs a minor update. Sent a pull request for the same - > https://github.com/sosreport/sos/pull/1593 > Thanks Soumya! Other Gluster maintainers - review and respond please. > Kindly review. From sankarshan.mukhopadhyay at gmail.com Wed Mar 20 01:57:47 2019 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Wed, 20 Mar 2019 07:27:47 +0530 Subject: [Gluster-devel] Coverity scan is back In-Reply-To: References: Message-ID: On Tue, Mar 19, 2019 at 2:48 PM Yaniv Kaul wrote: > > After a long shutdown, the upstream Coverity scan for Gluster is back[1]. There hasn't been much by the way of explaining why the service was unavailable for a reasonably long period, has there? Thanks for the update back to the list! > We were last time measured (January) @ 61 issues and went up to 93 items. > Overall, we are still at a very good 0.16 defect density, but we've regressed a bit. Probably something the maintainers can review based on the Impact rating/score and have bite sized tasks. > [1] https://scan.coverity.com/projects/gluster-glusterfs?tab=overview From vbellur at redhat.com Wed Mar 20 04:07:46 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Tue, 19 Mar 2019 21:07:46 -0700 Subject: [Gluster-devel] [Gluster-users] Proposal to mark few features as Deprecated / SunSet from Version 5.0 In-Reply-To: References: <2aa722b474de38085772f5513facefa878ff70f3.camel@gmail.com> Message-ID: I tried this configuration on my local setup and the test passed fine. Adding the fuse and write-behind maintainers in Gluster to check if they are aware of any oddities with using mmap & fuse. Thanks, Vijay On Tue, Mar 19, 2019 at 2:21 PM Jim Kinney wrote: > Volume Name: home > Type: Replicate > Volume ID: 5367adb1-99fc-44c3-98c4-71f7a41e628a > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp,rdma > Bricks: > Brick1: bmidata1:/data/glusterfs/home/brick/brick > Brick2: bmidata2:/data/glusterfs/home/brick/brick > Options Reconfigured: > performance.client-io-threads: off > storage.build-pgfid: on > cluster.self-heal-daemon: enable > performance.readdir-ahead: off > nfs.disable: off > > > There are 11 other volumes and all are similar. > > > On Tue, 2019-03-19 at 13:59 -0700, Vijay Bellur wrote: > > Thank you for the reproducer! Can you please let us know the output of > `gluster volume info`? > > Regards, > Vijay > > On Tue, Mar 19, 2019 at 12:53 PM Jim Kinney wrote: > > This python will fail when writing to a file in a glusterfs fuse mounted > directory. > > import mmap > > # write a simple example file > with open("hello.txt", "wb") as f: > f.write("Hello Python!\n") > > with open("hello.txt", "r+b") as f: > # memory-map the file, size 0 means whole file > mm = mmap.mmap(f.fileno(), 0) > # read content via standard file methods > print mm.readline() # prints "Hello Python!" > # read content via slice notation > print mm[:5] # prints "Hello" > # update content using slice notation; > # note that new content must have same size > mm[6:] = " world!\n" > # ... and read again using standard file methods > mm.seek(0) > print mm.readline() # prints "Hello world!" > # close the map > mm.close() > > > > > > > > On Tue, 2019-03-19 at 12:06 -0400, Jim Kinney wrote: > > Native mount issue with multiple clients (centos7 glusterfs 3.12). > > Seems to hit python 2.7 and 3+. User tries to open file(s) for write on > long process and system eventually times out. > > Switching to NFS stops the error. > > No bug notice yet. Too many pans on the fire :-( > > On Tue, 2019-03-19 at 18:42 +0530, Amar Tumballi Suryanarayan wrote: > > Hi Jim, > > On Tue, Mar 19, 2019 at 6:21 PM Jim Kinney wrote: > > > Issues with glusterfs fuse mounts cause issues with python file open for > write. We have to use nfs to avoid this. > > Really want to see better back-end tools to facilitate cleaning up of > glusterfs failures. If system is going to use hard linked ID, need a > mapping of id to file to fix things. That option is now on for all exports. > It should be the default If a host is down and users delete files by the > thousands, gluster _never_ catches up. Finding path names for ids across > even a 40TB mount, much less the 200+TB one, is a slow process. A network > outage of 2 minutes and one system didn't get the call to recursively > delete several dozen directories each with several thousand files. > > > > Are you talking about some issues in geo-replication module or some other > application using native mount? Happy to take the discussion forward about > these issues. > > Are there any bugs open on this? > > Thanks, > Amar > > > > > nfs > On March 19, 2019 8:09:01 AM EDT, Hans Henrik Happe wrote: > > Hi, > > Looking into something else I fell over this proposal. Being a shop that > are going into "Leaving GlusterFS" mode, I thought I would give my two > cents. > > While being partially an HPC shop with a few Lustre filesystems, we chose > GlusterFS for an archiving solution (2-3 PB), because we could find files > in the underlying ZFS filesystems if GlusterFS went sour. > > We have used the access to the underlying files plenty, because of the > continuous instability of GlusterFS'. Meanwhile, Lustre have been almost > effortless to run and mainly for that reason we are planning to move away > from GlusterFS. > > Reading this proposal kind of underlined that "Leaving GluserFS" is the > right thing to do. While I never understood why GlusterFS has been in > feature crazy mode instead of stabilizing mode, taking away crucial > features I don't get. With RoCE, RDMA is getting mainstream. Quotas are > very useful, even though the current implementation are not perfect. > Tiering also makes so much sense, but, for large files, not on a per-file > level. > > To be honest we only use quotas. We got scared of trying out new > performance features that potentially would open up a new back of issues. > > Sorry for being such a buzzkill. I really wanted it to be different. > > Cheers, > Hans Henrik > On 19/07/2018 08.56, Amar Tumballi wrote: > > > * Hi all, Over last 12 years of Gluster, we have developed many features, > and continue to support most of it till now. But along the way, we have > figured out better methods of doing things. Also we are not actively > maintaining some of these features. We are now thinking of cleaning up some > of these ?unsupported? features, and mark them as ?SunSet? (i.e., would be > totally taken out of codebase in following releases) in next upcoming > release, v5.0. The release notes will provide options for smoothly > migrating to the supported configurations. If you are using any of these > features, do let us know, so that we can help you with ?migration?.. Also, > we are happy to guide new developers to work on those components which are > not actively being maintained by current set of developers. List of > features hitting sunset: ?cluster/stripe? translator: This translator was > developed very early in the evolution of GlusterFS, and addressed one of > the very common question of Distributed FS, which is ?What happens if one > of my file is bigger than the available brick. Say, I have 2 TB hard drive, > exported in glusterfs, my file is 3 TB?. While it solved the purpose, it > was very hard to handle failure scenarios, and give a real good experience > to our users with this feature. Over the time, Gluster solved the problem > with it?s ?Shard? feature, which solves the problem in much better way, and > provides much better solution with existing well supported stack. Hence the > proposal for Deprecation. If you are using this feature, then do write to > us, as it needs a proper migration from existing volume to a new full > supported volume type before you upgrade. ?storage/bd? translator: This > feature got into the code base 5 years back with this patch > [1]. Plan was to use a block device > directly as a brick, which would help to handle disk-image storage much > easily in glusterfs. As the feature is not getting more contribution, and > we are not seeing any user traction on this, would like to propose for > Deprecation. If you are using the feature, plan to move to a supported > gluster volume configuration, and have your setup ?supported? before > upgrading to your new gluster version. ?RDMA? transport support: Gluster > started supporting RDMA while ib-verbs was still new, and very high-end > infra around that time were using Infiniband. Engineers did work with > Mellanox, and got the technology into GlusterFS for better data migration, > data copy. While current day kernels support very good speed with IPoIB > module itself, and there are no more bandwidth for experts in these area to > maintain the feature, we recommend migrating over to TCP (IP based) network > for your volume. If you are successfully using RDMA transport, do get in > touch with us to prioritize the migration plan for your volume. Plan is to > work on this after the release, so by version 6.0, we will have a cleaner > transport code, which just needs to support one type. ?Tiering? feature > Gluster?s tiering feature which was planned to be providing an option to > keep your ?hot? data in different location than your cold data, so one can > get better performance. While we saw some users for the feature, it needs > much more attention to be completely bug free. At the time, we are not > having any active maintainers for the feature, and hence suggesting to take > it out of the ?supported? tag. If you are willing to take it up, and > maintain it, do let us know, and we are happy to assist you. If you are > already using tiering feature, before upgrading, make sure to do gluster > volume tier detach all the bricks before upgrading to next release. Also, > we recommend you to use features like dmcache on your LVM setup to get best > performance from bricks. ?Quota? This is a call out for ?Quota? feature, to > let you all know that it will be ?no new development? state. While this > feature is ?actively? in use by many people, the challenges we have in > accounting mechanisms involved, has made it hard to achieve good > performance with the feature. Also, the amount of extended attribute > get/set operations while using the feature is not very ideal. Hence we > recommend our users to move towards setting quota on backend bricks > directly (ie, XFS project quota), or to use different volumes for different > directories etc. As the feature wouldn?t be deprecated immediately, the > feature doesn?t need a migration plan when you upgrade to newer version, > but if you are a new user, we wouldn?t recommend setting quota feature. By > the release dates, we will be publishing our best alternatives guide for > gluster?s current quota feature. Note that if you want to contribute to the > feature, we have project quota based issue open > [2] Happy to get > contributions, and help in getting a newer approach to Quota. > ------------------------------ These are our set of initial features which > we propose to take out of ?fully? supported features. While we are in the > process of making the user/developer experience of the project much better > with providing well maintained codebase, we may come up with few more set > of features which we may possibly consider to move out of support, and > hence keep watching this space. [1] - http://review.gluster.org/4809 > [2] - > https://github.com/gluster/glusterfs/issues/184 > Regards, Vijay, Shyam, > Amar * > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > > https://lists.glus > > > -- > > > Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.ter.org/mailman/listinfo/gluster-users > > > > > > -- > > > James P. Kinney III > > > Every time you stop a school, you will have to build a jail. What you > > gain at one end you lose at the other. It's like feeding a dog on his > > own tail. It won't fatten the dog. > > - Speech 11/23/1900 Mark Twain > > > http://heretothereideas.blogspot.com/ > > -- > > > James P. Kinney III > > > Every time you stop a school, you will have to build a jail. What you > > gain at one end you lose at the other. It's like feeding a dog on his > > own tail. It won't fatten the dog. > > - Speech 11/23/1900 Mark Twain > > > http://heretothereideas.blogspot.com/ > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > -- > > James P. Kinney III Every time you stop a school, you will have to build a > jail. What you gain at one end you lose at the other. It's like feeding a > dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark > Twain http://heretothereideas.blogspot.com/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amukherj at redhat.com Wed Mar 20 04:30:20 2019 From: amukherj at redhat.com (Atin Mukherjee) Date: Wed, 20 Mar 2019 10:00:20 +0530 Subject: [Gluster-devel] requesting review available gluster* plugins in sos In-Reply-To: References: <9f73a6a1-5736-7414-bc0f-fcbafa03a1e3@redhat.com> Message-ID: >From glusterd perspective couple of enhancements I'd propose to be added (a) to capture get-state dump and make it part of sosreport . Off late, we have seen get-state dump has been very helpful in debugging few cases apart from it's original purpose of providing source of cluster/volume information for tendrl (b) capture glusterd statedump Sanju - if you want to volunteer for sending this enhancement, please grab it :-) . Note that this both the dumps are generated in /var/run/gluster, so please check if we capture /var/run/gluster content in sosreport (which I doubt) and if not you can always have the dump captured in a specific file as you are already aware of. On Wed, Mar 20, 2019 at 7:25 AM Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On Tue, Mar 19, 2019 at 8:30 PM Soumya Koduri wrote: > > On 3/19/19 9:49 AM, Sankarshan Mukhopadhyay wrote: > > > is (as might just be widely known) > > > an extensible, portable, support data collection tool primarily aimed > > > at Linux distributions and other UNIX-like operating systems. > > > > > > At present there are 2 plugins > > > > > > and < > https://github.com/sosreport/sos/blob/master/sos/plugins/gluster_block.py> > > > I'd like to request that the maintainers do a quick review that this > > > sufficiently covers topics to help diagnose issues. > > > > There is one plugin available for nfs-ganesha as well - > > https://github.com/sosreport/sos/blob/master/sos/plugins/nfsganesha.py > > > > It needs a minor update. Sent a pull request for the same - > > https://github.com/sosreport/sos/pull/1593 > > > > Thanks Soumya! > > Other Gluster maintainers - review and respond please. > > > Kindly review. > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiubli at redhat.com Thu Mar 21 03:29:30 2019 From: xiubli at redhat.com (Xiubo Li) Date: Thu, 21 Mar 2019 11:29:30 +0800 Subject: [Gluster-devel] Network Block device (NBD) on top of glusterfs Message-ID: All, I am one of the contributor forgluster-block [1] project, and also I contribute to linux kernel andopen-iscsi project.[2] NBD was around for some time, but in recent time, linux kernel?s Network Block Device (NBD) is enhanced and made to work with more devices and also the option to integrate with netlink is added. So, I tried to provide a glusterfs client based NBD driver recently. Please refergithub issue #633 [3], and good news is I have a working code, with most basic things @nbd-runner project [4]. While this email is about announcing the project, and asking for more collaboration, I would also like to discuss more about the placement of the project itself. Currently nbd-runner project is expected to be shared by our friends at Ceph project too, to provide NBD driver for Ceph. I have personally worked with some of them closely while contributing to open-iSCSI project, and we would like to take this project to great success. Now few questions: 1. Can I continue to usehttp://github.com/gluster/nbd-runneras home for this project, even if its shared by other filesystem projects? * I personally am fine with this. 2. Should there be a separate organization for this repo? * While it may make sense in future, for now, I am not planning to start any new thing? It would be great if we have some consensus on this soon as nbd-runner is a new repository. If there are no concerns, I will continue to contribute to the existing repository. Regards, Xiubo Li (@lxbsz) [1] -https://github.com/gluster/gluster-block [2] -https://github.com/open-iscsi [3] -https://github.com/gluster/glusterfs/issues/633 [4] -https://github.com/gluster/nbd-runner -------------- next part -------------- An HTML attachment was scrubbed... URL: From kinglongmee at gmail.com Thu Mar 21 03:43:21 2019 From: kinglongmee at gmail.com (Kinglong Mee) Date: Thu, 21 Mar 2019 11:43:21 +0800 Subject: [Gluster-devel] The state of lock heal and inodelk/entrylk heal ? Message-ID: <1c9d0e12-0f01-1b75-c7fb-be6659c0b168@gmail.com> Hello folks, Lock self healing (recovery or replay) is added at https://review.gluster.org/#/c/glusterfs/+/2766/ But it is removed at https://review.gluster.org/#/c/glusterfs/+/12363/ I found some information about it at https://anoopcs.fedorapeople.org/Lock%20recovery%20in%20GlusterFS.txt I download the glusterfs source but cannot find any code about lock heal. I wanna know the state of lock heal, and inodelk/entrylk heal. Can someone show me some information about it? thanks, Kinglong Mee From pkarampu at redhat.com Thu Mar 21 06:20:56 2019 From: pkarampu at redhat.com (Pranith Kumar Karampuri) Date: Thu, 21 Mar 2019 11:50:56 +0530 Subject: [Gluster-devel] The state of lock heal and inodelk/entrylk heal ? In-Reply-To: <1c9d0e12-0f01-1b75-c7fb-be6659c0b168@gmail.com> References: <1c9d0e12-0f01-1b75-c7fb-be6659c0b168@gmail.com> Message-ID: On Thu, Mar 21, 2019 at 9:15 AM Kinglong Mee wrote: > Hello folks, > > Lock self healing (recovery or replay) is added at > https://review.gluster.org/#/c/glusterfs/+/2766/ > > But it is removed at > https://review.gluster.org/#/c/glusterfs/+/12363/ > > I found some information about it at > https://anoopcs.fedorapeople.org/Lock%20recovery%20in%20GlusterFS.txt > > I download the glusterfs source but cannot find any code about lock heal. > > I wanna know the state of lock heal, and inodelk/entrylk heal. > hi, At the moment lock heal doesn't happen. It is an open item that needs to be fixed. It is a problem that is gaining interest recently and we are thinking of solving this problem. Did you get a chance to think about this problem and do you have any solutions? > > Can someone show me some information about it? > > thanks, > Kinglong Mee > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > -- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: From pkarampu at redhat.com Thu Mar 21 06:59:33 2019 From: pkarampu at redhat.com (Pranith Kumar Karampuri) Date: Thu, 21 Mar 2019 12:29:33 +0530 Subject: [Gluster-devel] The state of lock heal and inodelk/entrylk heal ? In-Reply-To: References: <1c9d0e12-0f01-1b75-c7fb-be6659c0b168@gmail.com> Message-ID: On Thu, Mar 21, 2019 at 11:50 AM Pranith Kumar Karampuri < pkarampu at redhat.com> wrote: > > > On Thu, Mar 21, 2019 at 9:15 AM Kinglong Mee > wrote: > >> Hello folks, >> >> Lock self healing (recovery or replay) is added at >> https://review.gluster.org/#/c/glusterfs/+/2766/ >> >> But it is removed at >> https://review.gluster.org/#/c/glusterfs/+/12363/ >> >> I found some information about it at >> https://anoopcs.fedorapeople.org/Lock%20recovery%20in%20GlusterFS.txt >> >> I download the glusterfs source but cannot find any code about lock heal. >> >> I wanna know the state of lock heal, and inodelk/entrylk heal. >> > > hi, > At the moment lock heal doesn't happen. It is an open item that needs > to be fixed. It is a problem that is gaining interest recently and we are > thinking of solving this problem. Did you get a chance to think about this > problem and do you have any solutions? > I saw your question first @ https://review.gluster.org/#/c/glusterfs/+/22377/, let us continue the conversation here so that everyone can get involved :-). > > >> >> Can someone show me some information about it? >> >> thanks, >> Kinglong Mee >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-devel >> > > > -- > Pranith > -- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: From kinglongmee at gmail.com Thu Mar 21 07:12:48 2019 From: kinglongmee at gmail.com (Kinglong Mee) Date: Thu, 21 Mar 2019 15:12:48 +0800 Subject: [Gluster-devel] The state of lock heal and inodelk/entrylk heal ? In-Reply-To: References: <1c9d0e12-0f01-1b75-c7fb-be6659c0b168@gmail.com> Message-ID: <30445c31-4f13-1a5c-50e4-b100d88398ce@gmail.com> On 2019/3/21 14:59, Pranith Kumar Karampuri wrote:> On Thu, Mar 21, 2019 at 11:50 AM Pranith Kumar Karampuri > > wrote: > On Thu, Mar 21, 2019 at 9:15 AM Kinglong Mee > wrote: > > Hello folks, > > Lock self healing (recovery or replay) is added at > https://review.gluster.org/#/c/glusterfs/+/2766/ > > But it is removed at > https://review.gluster.org/#/c/glusterfs/+/12363/ > > I found some information about it at > https://anoopcs.fedorapeople.org/Lock%20recovery%20in%20GlusterFS.txt > > I download the glusterfs source but cannot find any code about > lock heal. > > I wanna know the state of lock heal, and inodelk/entrylk heal. > > > hi, > ???? At the moment lock heal doesn't happen. It is an open item > that needs to be fixed. It is a problem that is gaining interest > recently and we are thinking of solving this problem. Did you get a > chance to think about this problem and do you have any solutions? I'd like the mechanism NLM/NFS used lock recovery which is used years. Thanks, Kinglong Mee > > > I saw your question first @ > https://review.gluster.org/#/c/glusterfs/+/22377/, let us continue the > conversation here so that everyone can get involved :-). > > > Can someone show me some information about it? > > thanks, > Kinglong Mee > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > > > > -- > Pranith > > > > -- > Pranith From pkalever at redhat.com Thu Mar 21 10:09:06 2019 From: pkalever at redhat.com (Prasanna Kalever) Date: Thu, 21 Mar 2019 15:39:06 +0530 Subject: [Gluster-devel] Network Block device (NBD) on top of glusterfs In-Reply-To: References: Message-ID: On Thu, Mar 21, 2019 at 9:00 AM Xiubo Li wrote: > All, > > I am one of the contributor for gluster-block > [1] project, and also I > contribute to linux kernel and open-iscsi > project.[2] > > NBD was around for some time, but in recent time, linux kernel?s Network > Block Device (NBD) is enhanced and made to work with more devices and also > the option to integrate with netlink is added. So, I tried to provide a > glusterfs client based NBD driver recently. Please refer github issue #633 > [3], and good news is I > have a working code, with most basic things @ nbd-runner project > [4]. > > While this email is about announcing the project, and asking for more > collaboration, I would also like to discuss more about the placement of the > project itself. Currently nbd-runner project is expected to be shared by > our friends at Ceph project too, to provide NBD driver for Ceph. I have > personally worked with some of them closely while contributing to > open-iSCSI project, and we would like to take this project to great success. > > Now few questions: > > 1. Can I continue to use http://github.com/gluster/nbd-runner as home > for this project, even if its shared by other filesystem projects? > > > - I personally am fine with this. > > > 1. Should there be a separate organization for this repo? > > > - While it may make sense in future, for now, I am not planning to > start any new thing? > > It would be great if we have some consensus on this soon as nbd-runner is > a new repository. If there are no concerns, I will continue to contribute > to the existing repository. > Thanks Xiubo Li, for finally sending this email out. Since this email is out on gluster mailing list, I would like to take a stand from gluster community point of view *only* and share my views. My honest answer is "If we want to maintain this within gluster org, then 80% of the effort is common/duplicate of what we did all these days with gluster-block", like: * rpc/socket code * cli/daemon parser/helper logics * gfapi util functions * logger framework * inotify & dyn-config threads * configure/Makefile/specfiles * docsAboutGluster and etc .. The repository gluster-block is actually a home for all the block related stuff within gluster and its designed to accommodate alike functionalities, if I was you I would have simply copied nbd-runner.c into https://github.com/gluster/gluster-block/tree/master/daemon/ just like ceph plays it here https://github.com/ceph/ceph/blob/master/src/tools/rbd_nbd/rbd-nbd.cc and be done. Advantages of keeping nbd client within gluster-block: -> No worry about maintenance code burdon -> No worry about monitoring a new component -> shipping packages to fedora/centos/rhel is handled -> This helps improve and stabilize the current gluster-block framework -> We can build a common CI -> We can use reuse common test framework and etc .. If you have an impression that gluster-block is for management, then I would really want to correct you at this point. Some of my near future plans for gluster-block: * Allow exporting blocks with FUSE access via fileIO backstore to improve large-file workloads, draft: https://github.com/gluster/gluster-block/pull/58 * Accommodate kernel loopback handling for local only applications * The same way we can accommodate nbd app/client, and IMHO this effort shouldn't take 1 or 2 days to get it merged with in gluster-block and ready for a go release. Hope that clarifies it. Best Regards, -- Prasanna > Regards, > Xiubo Li (@lxbsz) > > [1] - https://github.com/gluster/gluster-block > [2] - https://github.com/open-iscsi > [3] - https://github.com/gluster/glusterfs/issues/633 > [4] - https://github.com/gluster/nbd-runner > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From amukherj at redhat.com Thu Mar 21 10:45:33 2019 From: amukherj at redhat.com (Atin Mukherjee) Date: Thu, 21 Mar 2019 16:15:33 +0530 Subject: [Gluster-devel] GF_CALLOC to GF_MALLOC conversion - is it safe? Message-ID: All, In the last few releases of glusterfs, with stability as a primary theme of the releases, there has been lots of changes done on the code optimization with an expectation that such changes will have gluster to provide better performance. While many of these changes do help, but off late we have started seeing some diverse effects of them, one especially being the calloc to malloc conversions. While I do understand that malloc syscall will eliminate the extra memset bottleneck which calloc bears, but with recent kernels having in-built strong compiler optimizations I am not sure whether that makes any significant difference, but as I mentioned earlier certainly if this isn't done carefully it can potentially introduce lot of bugs and I'm writing this email to share one of such experiences. Sanju & I were having troubles for last two days to figure out why https://review.gluster.org/#/c/glusterfs/+/22388/ wasn't working in Sanju's system but it had no problems running the same fix in my gluster containers. After spending a significant amount of time, what we now figured out is that a malloc call [1] (which was a calloc earlier) is the culprit here. As you all can see, in this function we allocate txn_id and copy the event->txn_id into it through gf_uuid_copy () . But when we were debugging this step wise through gdb, txn_id wasn't exactly copied with the exact event->txn_id and it had some junk values which made the glusterd_clear_txn_opinfo to be invoked with a wrong txn_id later on resulting the leaks to remain the same which was the original intention of the fix. This was quite painful to debug and we had to spend some time to figure this out. Considering we have converted many such calls in past, I'd urge that we review all such conversions and see if there're any side effects to it. Otherwise we might end up running into many potential memory related bugs later on. OTOH, going forward I'd request every patch owners/maintainers to pay some special attention to these conversions and see they are really beneficial and error free. IMO, general guideline should be - for bigger buffers, malloc would make better sense but has to be done carefully, for smaller size, we stick to calloc. What do others think about it? [1] https://github.com/gluster/glusterfs/blob/master/xlators/mgmt/glusterd/src/glusterd-op-sm.c#L5681 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Thu Mar 21 10:48:28 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Thu, 21 Mar 2019 16:18:28 +0530 Subject: [Gluster-devel] [Gluster-users] "rpc_clnt_ping_timer_expired" errors In-Reply-To: <93130243-E356-4425-8F15-69BE61562E2F@cmcc.it> References: <96B07283-D8AB-4F06-909D-E00424625528@cmcc.it> <42758A0E-8BE9-497D-BDE3-55D7DC633BC7@cmcc.it> <6A8CF4A4-98EA-48C3-A059-D60D1B2721C7@cmcc.it> <2CF49168-9C1B-4931-8C34-8157262A137A@cmcc.it> <7A151CC9-A0AE-4A45-B450-A4063D216D9E@cmcc.it> <32D53ECE-3F49-4415-A6EE-241B351AC2BA@cmcc.it> <4685A75B-5978-4338-9C9F-4A02FB40B9BC@cmcc.it> <4D2E6B04-C2E8-4FD5-B72D-E7C05931C6F9@cmcc.it> <4E332A56-B318-4BC1-9F44-84AB4392A5DE@cmcc.it> <832FD362-3B14-40D8-8530-604419300476@cmcc.it> <8D926643-1093-48ED-823F-D8F117F702CF@cmcc.it> <9D0BE438-DF11-4D0A-AB85-B44357032F29@cmcc.it> <5F0AC378-8170-4342-8473-9C17159CAC1D@cmcc.it> <7A50E86D-9E27-4EA7-883B-45E9F973991A@cmcc.it> <58B5DB7F-DCB4-4FBF-B1F7-681315B1613A@cmcc.it> <6327B44F-4E7E-46BB-A74C-70F4457DD1EB@cmcc.it> <0167DF4A-8329-4A1A-B439-857DFA6F78BB@cmcc.it> <763F334E-35B8-4729-B8E1-D30866754EEE@cmcc.it> <91DFC9AC-4805-41EB-AC6F-5722E018DD6E@cmcc.it> <8A9752B8-B231-4570-8FF4-8D3D781E7D42@cmcc.it> <47A24804-F975-4EE6-9FA5-67FCDA18D707@cmcc.it> <637FF5D2-D1F4-4686-9D48-646A96F67B96@cmcc.it> <4A87495F-3755-48F7-8507-085869069C64@cmcc.it> <3854BBF6-5B98-4AB3-A67E-E7DE59E69A63@cmcc.it> <313DA021-9173-4899-96B0-831B10B00B61@cmcc.it> <17996AFD-DFC8-40F3-9D09-DB6DDAD5B7D6@cmcc.it> <7074B5D8-955A-4802-A9F3-606C99734417@cmcc.it> <83B84BF9-8334-4230-B6F8-0BC4BFBEFF15@cmcc.it> <133B9AE4-9792-4F72-AD91-D36E7B9EC711@cmcc.it> <6611C4B0-57FD-4390-88B5-BD373555D4C5@cmcc.it> <93130243-E356-4425-8F15-69BE61562E2F@cmcc.it> Message-ID: On Thu, Mar 21, 2019 at 4:10 PM Mauro Tridici wrote: > Hi Raghavendra, > > the number of errors reduced, but during last days I received some error > notifications from Nagios server similar to the following one: > > > > > > > > > > > > > > > ****** Nagios *****Notification Type: PROBLEMService: Brick - > /gluster/mnt5/brickHost: s04Address: s04-stgState: CRITICALDate/Time: Mon > Mar 18 19:56:36 CET 2019Additional Info:CHECK_NRPE STATE CRITICAL: Socket > timeout after 10 seconds.* > > The error was related only to s04 gluster server. > > So, following your suggestions, I executed, on s04 node, the top command. > In attachment, you can find the related output. > top output doesn't contain cmd/thread names. Was there anything wrong. > Thank you very much for your help. > Regards, > Mauro > > > > On 14 Mar 2019, at 13:31, Raghavendra Gowdappa > wrote: > > Thanks Mauro. > > On Thu, Mar 14, 2019 at 3:38 PM Mauro Tridici > wrote: > >> Hi Raghavendra, >> >> I just changed the client option value to 8. >> I will check the volume behaviour during the next hours. >> >> The GlusterFS version is 3.12.14. >> >> I will provide you the logs as soon as the activity load will be high. >> Thank you, >> Mauro >> >> On 14 Mar 2019, at 04:57, Raghavendra Gowdappa >> wrote: >> >> >> >> On Wed, Mar 13, 2019 at 3:55 PM Mauro Tridici >> wrote: >> >>> Hi Raghavendra, >>> >>> Yes, server.event-thread has been changed from 4 to 8. >>> >> >> Was client.event-thread value too changed to 8? If not, I would like to >> know the results of including this tuning too. Also, if possible, can you >> get the output of following command from problematic clients and bricks >> (during the duration when load tends to be high and ping-timer-expiry is >> seen)? >> >> # top -bHd 3 >> >> This will help us to know CPU utilization of event-threads. >> >> And I forgot to ask, what version of Glusterfs are you using? >> >> During last days, I noticed that the error events are still here although >>> they have been considerably reduced. >>> >>> So, I used grep command against the log files in order to provide you a >>> global vision about the warning, error and critical events appeared today >>> at 06:xx (may be useful I hope). >>> I collected the info from s06 gluster server, but the behaviour is the >>> the almost the same on the other gluster servers. >>> >>> *ERRORS: * >>> *CWD: /var/log/glusterfs * >>> *COMMAND: grep " E " *.log |grep "2019-03-13 06:"* >>> >>> (I can see a lot of this kind of message in the same period but I'm >>> notifying you only one record for each type of error) >>> >>> glusterd.log:[2019-03-13 06:12:35.982863] E [MSGID: 101042] >>> [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of >>> /var/run/gluster/tier2_quota_list/ >>> >>> glustershd.log:[2019-03-13 06:14:28.666562] E >>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a71ddcebb] (--> >>> /lib64/libgfr >>> pc.so.0(saved_frames_unwind+0x1de)[0x7f4a71ba1d9e] (--> >>> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f4a71ba1ebe] (--> >>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup >>> +0x90)[0x7f4a71ba3640] (--> >>> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7f4a71ba4130] ))))) >>> 0-tier2-client-55: forced unwinding frame type(GlusterFS 3.3) >>> op(INODELK(29)) >>> called at 2019-03-13 06:14:14.858441 (xid=0x17fddb50) >>> >>> glustershd.log:[2019-03-13 06:17:48.883825] E >>> [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to >>> 192.168.0.55:49158 failed (Connection timed out); disco >>> nnecting socket >>> glustershd.log:[2019-03-13 06:19:58.931798] E >>> [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to >>> 192.168.0.55:49158 failed (Connection timed out); disco >>> nnecting socket >>> glustershd.log:[2019-03-13 06:22:08.979829] E >>> [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to >>> 192.168.0.55:49158 failed (Connection timed out); disco >>> nnecting socket >>> glustershd.log:[2019-03-13 06:22:36.226847] E [MSGID: 114031] >>> [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote >>> operation failed [Transport endpoint >>> is not connected] >>> glustershd.log:[2019-03-13 06:22:36.306669] E [MSGID: 114031] >>> [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote >>> operation failed [Transport endpoint >>> is not connected] >>> glustershd.log:[2019-03-13 06:22:36.385257] E [MSGID: 114031] >>> [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote >>> operation failed [Transport endpoint >>> is not connected] >>> >>> *WARNINGS:* >>> *CWD: /var/log/glusterfs * >>> *COMMAND: grep " W " *.log |grep "2019-03-13 06:"* >>> >>> (I can see a lot of this kind of message in the same period but I'm >>> notifying you only one record for each type of warnings) >>> >>> glustershd.log:[2019-03-13 06:14:28.666772] W [MSGID: 114031] >>> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-tier2-client-55: remote >>> operation failed. Path: >> 0f-f34d-4c25-bbe8-74bde0248d7e> (b6b35d0f-f34d-4c25-bbe8-74bde0248d7e). >>> Key: (null) [Transport endpoint is not connected] >>> >>> glustershd.log:[2019-03-13 06:14:31.421576] W [MSGID: 122035] >>> [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation >>> with some subvolumes unavailable (2) >>> >>> glustershd.log:[2019-03-13 06:15:31.547417] W [MSGID: 122032] >>> [ec-heald.c:266:ec_shd_index_sweep] 0-tier2-disperse-9: unable to get >>> index-dir on tier2-client-55 [Operation >>> now in progress] >>> >>> quota-mount-tier2.log:[2019-03-13 06:12:36.116277] W [MSGID: 101002] >>> [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is >>> deprecated, preferred is 'trans >>> port.address-family', continuing with correction >>> quota-mount-tier2.log:[2019-03-13 06:12:36.198430] W [MSGID: 101174] >>> [graph.c:363:_log_if_unknown_option] 0-tier2-readdir-ahead: option >>> 'parallel-readdir' is not recognized >>> quota-mount-tier2.log:[2019-03-13 06:12:37.945007] W >>> [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) >>> [0x7f340892be25] -->/usr/sbin/glusterfs(gluste >>> rfs_sigwaiter+0xe5) [0x55ef010164b5] >>> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55ef0101632b] ) 0-: >>> received signum (15), shutting down >>> >>> *CRITICALS:* >>> *CWD: /var/log/glusterfs * >>> *COMMAND: grep " C " *.log |grep "2019-03-13 06:"* >>> >>> no critical errors at 06:xx >>> only one critical error during the day >>> >>> *[root at s06 glusterfs]# grep " C " *.log |grep "2019-03-13"* >>> glustershd.log:[2019-03-13 02:21:29.126279] C >>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server >>> 192.168.0.55:49158 has not responded in the last 42 seconds, >>> disconnecting. >>> >>> >>> Thank you very much for your help. >>> Regards, >>> Mauro >>> >>> On 12 Mar 2019, at 05:17, Raghavendra Gowdappa >>> wrote: >>> >>> Was the suggestion to increase server.event-thread values tried? If yes, >>> what were the results? >>> >>> On Mon, Mar 11, 2019 at 2:40 PM Mauro Tridici >>> wrote: >>> >>>> Dear All, >>>> >>>> do you have any suggestions about the right way to "debug" this issue? >>>> In attachment, the updated logs of ?s06" gluster server. >>>> >>>> I noticed a lot of intermittent warning and error messages. >>>> >>>> Thank you in advance, >>>> Mauro >>>> >>>> >>>> >>>> On 4 Mar 2019, at 18:45, Raghavendra Gowdappa >>>> wrote: >>>> >>>> >>>> +Gluster Devel , +Gluster-users >>>> >>>> >>>> I would like to point out another issue. Even if what I suggested >>>> prevents disconnects, part of the solution would be only symptomatic >>>> treatment and doesn't address the root cause of the problem. In most of the >>>> ping-timer-expiry issues, the root cause is the increased load on bricks >>>> and the inability of bricks to be responsive under high load. So, the >>>> actual solution would be doing any or both of the following: >>>> * identify the source of increased load and if possible throttle it. >>>> Internal heal processes like self-heal, rebalance, quota heal are known to >>>> pump traffic into bricks without much throttling (io-threads _might_ do >>>> some throttling, but my understanding is its not sufficient). >>>> * identify the reason for bricks to become unresponsive during load. >>>> This may be fixable issues like not enough event-threads to read from >>>> network or difficult to fix issues like fsync on backend fs freezing the >>>> process or semi fixable issues (in code) like lock contention. >>>> >>>> So any genuine effort to fix ping-timer-issues (to be honest most of >>>> the times they are not issues related to rpc/network) would involve >>>> performance characterization of various subsystems on bricks and clients. >>>> Various subsystems can include (but not necessarily limited to), underlying >>>> OS/filesystem, glusterfs processes, CPU consumption etc >>>> >>>> regards, >>>> Raghavendra >>>> >>>> On Mon, Mar 4, 2019 at 9:31 PM Mauro Tridici >>>> wrote: >>>> >>>>> Thank you, let?s try! >>>>> I will inform you about the effects of the change. >>>>> >>>>> Regards, >>>>> Mauro >>>>> >>>>> On 4 Mar 2019, at 16:55, Raghavendra Gowdappa >>>>> wrote: >>>>> >>>>> >>>>> >>>>> On Mon, Mar 4, 2019 at 8:54 PM Mauro Tridici >>>>> wrote: >>>>> >>>>>> Hi Raghavendra, >>>>>> >>>>>> thank you for your reply. >>>>>> Yes, you are right. It is a problem that seems to happen randomly. >>>>>> At this moment, server.event-threads value is 4. I will try to >>>>>> increase this value to 8. Do you think that it could be a valid value ? >>>>>> >>>>> >>>>> Yes. We can try with that. You should see at least frequency of >>>>> ping-timer related disconnects reduce with this value (even if it doesn't >>>>> eliminate the problem completely). >>>>> >>>>> >>>>>> Regards, >>>>>> Mauro >>>>>> >>>>>> >>>>>> On 4 Mar 2019, at 15:36, Raghavendra Gowdappa >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Mar 4, 2019 at 8:01 PM Nithya Balachandran < >>>>>> nbalacha at redhat.com> wrote: >>>>>> >>>>>>> Hi Mauro, >>>>>>> >>>>>>> It looks like some problem on s06. Are all your other nodes ok? Can >>>>>>> you send us the gluster logs from this node? >>>>>>> >>>>>>> @Raghavendra G , do you have any idea as >>>>>>> to how this can be debugged? Maybe running top ? Or debug brick logs? >>>>>>> >>>>>> >>>>>> If we can reproduce the problem, collecting tcpdump on both ends of >>>>>> connection will help. But, one common problem is these bugs are >>>>>> inconsistently reproducible and hence we may not be able to capture tcpdump >>>>>> at correct intervals. Other than that, we can try to collect some evidence >>>>>> that poller threads were busy (waiting on locks). But, not sure what debug >>>>>> data provides that information. >>>>>> >>>>>> From what I know, its difficult to collect evidence for this issue >>>>>> and we could only reason about it. >>>>>> >>>>>> We can try a workaround though - try increasing server.event-threads >>>>>> and see whether ping-timer expiry issues go away with an optimal value. If >>>>>> that's the case, it kind of provides proof for our hypothesis. >>>>>> >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Nithya >>>>>>> >>>>>>> On Mon, 4 Mar 2019 at 15:25, Mauro Tridici >>>>>>> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> some minutes ago I received this message from NAGIOS server >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ****** Nagios *****Notification Type: PROBLEMService: Brick - >>>>>>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: CRITICALDate/Time: Mon >>>>>>>> Mar 4 10:25:33 CET 2019Additional Info:CHECK_NRPE STATE CRITICAL: Socket >>>>>>>> timeout after 10 seconds.* >>>>>>>> >>>>>>>> I checked the network, RAM and CPUs usage on s06 node and >>>>>>>> everything seems to be ok. >>>>>>>> No bricks are in error state. In /var/log/messages, I detected >>>>>>>> again a crash of ?check_vol_utili? that I think it is a module used by NRPE >>>>>>>> executable (that is the NAGIOS client). >>>>>>>> >>>>>>>> Mar 4 10:15:29 s06 kernel: traps: check_vol_utili[161224] general >>>>>>>> protection ip:7facffa0a66d sp:7ffe9f4e6fc0 error:0 in >>>>>>>> libglusterfs.so.0.0.1[7facff9b7000+f7000] >>>>>>>> Mar 4 10:15:29 s06 abrt-hook-ccpp: Process 161224 (python2.7) of >>>>>>>> user 0 killed by SIGSEGV - dumping core >>>>>>>> Mar 4 10:15:29 s06 abrt-server: Generating core_backtrace >>>>>>>> Mar 4 10:15:29 s06 abrt-server: Error: Unable to open >>>>>>>> './coredump': No such file or directory >>>>>>>> Mar 4 10:16:01 s06 systemd: Created slice User Slice of root. >>>>>>>> Mar 4 10:16:01 s06 systemd: Starting User Slice of root. >>>>>>>> Mar 4 10:16:01 s06 systemd: Started Session 201010 of user root. >>>>>>>> Mar 4 10:16:01 s06 systemd: Starting Session 201010 of user root. >>>>>>>> Mar 4 10:16:01 s06 systemd: Removed slice User Slice of root. >>>>>>>> Mar 4 10:16:01 s06 systemd: Stopping User Slice of root. >>>>>>>> Mar 4 10:16:24 s06 abrt-server: Duplicate: UUID >>>>>>>> Mar 4 10:16:24 s06 abrt-server: DUP_OF_DIR: >>>>>>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>>>>> Mar 4 10:16:24 s06 abrt-server: Deleting problem directory >>>>>>>> ccpp-2019-03-04-10:15:29-161224 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>>>>> Mar 4 10:16:24 s06 abrt-server: Generating core_backtrace >>>>>>>> Mar 4 10:16:24 s06 abrt-server: Error: Unable to open >>>>>>>> './coredump': No such file or directory >>>>>>>> Mar 4 10:16:24 s06 abrt-server: Cannot notify >>>>>>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>>>>>> 'report_uReport' exited with 1 >>>>>>>> Mar 4 10:16:24 s06 abrt-hook-ccpp: Process 161391 (python2.7) of >>>>>>>> user 0 killed by SIGABRT - dumping core >>>>>>>> Mar 4 10:16:25 s06 abrt-server: Generating core_backtrace >>>>>>>> Mar 4 10:16:25 s06 abrt-server: Error: Unable to open >>>>>>>> './coredump': No such file or directory >>>>>>>> Mar 4 10:17:01 s06 systemd: Created slice User Slice of root. >>>>>>>> >>>>>>>> Also, I noticed the following errors that I think are very critical: >>>>>>>> >>>>>>>> Mar 4 10:21:12 s06 glustershd[20355]: [2019-03-04 09:21:12.954798] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: >>>>>>>> server 192.168.0.55:49158 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 4 10:22:01 s06 systemd: Created slice User Slice of root. >>>>>>>> Mar 4 10:22:01 s06 systemd: Starting User Slice of root. >>>>>>>> Mar 4 10:22:01 s06 systemd: Started Session 201017 of user root. >>>>>>>> Mar 4 10:22:01 s06 systemd: Starting Session 201017 of user root. >>>>>>>> Mar 4 10:22:01 s06 systemd: Removed slice User Slice of root. >>>>>>>> Mar 4 10:22:01 s06 systemd: Stopping User Slice of root. >>>>>>>> Mar 4 10:22:03 s06 glustershd[20355]: [2019-03-04 09:22:03.964120] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: >>>>>>>> server 192.168.0.54:49165 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 4 10:23:01 s06 systemd: Created slice User Slice of root. >>>>>>>> Mar 4 10:23:01 s06 systemd: Starting User Slice of root. >>>>>>>> Mar 4 10:23:01 s06 systemd: Started Session 201018 of user root. >>>>>>>> Mar 4 10:23:01 s06 systemd: Starting Session 201018 of user root. >>>>>>>> Mar 4 10:23:02 s06 systemd: Removed slice User Slice of root. >>>>>>>> Mar 4 10:23:02 s06 systemd: Stopping User Slice of root. >>>>>>>> Mar 4 10:24:01 s06 systemd: Created slice User Slice of root. >>>>>>>> Mar 4 10:24:01 s06 systemd: Starting User Slice of root. >>>>>>>> Mar 4 10:24:01 s06 systemd: Started Session 201019 of user root. >>>>>>>> Mar 4 10:24:01 s06 systemd: Starting Session 201019 of user root. >>>>>>>> Mar 4 10:24:01 s06 systemd: Removed slice User Slice of root. >>>>>>>> Mar 4 10:24:01 s06 systemd: Stopping User Slice of root. >>>>>>>> Mar 4 10:24:03 s06 glustershd[20355]: [2019-03-04 09:24:03.982502] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-16: >>>>>>>> server 192.168.0.52:49158 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746109] C >>>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-3: server >>>>>>>> 192.168.0.51:49153 has not responded in the last 42 seconds, >>>>>>>> disconnecting. >>>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746215] C >>>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server >>>>>>>> 192.168.0.52:49156 has not responded in the last 42 seconds, >>>>>>>> disconnecting. >>>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746260] C >>>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-21: server >>>>>>>> 192.168.0.51:49159 has not responded in the last 42 seconds, >>>>>>>> disconnecting. >>>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746296] C >>>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server >>>>>>>> 192.168.0.52:49161 has not responded in the last 42 seconds, >>>>>>>> disconnecting. >>>>>>>> Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746413] C >>>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server >>>>>>>> 192.168.0.54:49165 has not responded in the last 42 seconds, >>>>>>>> disconnecting. >>>>>>>> Mar 4 10:24:07 s06 glustershd[20355]: [2019-03-04 09:24:07.982952] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-45: >>>>>>>> server 192.168.0.54:49155 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 4 10:24:18 s06 glustershd[20355]: [2019-03-04 09:24:18.990929] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: >>>>>>>> server 192.168.0.52:49161 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 4 10:24:31 s06 glustershd[20355]: [2019-03-04 09:24:31.995781] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-20: >>>>>>>> server 192.168.0.53:49159 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 4 10:25:01 s06 systemd: Created slice User Slice of root. >>>>>>>> Mar 4 10:25:01 s06 systemd: Starting User Slice of root. >>>>>>>> Mar 4 10:25:01 s06 systemd: Started Session 201020 of user root. >>>>>>>> Mar 4 10:25:01 s06 systemd: Starting Session 201020 of user root. >>>>>>>> Mar 4 10:25:01 s06 systemd: Removed slice User Slice of root. >>>>>>>> Mar 4 10:25:01 s06 systemd: Stopping User Slice of root. >>>>>>>> Mar 4 10:25:57 s06 systemd: Created slice User Slice of root. >>>>>>>> Mar 4 10:25:57 s06 systemd: Starting User Slice of root. >>>>>>>> Mar 4 10:25:57 s06 systemd-logind: New session 201021 of user root. >>>>>>>> Mar 4 10:25:57 s06 systemd: Started Session 201021 of user root. >>>>>>>> Mar 4 10:25:57 s06 systemd: Starting Session 201021 of user root. >>>>>>>> Mar 4 10:26:01 s06 systemd: Started Session 201022 of user root. >>>>>>>> Mar 4 10:26:01 s06 systemd: Starting Session 201022 of user root. >>>>>>>> Mar 4 10:26:21 s06 nrpe[162388]: Error: Could not complete SSL >>>>>>>> handshake with 192.168.1.56: 5 >>>>>>>> Mar 4 10:27:01 s06 systemd: Started Session 201023 of user root. >>>>>>>> Mar 4 10:27:01 s06 systemd: Starting Session 201023 of user root. >>>>>>>> Mar 4 10:28:01 s06 systemd: Started Session 201024 of user root. >>>>>>>> Mar 4 10:28:01 s06 systemd: Starting Session 201024 of user root. >>>>>>>> Mar 4 10:29:01 s06 systemd: Started Session 201025 of user root. >>>>>>>> Mar 4 10:29:01 s06 systemd: Starting Session 201025 of user root. >>>>>>>> >>>>>>>> But, unfortunately, I don?t understand why it is happening. >>>>>>>> Now, NAGIOS server shows that s06 status is ok: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ****** Nagios *****Notification Type: RECOVERYService: Brick - >>>>>>>> /gluster/mnt2/brickHost: s06Address: s06-stgState: OKDate/Time: Mon Mar 4 >>>>>>>> 10:35:23 CET 2019Additional Info:OK: Brick /gluster/mnt2/brick is up* >>>>>>>> >>>>>>>> Nothing is changed from RAM, CPUs, and NETWORK point of view. >>>>>>>> /var/log/message file has been updated: >>>>>>>> >>>>>>>> Mar 4 10:32:01 s06 systemd: Starting Session 201029 of user root. >>>>>>>> Mar 4 10:32:30 s06 glustershd[20355]: [2019-03-04 09:32:30.069082] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: >>>>>>>> server 192.168.0.52:49156 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 4 10:32:55 s06 glustershd[20355]: [2019-03-04 09:32:55.074689] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-66: >>>>>>>> server 192.168.0.54:49167 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 4 10:33:01 s06 systemd: Started Session 201030 of user root. >>>>>>>> Mar 4 10:33:01 s06 systemd: Starting Session 201030 of user root. >>>>>>>> Mar 4 10:34:01 s06 systemd: Started Session 201031 of user root. >>>>>>>> Mar 4 10:34:01 s06 systemd: Starting Session 201031 of user root. >>>>>>>> Mar 4 10:35:01 s06 nrpe[162562]: Could not read request from >>>>>>>> client 192.168.1.56, bailing out... >>>>>>>> Mar 4 10:35:01 s06 nrpe[162562]: INFO: SSL Socket Shutdown. >>>>>>>> Mar 4 10:35:01 s06 systemd: Started Session 201032 of user root. >>>>>>>> Mar 4 10:35:01 s06 systemd: Starting Session 201032 of user root. >>>>>>>> >>>>>>>> Could you please help me to understand what it?s happening ? >>>>>>>> Thank you in advance. >>>>>>>> >>>>>>>> Rergards, >>>>>>>> Mauro >>>>>>>> >>>>>>>> >>>>>>>> On 1 Mar 2019, at 12:17, Mauro Tridici >>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> Thank you, Milind. >>>>>>>> I executed the instructions you suggested: >>>>>>>> >>>>>>>> - grep ?blocked for? /var/log/messages on s06 returns no output (no >>>>>>>> ?blocked? word is detected in messages file); >>>>>>>> - in /var/log/messages file I can see this kind of error repeated >>>>>>>> for a lot of times: >>>>>>>> >>>>>>>> Mar 1 08:43:01 s06 systemd: Starting Session 196071 of user root. >>>>>>>> Mar 1 08:43:01 s06 systemd: Removed slice User Slice of root. >>>>>>>> Mar 1 08:43:01 s06 systemd: Stopping User Slice of root. >>>>>>>> Mar 1 08:43:02 s06 kernel: traps: check_vol_utili[57091] general >>>>>>>> protection ip:7f88e76ee66d sp:7ffe5a5bcc30 error:0 in >>>>>>>> libglusterfs.so.0.0.1[7f88e769b000+f7000] >>>>>>>> Mar 1 08:43:02 s06 abrt-hook-ccpp: Process 57091 (python2.7) of >>>>>>>> user 0 killed by SIGSEGV - dumping core >>>>>>>> Mar 1 08:43:02 s06 abrt-server: Generating core_backtrace >>>>>>>> Mar 1 08:43:02 s06 abrt-server: Error: Unable to open >>>>>>>> './coredump': No such file or directory >>>>>>>> Mar 1 08:43:58 s06 abrt-server: Duplicate: UUID >>>>>>>> Mar 1 08:43:58 s06 abrt-server: DUP_OF_DIR: >>>>>>>> /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041 >>>>>>>> Mar 1 08:43:58 s06 abrt-server: Deleting problem directory >>>>>>>> ccpp-2019-03-01-08:43:02-57091 (dup of ccpp-2018-09-25-12:27:42-13041) >>>>>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Activating service >>>>>>>> name='org.freedesktop.problems' (using servicehelper) >>>>>>>> Mar 1 08:43:58 s06 dbus[1872]: [system] Successfully activated >>>>>>>> service 'org.freedesktop.problems' >>>>>>>> Mar 1 08:43:58 s06 abrt-server: Generating core_backtrace >>>>>>>> Mar 1 08:43:58 s06 abrt-server: Error: Unable to open >>>>>>>> './coredump': No such file or directory >>>>>>>> Mar 1 08:43:58 s06 abrt-server: Cannot notify >>>>>>>> '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event >>>>>>>> 'report_uReport' exited with 1 >>>>>>>> Mar 1 08:44:01 s06 systemd: Created slice User Slice of root. >>>>>>>> Mar 1 08:44:01 s06 systemd: Starting User Slice of root. >>>>>>>> Mar 1 08:44:01 s06 systemd: Started Session 196072 of user root. >>>>>>>> Mar 1 08:44:01 s06 systemd: Starting Session 196072 of user root. >>>>>>>> Mar 1 08:44:01 s06 systemd: Removed slice User Slice of root. >>>>>>>> >>>>>>>> - in /var/log/messages file I can see also 4 errors related to >>>>>>>> other cluster servers: >>>>>>>> >>>>>>>> Mar 1 11:05:01 s06 systemd: Starting User Slice of root. >>>>>>>> Mar 1 11:05:01 s06 systemd: Started Session 196230 of user root. >>>>>>>> Mar 1 11:05:01 s06 systemd: Starting Session 196230 of user root. >>>>>>>> Mar 1 11:05:01 s06 systemd: Removed slice User Slice of root. >>>>>>>> Mar 1 11:05:01 s06 systemd: Stopping User Slice of root. >>>>>>>> Mar 1 11:05:59 s06 glustershd[70117]: [2019-03-01 10:05:59.347094] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-33: >>>>>>>> server 192.168.0.51:49163 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 1 11:06:01 s06 systemd: Created slice User Slice of root. >>>>>>>> Mar 1 11:06:01 s06 systemd: Starting User Slice of root. >>>>>>>> Mar 1 11:06:01 s06 systemd: Started Session 196231 of user root. >>>>>>>> Mar 1 11:06:01 s06 systemd: Starting Session 196231 of user root. >>>>>>>> Mar 1 11:06:01 s06 systemd: Removed slice User Slice of root. >>>>>>>> Mar 1 11:06:01 s06 systemd: Stopping User Slice of root. >>>>>>>> Mar 1 11:06:12 s06 glustershd[70117]: [2019-03-01 10:06:12.351319] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-1: >>>>>>>> server 192.168.0.52:49153 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 1 11:06:38 s06 glustershd[70117]: [2019-03-01 10:06:38.356920] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-7: >>>>>>>> server 192.168.0.52:49155 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 1 11:07:01 s06 systemd: Created slice User Slice of root. >>>>>>>> Mar 1 11:07:01 s06 systemd: Starting User Slice of root. >>>>>>>> Mar 1 11:07:01 s06 systemd: Started Session 196232 of user root. >>>>>>>> Mar 1 11:07:01 s06 systemd: Starting Session 196232 of user root. >>>>>>>> Mar 1 11:07:01 s06 systemd: Removed slice User Slice of root. >>>>>>>> Mar 1 11:07:01 s06 systemd: Stopping User Slice of root. >>>>>>>> Mar 1 11:07:36 s06 glustershd[70117]: [2019-03-01 10:07:36.366259] >>>>>>>> C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-0: >>>>>>>> server 192.168.0.51:49152 has not responded in the last 42 >>>>>>>> seconds, disconnecting. >>>>>>>> Mar 1 11:08:01 s06 systemd: Created slice User Slice of root. >>>>>>>> >>>>>>>> No ?blocked? word is in /var/log/messages files on other cluster >>>>>>>> servers. >>>>>>>> In attachment, the /var/log/messages file from s06 server. >>>>>>>> >>>>>>>> Thank you in advance, >>>>>>>> Mauro >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 1 Mar 2019, at 11:47, Milind Changire >>>>>>>> wrote: >>>>>>>> >>>>>>>> The traces of very high disk activity on the servers are often >>>>>>>> found in /var/log/messages >>>>>>>> You might want to grep for "blocked for" in /var/log/messages on >>>>>>>> s06 and correlate the timestamps to confirm the unresponsiveness as >>>>>>>> reported in gluster client logs. >>>>>>>> In cases of high disk activity, although the operating system >>>>>>>> continues to respond to ICMP pings, the processes writing to disks often >>>>>>>> get blocked to a large flush to the disk which could span beyond 42 seconds >>>>>>>> and hence result in ping-timer-expiry logs. >>>>>>>> >>>>>>>> As a side note: >>>>>>>> If you indeed find gluster processes being blocked in >>>>>>>> /var/log/messages, you might want to tweak sysctl tunables called >>>>>>>> vm.dirty_background_ratio or vm.dirty_background_bytes to a smaller value >>>>>>>> than the existing. Please read up more on those tunables before touching >>>>>>>> the settings. >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Mar 1, 2019 at 4:06 PM Mauro Tridici >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> in attachment the client log captured after changing >>>>>>>>> network.ping-timeout option. >>>>>>>>> I noticed this error involving server 192.168.0.56 (s06) >>>>>>>>> >>>>>>>>> [2019-03-01 09:23:36.077287] I [rpc-clnt.c:1962:rpc_clnt_reconfig] >>>>>>>>> 0-tier2-client-71: changing ping timeout to 42 (from 0) >>>>>>>>> [2019-03-01 09:23:36.078213] I >>>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>>> volfile,continuing >>>>>>>>> [2019-03-01 09:23:36.078432] I >>>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>>> volfile,continuing >>>>>>>>> [2019-03-01 09:23:36.092357] I >>>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>>> volfile,continuing >>>>>>>>> [2019-03-01 09:23:36.094146] I >>>>>>>>> [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in >>>>>>>>> volfile,continuing >>>>>>>>> [2019-03-01 10:06:24.708082] C >>>>>>>>> [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-50: server >>>>>>>>> 192.168.0.56:49156 has not responded in the last 42 seconds, >>>>>>>>> disconnecting. >>>>>>>>> >>>>>>>>> I don?t know why it happens, s06 server seems to be reachable. >>>>>>>>> >>>>>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>>>>> Trying 192.168.0.56... >>>>>>>>> Connected to 192.168.0.56. >>>>>>>>> Escape character is '^]'. >>>>>>>>> ^CConnection closed by foreign host. >>>>>>>>> [athena_login2][/users/home/sysm02/]> ping 192.168.0.56 >>>>>>>>> PING 192.168.0.56 (192.168.0.56) 56(84) bytes of data. >>>>>>>>> 64 bytes from 192.168.0.56: icmp_seq=1 ttl=64 time=0.116 ms >>>>>>>>> 64 bytes from 192.168.0.56: icmp_seq=2 ttl=64 time=0.101 ms >>>>>>>>> >>>>>>>>> --- 192.168.0.56 ping statistics --- >>>>>>>>> 2 packets transmitted, 2 received, 0% packet loss, time 1528ms >>>>>>>>> rtt min/avg/max/mdev = 0.101/0.108/0.116/0.012 ms >>>>>>>>> >>>>>>>>> [athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156 >>>>>>>>> Trying 192.168.0.56... >>>>>>>>> Connected to 192.168.0.56. >>>>>>>>> Escape character is '^]'. >>>>>>>>> >>>>>>>>> Thank you for your help, >>>>>>>>> Mauro >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 1 Mar 2019, at 10:29, Mauro Tridici >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> thank you for the explanation. >>>>>>>>> I just changed network.ping-timeout option to default value >>>>>>>>> (network.ping-timeout=42). >>>>>>>>> >>>>>>>>> I will check the logs to see if the errors will appear again. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Mauro >>>>>>>>> >>>>>>>>> On 1 Mar 2019, at 04:43, Milind Changire >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> network.ping-timeout should not be set to zero for non-glusterd >>>>>>>>> clients. >>>>>>>>> glusterd is a special case for which ping-timeout is set to zero >>>>>>>>> via /etc/glusterfs/glusterd.vol >>>>>>>>> >>>>>>>>> Setting network.ping-timeout to zero disables arming of the ping >>>>>>>>> timer for connections. This disables testing the connection for >>>>>>>>> responsiveness and hence avoids proactive fail-over. >>>>>>>>> >>>>>>>>> Please reset network.ping-timeout to a non-zero positive value, >>>>>>>>> eg. 42 >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Feb 28, 2019 at 5:07 PM Nithya Balachandran < >>>>>>>>> nbalacha at redhat.com> wrote: >>>>>>>>> >>>>>>>>>> Adding Raghavendra and Milind to comment on this. >>>>>>>>>> >>>>>>>>>> What is the effect of setting network.ping-timeout to 0 and >>>>>>>>>> should it be set back to 42? >>>>>>>>>> Regards, >>>>>>>>>> Nithya >>>>>>>>>> >>>>>>>>>> On Thu, 28 Feb 2019 at 16:01, Mauro Tridici < >>>>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Nithya, >>>>>>>>>>> >>>>>>>>>>> sorry for the late. >>>>>>>>>>> network.ping-timeout has been set to 0 in order to try to solve >>>>>>>>>>> some timeout problems, but it didn?t help. >>>>>>>>>>> I can set it to the default value. >>>>>>>>>>> >>>>>>>>>>> Can I proceed with the change? >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> Mauro >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 28 Feb 2019, at 04:41, Nithya Balachandran < >>>>>>>>>>> nbalacha at redhat.com> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Mauro, >>>>>>>>>>> >>>>>>>>>>> Is network.ping-timeout still set to 0. The default value is 42. >>>>>>>>>>> Is there a particular reason why this was changed? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Nithya >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, 27 Feb 2019 at 21:32, Mauro Tridici < >>>>>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Xavi, >>>>>>>>>>>> >>>>>>>>>>>> thank you for the detailed explanation and suggestions. >>>>>>>>>>>> Yes, transport.listen-backlog option is still set to 1024. >>>>>>>>>>>> >>>>>>>>>>>> I will check the network and connectivity status using ?ping? >>>>>>>>>>>> and ?telnet? as soon as the errors will come back again. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Mauro >>>>>>>>>>>> >>>>>>>>>>>> Il giorno 27 feb 2019, alle ore 16:42, Xavi Hernandez < >>>>>>>>>>>> jahernan at redhat.com> ha scritto: >>>>>>>>>>>> >>>>>>>>>>>> Hi Mauro, >>>>>>>>>>>> >>>>>>>>>>>> those errors say that the mount point is not connected to some >>>>>>>>>>>> of the bricks while executing operations. I see references to 3rd and 6th >>>>>>>>>>>> bricks of several disperse sets, which seem to map to server s06. For some >>>>>>>>>>>> reason, gluster is having troubles connecting from the client machine to >>>>>>>>>>>> that particular server. At the end of the log I see that after long time a >>>>>>>>>>>> reconnect is done to both of them. However little after, other bricks from >>>>>>>>>>>> the s05 get disconnected and a reconnect times out. >>>>>>>>>>>> >>>>>>>>>>>> That's really odd. It seems like if server/communication is cut >>>>>>>>>>>> to s06 for some time, then restored, and then the same happens to the next >>>>>>>>>>>> server. >>>>>>>>>>>> >>>>>>>>>>>> If the servers are really online and it's only a communication >>>>>>>>>>>> issue, it explains why server memory and network has increased: if the >>>>>>>>>>>> problem only exists between the client and servers, any write made by the >>>>>>>>>>>> client will automatically mark the file as damaged, since some of the >>>>>>>>>>>> servers have not been updated. Since self-heal runs from the server nodes, >>>>>>>>>>>> they will probably be correctly connected to all bricks, which allows them >>>>>>>>>>>> to heal the just damaged file, which increases memory and network usage. >>>>>>>>>>>> >>>>>>>>>>>> I guess you still have transport.listen-backlog set to 1024, >>>>>>>>>>>> right ? >>>>>>>>>>>> >>>>>>>>>>>> Just to try to identify if the problem really comes from >>>>>>>>>>>> network, can you check if you lose some pings from the client to all of the >>>>>>>>>>>> servers while you are seeing those errors in the log file ? >>>>>>>>>>>> >>>>>>>>>>>> You can also check if during those errors, you can telnet to >>>>>>>>>>>> the port of the brick from the client. >>>>>>>>>>>> >>>>>>>>>>>> Xavi >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Feb 26, 2019 at 10:17 AM Mauro Tridici < >>>>>>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Nithya, >>>>>>>>>>>>> >>>>>>>>>>>>> ?df -h? operation is not still slow, but no users are using >>>>>>>>>>>>> the volume, RAM and NETWORK usage is ok on the client node. >>>>>>>>>>>>> >>>>>>>>>>>>> I was worried about this kind of warnings/errors: >>>>>>>>>>>>> >>>>>>>>>>>>> [2019-02-25 10:59:00.664323] W [MSGID: 122035] >>>>>>>>>>>>> [ec-common.c:571:ec_child_select] 0-tier2-disperse-6: Executing operation >>>>>>>>>>>>> with some subvolumes unavailable (20) >>>>>>>>>>>>> >>>>>>>>>>>>> [2019-02-26 03:11:35.212603] E >>>>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>>>>> called at 2019-02-26 03:10:56.549903 (xid=0x106f1c5) >>>>>>>>>>>>> >>>>>>>>>>>>> [2019-02-26 03:13:03.313831] E >>>>>>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-50: connection to >>>>>>>>>>>>> 192.168.0.56:49156 failed (Timeout della connessione); >>>>>>>>>>>>> disconnecting socket >>>>>>>>>>>>> >>>>>>>>>>>>> It seems that some subvolumes are not available and >>>>>>>>>>>>> 192.168.0.56 server (s06) is not reachable. >>>>>>>>>>>>> But gluster servers are up&running and bricks are ok. >>>>>>>>>>>>> >>>>>>>>>>>>> In attachment the updated tier2.log file. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you. >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Mauro >>>>>>>>>>>>> >>>>>>>>>>>>> Il giorno 26 feb 2019, alle ore 04:03, Nithya Balachandran < >>>>>>>>>>>>> nbalacha at redhat.com> ha scritto: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> I see a lot of EC messages in the log but they don't seem very >>>>>>>>>>>>> serious. Xavi, can you take a look? >>>>>>>>>>>>> >>>>>>>>>>>>> The only errors I see are: >>>>>>>>>>>>> [2019-02-25 10:58:45.519871] E >>>>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>>>>> 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>>>>> called at 2019-02-25 10:57:47.429969 (xid=0xd26fe7) >>>>>>>>>>>>> [2019-02-25 10:58:51.461493] E >>>>>>>>>>>>> [rpc-clnt.c:350:saved_frames_unwind] (--> >>>>>>>>>>>>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> >>>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> >>>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> >>>>>>>>>>>>> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] >>>>>>>>>>>>> (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) >>>>>>>>>>>>> 0-tier2-client-41: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) >>>>>>>>>>>>> called at 2019-02-25 10:57:47.499174 (xid=0xf47d6a) >>>>>>>>>>>>> [2019-02-25 11:07:57.152874] E >>>>>>>>>>>>> [socket.c:2376:socket_connect_finish] 0-tier2-client-70: connection to >>>>>>>>>>>>> 192.168.0.55:49163 failed (Timeout della connessione); >>>>>>>>>>>>> disconnecting socket >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Is the df -h operation still slow? If yes, can you take a >>>>>>>>>>>>> tcpdump of the client while running df -h and send that across? >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Nithya >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, 25 Feb 2019 at 17:27, Mauro Tridici < >>>>>>>>>>>>> mauro.tridici at cmcc.it> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sorry, some minutes after my last mail message, I noticed >>>>>>>>>>>>>> that ?df -h? command hanged for a while before returns the prompt. >>>>>>>>>>>>>> Yesterday, everything was ok in the gluster client log, but, >>>>>>>>>>>>>> today, I see a lot of errors (please, take a look to the attached file). >>>>>>>>>>>>>> >>>>>>>>>>>>>> On the client node, I detected an important RAM e NETWORK >>>>>>>>>>>>>> usage. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Do you think that the errors have been caused by the client >>>>>>>>>>>>>> resources usage? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you in advance, >>>>>>>>>>>>>> Mauro >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Thu Mar 21 10:55:09 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Thu, 21 Mar 2019 16:25:09 +0530 Subject: [Gluster-devel] [Gluster-Maintainers] GF_CALLOC to GF_MALLOC conversion - is it safe? In-Reply-To: References: Message-ID: On Thu, Mar 21, 2019 at 4:16 PM Atin Mukherjee wrote: > All, > > In the last few releases of glusterfs, with stability as a primary theme > of the releases, there has been lots of changes done on the code > optimization with an expectation that such changes will have gluster to > provide better performance. While many of these changes do help, but off > late we have started seeing some diverse effects of them, one especially > being the calloc to malloc conversions. While I do understand that malloc > syscall will eliminate the extra memset bottleneck which calloc bears, but > with recent kernels having in-built strong compiler optimizations I am not > sure whether that makes any significant difference, but as I mentioned > earlier certainly if this isn't done carefully it can potentially introduce > lot of bugs and I'm writing this email to share one of such experiences. > > Sanju & I were having troubles for last two days to figure out why > https://review.gluster.org/#/c/glusterfs/+/22388/ wasn't working in > Sanju's system but it had no problems running the same fix in my gluster > containers. After spending a significant amount of time, what we now > figured out is that a malloc call [1] (which was a calloc earlier) is the > culprit here. As you all can see, in this function we allocate txn_id and > copy the event->txn_id into it through gf_uuid_copy () . But when we were > debugging this step wise through gdb, txn_id wasn't exactly copied with the > exact event->txn_id and it had some junk values which made the > glusterd_clear_txn_opinfo to be invoked with a wrong txn_id later on > resulting the leaks to remain the same which was the original intention of > the fix. > > This was quite painful to debug and we had to spend some time to figure > this out. Considering we have converted many such calls in past, I'd urge > that we review all such conversions and see if there're any side effects to > it. Otherwise we might end up running into many potential memory related > bugs later on. OTOH, going forward I'd request every patch > owners/maintainers to pay some special attention to these conversions and > see they are really beneficial and error free. IMO, general guideline > should be - for bigger buffers, malloc would make better sense but has to > be done carefully, for smaller size, we stick to calloc. > > What do others think about it? > I too am afraid of unknown effects of this change as much of the codebase relies on the assumption of zero-initialized data structures. I vote for reverting these patches unless it can be demonstrated that performance benefits are indeed significant. Otherwise the trade off in stability is not worth the cost. > > [1] > https://github.com/gluster/glusterfs/blob/master/xlators/mgmt/glusterd/src/glusterd-op-sm.c#L5681 > _______________________________________________ > maintainers mailing list > maintainers at gluster.org > https://lists.gluster.org/mailman/listinfo/maintainers > -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiubli at redhat.com Thu Mar 21 13:01:02 2019 From: xiubli at redhat.com (Xiubo Li) Date: Thu, 21 Mar 2019 21:01:02 +0800 Subject: [Gluster-devel] Network Block device (NBD) on top of glusterfs In-Reply-To: References: Message-ID: <072c6bb2-eee1-8374-9b53-b9561deebfc7@redhat.com> On 2019/3/21 18:09, Prasanna Kalever wrote: > > > On Thu, Mar 21, 2019 at 9:00 AM Xiubo Li > wrote: > > All, > > I am one of the contributor forgluster-block > [1] project, and also I > contribute to linux kernel andopen-iscsi > project.[2] > > NBD was around for some time, but in recent time, linux kernel?s > Network Block Device (NBD) is enhanced and made to work with more > devices and also the option to integrate with netlink is added. > So, I tried to provide a glusterfs client based NBD driver > recently. Please refergithub issue #633 > [3], and good > news is I have a working code, with most basic things @nbd-runner > project [4]. > > While this email is about announcing the project, and asking for > more collaboration, I would also like to discuss more about the > placement of the project itself. Currently nbd-runner project is > expected to be shared by our friends at Ceph project too, to > provide NBD driver for Ceph. I have personally worked with some of > them closely while contributing to open-iSCSI project, and we > would like to take this project to great success. > > Now few questions: > > 1. Can I continue to usehttp://github.com/gluster/nbd-runneras > home for this project, even if its shared by other filesystem > projects? > > * I personally am fine with this. > > 2. Should there be a separate organization for this repo? > > * While it may make sense in future, for now, I am not planning > to start any new thing? > > It would be great if we have some consensus on this soon as > nbd-runner is a new repository. If there are no concerns, I will > continue to contribute to the existing repository. > > > Thanks Xiubo Li, for finally sending this email out. Since this email > is out on gluster mailing list, I would like to take a stand from > gluster community point of view *only* and share my views. > > My honest answer is "If we want to maintain this within gluster org, > then 80% of the effort is common/duplicate of what we did all these > days with gluster-block", > The great idea came from Mike Christie days ago and the nbd-runner project's framework is initially emulated from tcmu-runner. This is why I name this project as nbd-runner, which will work for all the other Distributed Storages, such as Gluster/Ceph/Azure, as discussed with Mike before. nbd-runner(NBD proto) and tcmu-runner(iSCSI proto) are almost the same and both are working as lower IO(READ/WRITE/...) stuff, not the management layer like ceph-iscsi-gateway and gluster-block currently do. Currently since I only implemented the Gluster handler and also using the RPC like glusterfs and gluster-block, most of the other code (about 70%) in nbd-runner are for the NBD proto and these are very different from tcmu-runner/glusterfs/gluster-block projects, and there are many new features in NBD module that not yet supported and then there will be more different in future. The framework coding has been done and the nbd-runner project is already stable and could already work well for me now. > like: > * rpc/socket code > * cli/daemon parser/helper logics > * gfapi util functions > * logger framework > * inotify & dyn-config threads Yeah, these features were initially from tcmu-runner project, Mike and I coded two years ago. Currently nbd-runner also has copied them from tcmu-runner. Very appreciated for you great ideas here Prasanna and hope nbd-runner could be more generically and successfully used in future. BRs Xiubo Li > * configure/Makefile/specfiles > * docsAboutGluster and etc .. > > The repository gluster-block is actually a home for all the block > related stuff within gluster and its designed to accommodate alike > functionalities, if I was you I would have simply copied nbd-runner.c > into https://github.com/gluster/gluster-block/tree/master/daemon/ just > like ceph plays it here > https://github.com/ceph/ceph/blob/master/src/tools/rbd_nbd/rbd-nbd.cc > and be done. > > Advantages of keeping nbd client within gluster-block: > -> No worry about maintenance code burdon > -> No worry about monitoring a new component > -> shipping packages to fedora/centos/rhel is handled > -> This helps improve and stabilize the current gluster-block framework > -> We can build a common CI > -> We can use reuse common test framework and etc .. > > If you have an impression that gluster-block is for management, then I > would really want to correct you at this point. > > Some of my near future plans for gluster-block: > * Allow exporting blocks with FUSE access via fileIO backstore to > improve large-file workloads, draft: > https://github.com/gluster/gluster-block/pull/58 > * Accommodate kernel loopback handling for local only applications > * The same way we can accommodate nbd app/client, and IMHO this effort > shouldn't take 1 or 2 days to get it merged with in gluster-block and > ready for a go release. > > > Hope that clarifies it. > > > Best Regards, > -- > Prasanna > > Regards, > Xiubo Li (@lxbsz) > > [1] -https://github.com/gluster/gluster-block > [2] -https://github.com/open-iscsi > [3] -https://github.com/gluster/glusterfs/issues/633 > [4] -https://github.com/gluster/nbd-runner > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ykaul at redhat.com Thu Mar 21 13:57:13 2019 From: ykaul at redhat.com (Yaniv Kaul) Date: Thu, 21 Mar 2019 15:57:13 +0200 Subject: [Gluster-devel] GF_CALLOC to GF_MALLOC conversion - is it safe? In-Reply-To: References: Message-ID: On Thu, Mar 21, 2019 at 12:45 PM Atin Mukherjee wrote: > All, > > In the last few releases of glusterfs, with stability as a primary theme > of the releases, there has been lots of changes done on the code > optimization with an expectation that such changes will have gluster to > provide better performance. While many of these changes do help, but off > late we have started seeing some diverse effects of them, one especially > being the calloc to malloc conversions. While I do understand that malloc > syscall will eliminate the extra memset bottleneck which calloc bears, but > with recent kernels having in-built strong compiler optimizations I am not > sure whether that makes any significant difference, but as I mentioned > earlier certainly if this isn't done carefully it can potentially introduce > lot of bugs and I'm writing this email to share one of such experiences. > > Sanju & I were having troubles for last two days to figure out why > https://review.gluster.org/#/c/glusterfs/+/22388/ wasn't working in > Sanju's system but it had no problems running the same fix in my gluster > containers. After spending a significant amount of time, what we now > figured out is that a malloc call [1] (which was a calloc earlier) is the > culprit here. As you all can see, in this function we allocate txn_id and > copy the event->txn_id into it through gf_uuid_copy () . But when we were > debugging this step wise through gdb, txn_id wasn't exactly copied with the > exact event->txn_id and it had some junk values which made the > glusterd_clear_txn_opinfo to be invoked with a wrong txn_id later on > resulting the leaks to remain the same which was the original intention of > the fix. > - I'm not sure I understand what 'wasn't exactly copied' means? It either copied or did not copy the event->txn_id ? Is event->txn_id not fully populated somehow? - This is a regression caused by 81cbbfd1d870bea49b8aafe7bebb9e8251190918 which I introduced in August 4th, and we are just now discovering it. This is not good. Without looking, I assume almost all CALLOC->MALLOC changes are done on positive paths of the code, which means it's not tested well. This file, while having a low code coverage, seems to be covered[1], so I'm not sure how we are finding this now? > > This was quite painful to debug and we had to spend some time to figure > this out. Considering we have converted many such calls in past, I'd urge > that we review all such conversions and see if there're any side effects to > it. Otherwise we might end up running into many potential memory related > bugs later on. OTOH, going forward I'd request every patch > owners/maintainers to pay some special attention to these conversions and > see they are really beneficial and error free. IMO, general guideline > should be - for bigger buffers, malloc would make better sense but has to > be done carefully, for smaller size, we stick to calloc. > > What do others think about it? > I think I might have been aggressive with the changes, but I do feel they are important in some areas where it makes sense. For example: libglusterfs/src/inode.c : new->inode_hash = (void *)GF_CALLOC(*65536, sizeof(struct list_head)*, gf_common_mt_list_head); if (!new->inode_hash) goto out; new->name_hash = (void *)GF_CALLOC(*new->hashsize, sizeof(struct list_head)*, gf_common_mt_list_head); if (!new->name_hash) goto out; And just few lines later: for (i = 0; i < *65536*; i++) { INIT_LIST_HEAD(&new->inode_hash[i]); } for (i = 0; i < *new->hashsize*; i++) { INIT_LIST_HEAD(&new->name_hash[i]); } So this is really a waste of cycles for no good reason. I agree not every CALLOC is worth converting. One more note, I'd love to be able to measure the effect. But there's no CI job with benchmarks, inc. CPU and memory consumption, which we can evaluate the changes. And lastly, we need better performance. We need better scalability. We are not keeping up with HW advancements (especially NVMe, pmem and such) and (just like other storage stacks!) becoming somewhat of a performance bottleneck. Y. > > [1] > https://github.com/gluster/glusterfs/blob/master/xlators/mgmt/glusterd/src/glusterd-op-sm.c#L5681 > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From pkalever at redhat.com Thu Mar 21 14:23:19 2019 From: pkalever at redhat.com (Prasanna Kalever) Date: Thu, 21 Mar 2019 19:53:19 +0530 Subject: [Gluster-devel] Network Block device (NBD) on top of glusterfs In-Reply-To: <072c6bb2-eee1-8374-9b53-b9561deebfc7@redhat.com> References: <072c6bb2-eee1-8374-9b53-b9561deebfc7@redhat.com> Message-ID: On Thu, Mar 21, 2019 at 6:31 PM Xiubo Li wrote: > On 2019/3/21 18:09, Prasanna Kalever wrote: > > > > On Thu, Mar 21, 2019 at 9:00 AM Xiubo Li wrote: > >> All, >> >> I am one of the contributor for gluster-block >> [1] project, and also I >> contribute to linux kernel and open-iscsi >> project.[2] >> >> NBD was around for some time, but in recent time, linux kernel?s Network >> Block Device (NBD) is enhanced and made to work with more devices and also >> the option to integrate with netlink is added. So, I tried to provide a >> glusterfs client based NBD driver recently. Please refer github issue >> #633 [3], and good news >> is I have a working code, with most basic things @ nbd-runner project >> [4]. >> >> While this email is about announcing the project, and asking for more >> collaboration, I would also like to discuss more about the placement of the >> project itself. Currently nbd-runner project is expected to be shared by >> our friends at Ceph project too, to provide NBD driver for Ceph. I have >> personally worked with some of them closely while contributing to >> open-iSCSI project, and we would like to take this project to great success. >> >> Now few questions: >> >> 1. Can I continue to use http://github.com/gluster/nbd-runner as home >> for this project, even if its shared by other filesystem projects? >> >> >> - I personally am fine with this. >> >> >> 1. Should there be a separate organization for this repo? >> >> >> - While it may make sense in future, for now, I am not planning to >> start any new thing? >> >> It would be great if we have some consensus on this soon as nbd-runner is >> a new repository. If there are no concerns, I will continue to contribute >> to the existing repository. >> > > Thanks Xiubo Li, for finally sending this email out. Since this email is > out on gluster mailing list, I would like to take a stand from gluster > community point of view *only* and share my views. > > My honest answer is "If we want to maintain this within gluster org, then > 80% of the effort is common/duplicate of what we did all these days with > gluster-block", > > The great idea came from Mike Christie days ago and the nbd-runner > project's framework is initially emulated from tcmu-runner. This is why I > name this project as nbd-runner, which will work for all the other > Distributed Storages, such as Gluster/Ceph/Azure, as discussed with Mike > before. > > nbd-runner(NBD proto) and tcmu-runner(iSCSI proto) are almost the same and > both are working as lower IO(READ/WRITE/...) stuff, not the management > layer like ceph-iscsi-gateway and gluster-block currently do. > > Currently since I only implemented the Gluster handler and also using the > RPC like glusterfs and gluster-block, most of the other code (about 70%) in > nbd-runner are for the NBD proto and these are very different from > tcmu-runner/glusterfs/gluster-block projects, and there are many new > features in NBD module that not yet supported and then there will be more > different in future. > > The framework coding has been done and the nbd-runner project is already > stable and could already work well for me now. > > like: > * rpc/socket code > * cli/daemon parser/helper logics > * gfapi util functions > * logger framework > * inotify & dyn-config threads > > Yeah, these features were initially from tcmu-runner project, Mike and I > coded two years ago. Currently nbd-runner also has copied them from > tcmu-runner. > I don't think tcmu-runner has any of, -> cli/daemon approach routines -> rpc low-level clnt/svc routines -> gfapi level file create/delete util functions -> Json parser support -> socket bound/listener related functionalities -> autoMake build frame-work, and -> many other maintenance files I actually can go in detail and furnish a long list of reference made here and you cannot deny the fact, but its **all okay** to take references from other alike projects. But my intention was not to point about the copy made here, but rather saying we are just wasting our efforts rewriting, copy-pasting, maintaining and fixing the same functionality framework. Again all I'm trying to make is, if at all you want to maintain nbd client as part of gluster.org, why not use gluster-block itself ? which is well tested and stable enough. Apart from all the examples I have mentioned in my previous thread, there are other great advantages from user perspective as-well, like: * The top layers such as heketi consuming gluster's block storage really don't have to care whether the backend provider is tcmu-runner or nbd-runner or qemu-tcmu or kernel loopback or fileIO or something else ... They simply call gluster-block and get a block device out there. * We can reuse the existing gluster-block's rest api interface too. ** Believe me, over the years I have learned it from my experience and its a very fact that, we can save lot of energy and time by reusing existing stable framework rather than building a new one from scratch ** I will try to spend few hours over my weekends and send a nbd client application PR for gluster-block (IMO this shouldn't exceed ~200 lines), will request your review there. Cheers! -- Prasanna > Very appreciated for you great ideas here Prasanna and hope nbd-runner > could be more generically and successfully used in future. > > BRs > > Xiubo Li > > > * configure/Makefile/specfiles > * docsAboutGluster and etc .. > > The repository gluster-block is actually a home for all the block related > stuff within gluster and its designed to accommodate alike functionalities, > if I was you I would have simply copied nbd-runner.c into > https://github.com/gluster/gluster-block/tree/master/daemon/ just like > ceph plays it here > https://github.com/ceph/ceph/blob/master/src/tools/rbd_nbd/rbd-nbd.cc and > be done. > > Advantages of keeping nbd client within gluster-block: > -> No worry about maintenance code burdon > -> No worry about monitoring a new component > -> shipping packages to fedora/centos/rhel is handled > -> This helps improve and stabilize the current gluster-block framework > -> We can build a common CI > -> We can use reuse common test framework and etc .. > > If you have an impression that gluster-block is for management, then I > would really want to correct you at this point. > > Some of my near future plans for gluster-block: > * Allow exporting blocks with FUSE access via fileIO backstore to improve > large-file workloads, draft: > https://github.com/gluster/gluster-block/pull/58 > * Accommodate kernel loopback handling for local only applications > * The same way we can accommodate nbd app/client, and IMHO this effort > shouldn't take 1 or 2 days to get it merged with in gluster-block and ready > for a go release. > > > Hope that clarifies it. > > > Best Regards, > -- > Prasanna > > >> Regards, >> Xiubo Li (@lxbsz) >> >> [1] - https://github.com/gluster/gluster-block >> [2] - https://github.com/open-iscsi >> [3] - https://github.com/gluster/glusterfs/issues/633 >> [4] - https://github.com/gluster/nbd-runner >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-devel > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nbalacha at redhat.com Thu Mar 21 15:22:45 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Thu, 21 Mar 2019 20:52:45 +0530 Subject: [Gluster-devel] [Gluster-Maintainers] GF_CALLOC to GF_MALLOC conversion - is it safe? In-Reply-To: References: Message-ID: On Thu, 21 Mar 2019 at 16:16, Atin Mukherjee wrote: > All, > > In the last few releases of glusterfs, with stability as a primary theme > of the releases, there has been lots of changes done on the code > optimization with an expectation that such changes will have gluster to > provide better performance. While many of these changes do help, but off > late we have started seeing some diverse effects of them, one especially > being the calloc to malloc conversions. While I do understand that malloc > syscall will eliminate the extra memset bottleneck which calloc bears, but > with recent kernels having in-built strong compiler optimizations I am not > sure whether that makes any significant difference, but as I mentioned > earlier certainly if this isn't done carefully it can potentially introduce > lot of bugs and I'm writing this email to share one of such experiences. > > Sanju & I were having troubles for last two days to figure out why > https://review.gluster.org/#/c/glusterfs/+/22388/ wasn't working in > Sanju's system but it had no problems running the same fix in my gluster > containers. After spending a significant amount of time, what we now > figured out is that a malloc call [1] (which was a calloc earlier) is the > culprit here. As you all can see, in this function we allocate txn_id and > copy the event->txn_id into it through gf_uuid_copy () . But when we were > debugging this step wise through gdb, txn_id wasn't exactly copied with the > exact event->txn_id and it had some junk values which made the > glusterd_clear_txn_opinfo to be invoked with a wrong txn_id later on > resulting the leaks to remain the same which was the original intention of > the fix. > > This was quite painful to debug and we had to spend some time to figure > this out. Considering we have converted many such calls in past, I'd urge > that we review all such conversions and see if there're any side effects to > it. Otherwise we might end up running into many potential memory related > bugs later on. OTOH, going forward I'd request every patch > owners/maintainers to pay some special attention to these conversions and > see they are really beneficial and error free. IMO, general guideline > should be - for bigger buffers, malloc would make better sense but has to > be done carefully, for smaller size, we stick to calloc. > > What do others think about it? > I believe that replacing calloc with malloc everywhere without adequate testing and review is not safe and am against doing so for the following reasons: 1. Most of these patches have not been tested, especially the error paths.I have seen some that introduced issues in error scenarios with pointers being non-null. 2. As Raghavendra said, the code assumes that certain elements will be initialized to null/zero and changing that can have consequences which are not immediately obvious. I think it might be worthwhile to go through the already merged calloc->malloc patches to check error paths and so on to see if they are safe. 3. Making such changes to the libglusterfs code while we are currently working to stabilize the product is not a good idea. The patches take time to review and any errors introduced in the core pieces affect all processes and require significant effort to debug. Yaniv, while the example you provided might make sense to change to malloc, a lot of the other changes, in my opinion, do not for the effort required. For performance testing, smallfile might be a useful tool to see if any of the changes make a difference. That said, I am reluctant to take in patches that change core code significantly without being tested or providing proof of benefits. We need better performance and scalability but that is going to need changes in our algorithms and fop handling and that is what we need to be working on. Such changes, when done right, will provide more benefits than the micro optimizations. I think it unlikely the micro optimizations will provide much benefit but am willing to be proven wrong if you have numbers that show otherwise. Regards, Nithya > > [1] > https://github.com/gluster/glusterfs/blob/master/xlators/mgmt/glusterd/src/glusterd-op-sm.c#L5681 > _______________________________________________ > maintainers mailing list > maintainers at gluster.org > https://lists.gluster.org/mailman/listinfo/maintainers > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ykaul at redhat.com Thu Mar 21 15:43:27 2019 From: ykaul at redhat.com (Yaniv Kaul) Date: Thu, 21 Mar 2019 17:43:27 +0200 Subject: [Gluster-devel] [Gluster-Maintainers] GF_CALLOC to GF_MALLOC conversion - is it safe? In-Reply-To: References: Message-ID: On Thu, Mar 21, 2019 at 5:23 PM Nithya Balachandran wrote: > > > On Thu, 21 Mar 2019 at 16:16, Atin Mukherjee wrote: > >> All, >> >> In the last few releases of glusterfs, with stability as a primary theme >> of the releases, there has been lots of changes done on the code >> optimization with an expectation that such changes will have gluster to >> provide better performance. While many of these changes do help, but off >> late we have started seeing some diverse effects of them, one especially >> being the calloc to malloc conversions. While I do understand that malloc >> syscall will eliminate the extra memset bottleneck which calloc bears, but >> with recent kernels having in-built strong compiler optimizations I am not >> sure whether that makes any significant difference, but as I mentioned >> earlier certainly if this isn't done carefully it can potentially introduce >> lot of bugs and I'm writing this email to share one of such experiences. >> >> Sanju & I were having troubles for last two days to figure out why >> https://review.gluster.org/#/c/glusterfs/+/22388/ wasn't working in >> Sanju's system but it had no problems running the same fix in my gluster >> containers. After spending a significant amount of time, what we now >> figured out is that a malloc call [1] (which was a calloc earlier) is the >> culprit here. As you all can see, in this function we allocate txn_id and >> copy the event->txn_id into it through gf_uuid_copy () . But when we were >> debugging this step wise through gdb, txn_id wasn't exactly copied with the >> exact event->txn_id and it had some junk values which made the >> glusterd_clear_txn_opinfo to be invoked with a wrong txn_id later on >> resulting the leaks to remain the same which was the original intention of >> the fix. >> >> This was quite painful to debug and we had to spend some time to figure >> this out. Considering we have converted many such calls in past, I'd urge >> that we review all such conversions and see if there're any side effects to >> it. Otherwise we might end up running into many potential memory related >> bugs later on. OTOH, going forward I'd request every patch >> owners/maintainers to pay some special attention to these conversions and >> see they are really beneficial and error free. IMO, general guideline >> should be - for bigger buffers, malloc would make better sense but has to >> be done carefully, for smaller size, we stick to calloc. >> > >> What do others think about it? >> > > I believe that replacing calloc with malloc everywhere without adequate > testing and review is not safe and am against doing so for the following > reasons: > No patch should get in without adequate testing and thorough review. > > 1. Most of these patches have not been tested, especially the error > paths.I have seen some that introduced issues in error scenarios with > pointers being non-null. > > You raise an interesting issue. Why are free'd memory pointers are not NULL'ified? Why does FREE() set ptr = (void *)0xeeeeeeee and not NULL? This is a potential cause for failure. A re-occuring FREE(NULL) is harmless. A FREE(0xeeeeeeee) is a bit more problematic. > 1. > 2. As Raghavendra said, the code assumes that certain elements will be > initialized to null/zero and changing that can have consequences which are > not immediately obvious. I think it might be worthwhile to go through the > already merged calloc->malloc patches to check error paths and so on to see > if they are safe. > > Agreed. > > 1. > 2. Making such changes to the libglusterfs code while we are currently > working to stabilize the product is not a good idea. The patches take time > to review and any errors introduced in the core pieces affect all processes > and require significant effort to debug. > > Let me know when we consider the project stable. I'd argue the way to stabilize it is not stop improving it, but improving its testing. From more tests to cover more code via more tests to more static analysis coverage, to ensuring CI is rock-solid (inc. random errors that pop up from time to time). Not accepting patches to master is not the right approach, unless it's time-boxed somehow. If it is, then it means we don't trust our CI enough, btw. > 1. > > Yaniv, while the example you provided might make sense to change to > malloc, a lot of the other changes, in my opinion, do not for the effort > required. For performance testing, smallfile might be a useful tool to see > if any of the changes make a difference. That said, I am reluctant to take > in patches that change core code significantly without being tested or > providing proof of benefits. > Smallfile is part of CI? I am happy to see it documented @ https://docs.gluster.org/en/latest/Administrator%20Guide/Performance%20Testing/#smallfile-distributed-io-benchmark , so at least one can know how to execute it manually. > > We need better performance and scalability but that is going to need > changes in our algorithms and fop handling and that is what we need to be > working on. Such changes, when done right, will provide more benefits than > the micro optimizations. I think it unlikely the micro optimizations will > provide much benefit but am willing to be proven wrong if you have numbers > that show otherwise. > I call them nano-optimizations. I believe they really shave only that much, microsecond would be great [looking at https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html , I'm in that ballpark figure.]. Unless we got enough cache misses (btw, would be nice to add likely()/unlikely() to our code, to give a hint to the compiler on code paths). Not sure that helps that much as well, of course - same figures, I estimate. But I do like the fact they remove the false assumption that everything is NULL and we don't need to worry about it - it means we need to look at our structures (and while at it, align them!) and make sure we know what we initialize. But I certainly understand the resistance and will abandon the patches if not welcome. Y. > Regards, > Nithya > > >> >> [1] >> https://github.com/gluster/glusterfs/blob/master/xlators/mgmt/glusterd/src/glusterd-op-sm.c#L5681 >> _______________________________________________ >> maintainers mailing list >> maintainers at gluster.org >> https://lists.gluster.org/mailman/listinfo/maintainers >> > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ykaul at redhat.com Thu Mar 21 15:53:22 2019 From: ykaul at redhat.com (Yaniv Kaul) Date: Thu, 21 Mar 2019 17:53:22 +0200 Subject: [Gluster-devel] [Gluster-Maintainers] GF_CALLOC to GF_MALLOC conversion - is it safe? In-Reply-To: References: Message-ID: On Thu, Mar 21, 2019 at 5:43 PM Yaniv Kaul wrote: > > > On Thu, Mar 21, 2019 at 5:23 PM Nithya Balachandran > wrote: > >> >> >> On Thu, 21 Mar 2019 at 16:16, Atin Mukherjee wrote: >> >>> All, >>> >>> In the last few releases of glusterfs, with stability as a primary theme >>> of the releases, there has been lots of changes done on the code >>> optimization with an expectation that such changes will have gluster to >>> provide better performance. While many of these changes do help, but off >>> late we have started seeing some diverse effects of them, one especially >>> being the calloc to malloc conversions. While I do understand that malloc >>> syscall will eliminate the extra memset bottleneck which calloc bears, but >>> with recent kernels having in-built strong compiler optimizations I am not >>> sure whether that makes any significant difference, but as I mentioned >>> earlier certainly if this isn't done carefully it can potentially introduce >>> lot of bugs and I'm writing this email to share one of such experiences. >>> >>> Sanju & I were having troubles for last two days to figure out why >>> https://review.gluster.org/#/c/glusterfs/+/22388/ wasn't working in >>> Sanju's system but it had no problems running the same fix in my gluster >>> containers. After spending a significant amount of time, what we now >>> figured out is that a malloc call [1] (which was a calloc earlier) is the >>> culprit here. As you all can see, in this function we allocate txn_id and >>> copy the event->txn_id into it through gf_uuid_copy () . But when we were >>> debugging this step wise through gdb, txn_id wasn't exactly copied with the >>> exact event->txn_id and it had some junk values which made the >>> glusterd_clear_txn_opinfo to be invoked with a wrong txn_id later on >>> resulting the leaks to remain the same which was the original intention of >>> the fix. >>> >>> This was quite painful to debug and we had to spend some time to figure >>> this out. Considering we have converted many such calls in past, I'd urge >>> that we review all such conversions and see if there're any side effects to >>> it. Otherwise we might end up running into many potential memory related >>> bugs later on. OTOH, going forward I'd request every patch >>> owners/maintainers to pay some special attention to these conversions and >>> see they are really beneficial and error free. IMO, general guideline >>> should be - for bigger buffers, malloc would make better sense but has to >>> be done carefully, for smaller size, we stick to calloc. >>> >> >>> What do others think about it? >>> >> >> I believe that replacing calloc with malloc everywhere without adequate >> testing and review is not safe and am against doing so for the following >> reasons: >> > > No patch should get in without adequate testing and thorough review. > >> >> 1. Most of these patches have not been tested, especially the error >> paths.I have seen some that introduced issues in error scenarios with >> pointers being non-null. >> >> > You raise an interesting issue. Why are free'd memory pointers are not > NULL'ified? Why does FREE() set ptr = (void *)0xeeeeeeee and not NULL? > This is a potential cause for failure. A re-occuring FREE(NULL) is > harmless. A FREE(0xeeeeeeee) is a bit more problematic. > > >> 1. >> 2. As Raghavendra said, the code assumes that certain elements will >> be initialized to null/zero and changing that can have consequences which >> are not immediately obvious. I think it might be worthwhile to go through >> the already merged calloc->malloc patches to check error paths and so on to >> see if they are safe. >> >> > Agreed. > >> >> 1. >> 2. Making such changes to the libglusterfs code while we are >> currently working to stabilize the product is not a good idea. The patches >> take time to review and any errors introduced in the core pieces affect all >> processes and require significant effort to debug. >> >> > Let me know when we consider the project stable. I'd argue the way to > stabilize it is not stop improving it, but improving its testing. From more > tests to cover more code via more tests to more static analysis coverage, > to ensuring CI is rock-solid (inc. random errors that pop up from time to > time). Not accepting patches to master is not the right approach, unless > it's time-boxed somehow. If it is, then it means we don't trust our CI > enough, btw. > > >> 1. >> >> Yaniv, while the example you provided might make sense to change to >> malloc, a lot of the other changes, in my opinion, do not for the effort >> required. For performance testing, smallfile might be a useful tool to see >> if any of the changes make a difference. That said, I am reluctant to take >> in patches that change core code significantly without being tested or >> providing proof of benefits. >> > > Smallfile is part of CI? I am happy to see it documented @ > https://docs.gluster.org/en/latest/Administrator%20Guide/Performance%20Testing/#smallfile-distributed-io-benchmark > , so at least one can know how to execute it manually. > Following up the above link to the smallfile repo leads to 404 (I'm assuming we don't have a link checker running on our documentation, so it can break from time to time?) I assume it's https://github.com/distributed-system-analysis/smallfile ? Y. > >> We need better performance and scalability but that is going to need >> changes in our algorithms and fop handling and that is what we need to be >> working on. Such changes, when done right, will provide more benefits than >> the micro optimizations. I think it unlikely the micro optimizations will >> provide much benefit but am willing to be proven wrong if you have numbers >> that show otherwise. >> > > I call them nano-optimizations. I believe they really shave only that > much, microsecond would be great [looking at > https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html , > I'm in that ballpark figure.]. > Unless we got enough cache misses (btw, would be nice to add > likely()/unlikely() to our code, to give a hint to the compiler on code > paths). Not sure that helps that much as well, of course - same figures, I > estimate. > But I do like the fact they remove the false assumption that everything is > NULL and we don't need to worry about it - it means we need to look at our > structures (and while at it, align them!) and make sure we know what we > initialize. > > But I certainly understand the resistance and will abandon the patches if > not welcome. > Y. > > >> Regards, >> Nithya >> >> >>> >>> [1] >>> https://github.com/gluster/glusterfs/blob/master/xlators/mgmt/glusterd/src/glusterd-op-sm.c#L5681 >>> _______________________________________________ >>> maintainers mailing list >>> maintainers at gluster.org >>> https://lists.gluster.org/mailman/listinfo/maintainers >>> >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Fri Mar 22 02:21:37 2019 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Fri, 22 Mar 2019 07:51:37 +0530 Subject: [Gluster-devel] [Gluster-Maintainers] GF_CALLOC to GF_MALLOC conversion - is it safe? In-Reply-To: References: Message-ID: On Thu, Mar 21, 2019 at 9:24 PM Yaniv Kaul wrote: >> Smallfile is part of CI? I am happy to see it documented @ https://docs.gluster.org/en/latest/Administrator%20Guide/Performance%20Testing/#smallfile-distributed-io-benchmark , so at least one can know how to execute it manually. > > > Following up the above link to the smallfile repo leads to 404 (I'm assuming we don't have a link checker running on our documentation, so it can break from time to time?) Hmm... that needs to be addressed. > I assume it's https://github.com/distributed-system-analysis/smallfile ? Yes. From nbalacha at redhat.com Fri Mar 22 03:14:02 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Fri, 22 Mar 2019 08:44:02 +0530 Subject: [Gluster-devel] [Gluster-Maintainers] GF_CALLOC to GF_MALLOC conversion - is it safe? In-Reply-To: References: Message-ID: On Thu, 21 Mar 2019 at 21:14, Yaniv Kaul wrote: > > > On Thu, Mar 21, 2019 at 5:23 PM Nithya Balachandran > wrote: > >> >> >> On Thu, 21 Mar 2019 at 16:16, Atin Mukherjee wrote: >> >>> All, >>> >>> In the last few releases of glusterfs, with stability as a primary theme >>> of the releases, there has been lots of changes done on the code >>> optimization with an expectation that such changes will have gluster to >>> provide better performance. While many of these changes do help, but off >>> late we have started seeing some diverse effects of them, one especially >>> being the calloc to malloc conversions. While I do understand that malloc >>> syscall will eliminate the extra memset bottleneck which calloc bears, but >>> with recent kernels having in-built strong compiler optimizations I am not >>> sure whether that makes any significant difference, but as I mentioned >>> earlier certainly if this isn't done carefully it can potentially introduce >>> lot of bugs and I'm writing this email to share one of such experiences. >>> >>> Sanju & I were having troubles for last two days to figure out why >>> https://review.gluster.org/#/c/glusterfs/+/22388/ wasn't working in >>> Sanju's system but it had no problems running the same fix in my gluster >>> containers. After spending a significant amount of time, what we now >>> figured out is that a malloc call [1] (which was a calloc earlier) is the >>> culprit here. As you all can see, in this function we allocate txn_id and >>> copy the event->txn_id into it through gf_uuid_copy () . But when we were >>> debugging this step wise through gdb, txn_id wasn't exactly copied with the >>> exact event->txn_id and it had some junk values which made the >>> glusterd_clear_txn_opinfo to be invoked with a wrong txn_id later on >>> resulting the leaks to remain the same which was the original intention of >>> the fix. >>> >>> This was quite painful to debug and we had to spend some time to figure >>> this out. Considering we have converted many such calls in past, I'd urge >>> that we review all such conversions and see if there're any side effects to >>> it. Otherwise we might end up running into many potential memory related >>> bugs later on. OTOH, going forward I'd request every patch >>> owners/maintainers to pay some special attention to these conversions and >>> see they are really beneficial and error free. IMO, general guideline >>> should be - for bigger buffers, malloc would make better sense but has to >>> be done carefully, for smaller size, we stick to calloc. >>> >> >>> What do others think about it? >>> >> >> I believe that replacing calloc with malloc everywhere without adequate >> testing and review is not safe and am against doing so for the following >> reasons: >> > > No patch should get in without adequate testing and thorough review. > >> 1. Most of these patches have not been tested, especially the error >> paths.I have seen some that introduced issues in error scenarios with >> pointers being non-null. >> >> > You raise an interesting issue. Why are free'd memory pointers are not > NULL'ified? Why does FREE() set ptr = (void *)0xeeeeeeee and not NULL? > The problem I'm referring to here is in error paths when incompletely initialised structures are cleaned up. A non-null never allocated pointer will be attempted to be freed. This is a potential cause for failure. A re-occuring FREE(NULL) is > harmless. A FREE(0xeeeeeeee) is a bit more problematic. > > >> 1. >> 2. As Raghavendra said, the code assumes that certain elements will >> be initialized to null/zero and changing that can have consequences which >> are not immediately obvious. I think it might be worthwhile to go through >> the already merged calloc->malloc patches to check error paths and so on to >> see if they are safe. >> >> > Agreed. > >> >> 1. >> 2. Making such changes to the libglusterfs code while we are >> currently working to stabilize the product is not a good idea. The patches >> take time to review and any errors introduced in the core pieces affect all >> processes and require significant effort to debug. >> >> > Let me know when we consider the project stable. I'd argue the way to > stabilize it is not stop improving it, but improving its testing. From more > tests to cover more code via more tests to more static analysis coverage, > to ensuring CI is rock-solid (inc. random errors that pop up from time to > time). > Agreed. We need better CI coverage. More patches there would be very welcome. > Not accepting patches to master is not the right approach, unless it's > time-boxed somehow. If it is, then it means we don't trust our CI enough, > btw. > We are not blocking patches to master. We are raising concerns about patches which are likely to impact code stability (such as the malloc patches which have introduced issues in some cases) but require considerable effort to review or test. A patch that solves a known problem or a bug will always be gratefully accepted. There are several open upstream BZs for which we would welcome patches. > >> 1. >> >> Yaniv, while the example you provided might make sense to change to >> malloc, a lot of the other changes, in my opinion, do not for the effort >> required. For performance testing, smallfile might be a useful tool to see >> if any of the changes make a difference. That said, I am reluctant to take >> in patches that change core code significantly without being tested or >> providing proof of benefits. >> > > Smallfile is part of CI? I am happy to see it documented @ > https://docs.gluster.org/en/latest/Administrator%20Guide/Performance%20Testing/#smallfile-distributed-io-benchmark > , so at least one can know how to execute it manually. > >> >> We need better performance and scalability but that is going to need >> changes in our algorithms and fop handling and that is what we need to be >> working on. Such changes, when done right, will provide more benefits than >> the micro optimizations. I think it unlikely the micro optimizations will >> provide much benefit but am willing to be proven wrong if you have numbers >> that show otherwise. >> > > I call them nano-optimizations. I believe they really shave only that > much, microsecond would be great [looking at > https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html , > I'm in that ballpark figure.]. > Unless we got enough cache misses (btw, would be nice to add > likely()/unlikely() to our code, to give a hint to the compiler on code > paths). Not sure that helps that much as well, of course - same figures, I > estimate. > But I do like the fact they remove the false assumption that everything is > NULL and we don't need to worry about it - it means we need to look at our > structures (and while at it, align them!) and make sure we know what we > initialize. > Patches to align structures are welcome. The initialization bit we might want to hold off on now. Regards, Nithya > > But I certainly understand the resistance and will abandon the patches if > not welcome. > Y. > > >> Regards, >> Nithya >> >> >>> >>> [1] >>> https://github.com/gluster/glusterfs/blob/master/xlators/mgmt/glusterd/src/glusterd-op-sm.c#L5681 >>> _______________________________________________ >>> maintainers mailing list >>> maintainers at gluster.org >>> https://lists.gluster.org/mailman/listinfo/maintainers >>> >> _______________________________________________ >> Gluster-devel mailing list >> Gluster-devel at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abhishpaliwal at gmail.com Fri Mar 22 10:45:23 2019 From: abhishpaliwal at gmail.com (ABHISHEK PALIWAL) Date: Fri, 22 Mar 2019 16:15:23 +0530 Subject: [Gluster-devel] Gluster version EOL date Message-ID: Hi, As per gluster community seems the latest version is 5.5. Could any one tell me what would be the EOL date for version 5.5? is it after 12 month of release date or what? -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: From srangana at redhat.com Fri Mar 22 13:02:36 2019 From: srangana at redhat.com (Shyam Ranganathan) Date: Fri, 22 Mar 2019 09:02:36 -0400 Subject: [Gluster-devel] Gluster version EOL date In-Reply-To: References: Message-ID: <17eedca3-0736-aa33-b2a1-15db9a65f99b@redhat.com> As per the release schedule page [1] 5 will EOL when release 8 is out. Releases are 4 months apart, hence 12 months from when release 5 was out, it would be EOL'd. Major releases receive minor updates 5.x, which are bug fixes and stability fixes. These do not extend the 12 month cycle for the release. Shyam [1] Release schedule: https://www.gluster.org/release-schedule/ On 3/22/19 6:45 AM, ABHISHEK PALIWAL wrote: > Hi, > > As per gluster community seems the latest version is 5.5. Could any one > tell me what would be the EOL date for version 5.5? is it after 12 month > of release date or what? > > -- > > > > > Regards > Abhishek Paliwal > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > From sankarshan.mukhopadhyay at gmail.com Fri Mar 22 14:37:14 2019 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Fri, 22 Mar 2019 20:07:14 +0530 Subject: [Gluster-devel] requesting review available gluster* plugins in sos In-Reply-To: References: <9f73a6a1-5736-7414-bc0f-fcbafa03a1e3@redhat.com> Message-ID: On Wed, Mar 20, 2019 at 10:00 AM Atin Mukherjee wrote: > > From glusterd perspective couple of enhancements I'd propose to be added (a) to capture get-state dump and make it part of sosreport . Off late, we have seen get-state dump has been very helpful in debugging few cases apart from it's original purpose of providing source of cluster/volume information for tendrl (b) capture glusterd statedump > How large can these payloads be? One of the challenges I've heard is that users are often challenged when attempting to push large ( > 5GB) payloads making the total size of the sos archive fairly big. From atin.mukherjee83 at gmail.com Fri Mar 22 17:30:46 2019 From: atin.mukherjee83 at gmail.com (Atin Mukherjee) Date: Fri, 22 Mar 2019 23:00:46 +0530 Subject: [Gluster-devel] requesting review available gluster* plugins in sos In-Reply-To: References: <9f73a6a1-5736-7414-bc0f-fcbafa03a1e3@redhat.com> Message-ID: On Fri, 22 Mar 2019 at 20:07, Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On Wed, Mar 20, 2019 at 10:00 AM Atin Mukherjee > wrote: > > > > From glusterd perspective couple of enhancements I'd propose to be added > (a) to capture get-state dump and make it part of sosreport . Off late, we > have seen get-state dump has been very helpful in debugging few cases apart > from it's original purpose of providing source of cluster/volume > information for tendrl (b) capture glusterd statedump > > > > How large can these payloads be? One of the challenges I've heard is > that users are often challenged when attempting to push large ( > 5GB) > payloads making the total size of the sos archive fairly big. get-state and glusterd statedump are just mere text files of few KBs . Your example is referring to brick statedump file having many multiplexed bricks in a single process. > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > -- --Atin -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiubli at redhat.com Sat Mar 23 00:47:59 2019 From: xiubli at redhat.com (Xiubo Li) Date: Sat, 23 Mar 2019 08:47:59 +0800 Subject: [Gluster-devel] [Gluster-users] Network Block device (NBD) on top of glusterfs In-Reply-To: References: Message-ID: On 2019/3/21 11:29, Xiubo Li wrote: > > All, > > I am one of the contributor forgluster-block > [1] project, and also I > contribute to linux kernel andopen-iscsi > project.[2] > > NBD was around for some time, but in recent time, linux kernel?s > Network Block Device (NBD) is enhanced and made to work with more > devices and also the option to integrate with netlink is added. So, I > tried to provide a glusterfs client based NBD driver recently. Please > refergithub issue #633 > [3], and good news is > I have a working code, with most basic things @nbd-runner project > [4]. > As mentioned the nbd-runner(NBD proto) will work in the same layer with tcmu-runner(iSCSI proto), this is not trying to replace the gluster-block/ceph-iscsi-gateway great projects. It just provides the common library to do the low level stuff, like the sysfs/netlink operations and the IOs from the nbd kernel socket, and the great tcmu-runner project is doing the sysfs/uio operations and IOs from the kernel SCSI/iSCSI. The nbd-cli tool will work like the iscsi-initiator-utils, and the nbd-runner daemon will work like the tcmu-runner daemon, that's all. In tcmu-runner for different backend storages, they have separate handlers, glfs.c handler for Gluster, rbd.c handler for Ceph, etc. And what the handlers here are doing the actual IOs with the backend storage services once the IO paths setup are done by ceph-iscsi-gateway/gluster-block.... Then we can support all the kind of backend storages, like the Gluster/Ceph/Azure... as one separate handler in nbd-runner, which no need to care about the NBD low level's stuff updates and changes. Thanks. > While this email is about announcing the project, and asking for more > collaboration, I would also like to discuss more about the placement > of the project itself. Currently nbd-runner project is expected to be > shared by our friends at Ceph project too, to provide NBD driver for > Ceph. I have personally worked with some of them closely while > contributing to open-iSCSI project, and we would like to take this > project to great success. > > Now few questions: > > 1. Can I continue to usehttp://github.com/gluster/nbd-runneras home > for this project, even if its shared by other filesystem projects? > > * I personally am fine with this. > > 2. Should there be a separate organization for this repo? > > * While it may make sense in future, for now, I am not planning to > start any new thing? > > It would be great if we have some consensus on this soon as nbd-runner > is a new repository. If there are no concerns, I will continue to > contribute to the existing repository. > > Regards, > Xiubo Li (@lxbsz) > > [1] -https://github.com/gluster/gluster-block > [2] -https://github.com/open-iscsi > [3] -https://github.com/gluster/glusterfs/issues/633 > [4] -https://github.com/gluster/nbd-runner > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From jenkins at build.gluster.org Mon Mar 25 01:45:02 2019 From: jenkins at build.gluster.org (jenkins at build.gluster.org) Date: Mon, 25 Mar 2019 01:45:02 +0000 (UTC) Subject: [Gluster-devel] Weekly Untriaged Bugs Message-ID: <1651099001.24.1553478303237.JavaMail.jenkins@jenkins-el7.rht.gluster.org> [...truncated 6 lines...] https://bugzilla.redhat.com/1688226 / core: Brick Still Died After Restart Glusterd & Glusterfsd Services https://bugzilla.redhat.com/1691833 / core: Client sends 128KByte network packet for 0 length file copy https://bugzilla.redhat.com/1685023 / core: FD processes for larger files are not closed soon after FOP finished https://bugzilla.redhat.com/1690769 / core: GlusterFS 5.5 crashes in 1x4 replicate setup. https://bugzilla.redhat.com/1686396 / core: ls and rm run on contents of same directory from a single mount point results in ENOENT errors https://bugzilla.redhat.com/1683815 / core: Memory leak when peer detach fails https://bugzilla.redhat.com/1689981 / geo-replication: OSError: [Errno 1] Operation not permitted - failing with socket files? https://bugzilla.redhat.com/1683526 / glusterd: rebalance start command doesn't throw up error message if the command fails https://bugzilla.redhat.com/1690254 / glusterd: Volume create fails with "Commit failed" message if volumes is created using 3 nodes with glusterd restarts on 4th node. https://bugzilla.redhat.com/1690753 / glusterd: Volume stop when quorum not met is successful https://bugzilla.redhat.com/1686353 / libgfapi: flooding of "dict is NULL" logging https://bugzilla.redhat.com/1687063 / locks: glusterd :symbol lookup error: undefined symbol :use_spinlocks https://bugzilla.redhat.com/1690454 / posix-acl: mount-shared-storage.sh does not implement mount options https://bugzilla.redhat.com/1691617 / project-infrastructure: clang-scan tests are failing nightly. https://bugzilla.redhat.com/1691357 / project-infrastructure: core archive link from regression jobs throw not found error https://bugzilla.redhat.com/1685813 / project-infrastructure: Not able to run centos-regression getting exception error https://bugzilla.redhat.com/1691789 / project-infrastructure: rpc-statd service stops on AWS builders https://bugzilla.redhat.com/1686461 / quota: Quotad.log filled with 0-dict is not sent on wire [Invalid argument] messages https://bugzilla.redhat.com/1682925 / replicate: Gluster volumes never heal during oVirt 4.2->4.3 upgrade https://bugzilla.redhat.com/1683317 / tests: ./tests/bugs/glusterfs/bug-866459.t failing on s390x [...truncated 2 lines...] -------------- next part -------------- A non-text attachment was scrubbed... Name: build.log Type: application/octet-stream Size: 2591 bytes Desc: not available URL: From vbellur at redhat.com Mon Mar 25 06:36:46 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Sun, 24 Mar 2019 23:36:46 -0700 Subject: [Gluster-devel] [Gluster-users] Network Block device (NBD) on top of glusterfs In-Reply-To: References: Message-ID: Hi Xiubo, On Fri, Mar 22, 2019 at 5:48 PM Xiubo Li wrote: > On 2019/3/21 11:29, Xiubo Li wrote: > > All, > > I am one of the contributor for gluster-block > [1] project, and also I > contribute to linux kernel and open-iscsi > project.[2] > > NBD was around for some time, but in recent time, linux kernel?s Network > Block Device (NBD) is enhanced and made to work with more devices and also > the option to integrate with netlink is added. So, I tried to provide a > glusterfs client based NBD driver recently. Please refer github issue #633 > [3], and good news is I > have a working code, with most basic things @ nbd-runner project > [4]. > > This is nice. Thank you for your work! > As mentioned the nbd-runner(NBD proto) will work in the same layer with > tcmu-runner(iSCSI proto), this is not trying to replace the > gluster-block/ceph-iscsi-gateway great projects. > > It just provides the common library to do the low level stuff, like the > sysfs/netlink operations and the IOs from the nbd kernel socket, and the > great tcmu-runner project is doing the sysfs/uio operations and IOs from > the kernel SCSI/iSCSI. > > The nbd-cli tool will work like the iscsi-initiator-utils, and the > nbd-runner daemon will work like the tcmu-runner daemon, that's all. > Do you have thoughts on how nbd-runner currently differs or would differ from tcmu-runner? It might be useful to document the differences in github (or elsewhere) so that users can make an informed choice between nbd-runner & tcmu-runner. In tcmu-runner for different backend storages, they have separate handlers, > glfs.c handler for Gluster, rbd.c handler for Ceph, etc. And what the > handlers here are doing the actual IOs with the backend storage services > once the IO paths setup are done by ceph-iscsi-gateway/gluster-block.... > > Then we can support all the kind of backend storages, like the > Gluster/Ceph/Azure... as one separate handler in nbd-runner, which no need > to care about the NBD low level's stuff updates and changes. > Given that the charter for this project is to support multiple backend storage projects, would not it be better to host the project in the github repository associated with nbd [5]? Doing it that way could provide a more neutral (as perceived by users) venue for hosting nbd-runner and help you in getting more adoption for your work. Thanks, Vijay [5] https://github.com/NetworkBlockDevice/nbd > Thanks. > > > While this email is about announcing the project, and asking for more > collaboration, I would also like to discuss more about the placement of the > project itself. Currently nbd-runner project is expected to be shared by > our friends at Ceph project too, to provide NBD driver for Ceph. I have > personally worked with some of them closely while contributing to > open-iSCSI project, and we would like to take this project to great success. > > Now few questions: > > 1. Can I continue to use http://github.com/gluster/nbd-runner as home > for this project, even if its shared by other filesystem projects? > > > - I personally am fine with this. > > > 1. Should there be a separate organization for this repo? > > > - While it may make sense in future, for now, I am not planning to > start any new thing? > > It would be great if we have some consensus on this soon as nbd-runner is > a new repository. If there are no concerns, I will continue to contribute > to the existing repository. > > Regards, > Xiubo Li (@lxbsz) > > [1] - https://github.com/gluster/gluster-block > [2] - https://github.com/open-iscsi > [3] - https://github.com/gluster/glusterfs/issues/633 > [4] - https://github.com/gluster/nbd-runner > > _______________________________________________ > Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From atumball at redhat.com Mon Mar 25 09:18:59 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Mon, 25 Mar 2019 14:48:59 +0530 Subject: [Gluster-devel] GlusterFS v7.0 (and v8.0) roadmap discussion Message-ID: Hello Gluster Members, We are now done with glusterfs-6.0 release, and the next up is glusterfs-7.0. But considering for many 'initiatives', 3-4 months are not enough time to complete the tasks, we would like to call for a road-map discussion meeting for calendar year 2019 (covers both glusterfs-7.0, and 8.0). It would be good to use the meeting slot of community meeting for this. While talking to team locally, I compiled a presentation here: < https://docs.google.com/presentation/d/1rtn38S4YBe77KK5IjczWmoAR-ZSO-i3tNHg9pAH8Wt8/edit?usp=sharing>, please go through and let me know what more can be added, or what can be dropped? We can start having discussions in https://hackmd.io/jlnWqzwCRvC9uoEU2h01Zw Regards, Amar -------------- next part -------------- An HTML attachment was scrubbed... URL: From atumball at redhat.com Mon Mar 25 09:25:27 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Mon, 25 Mar 2019 14:55:27 +0530 Subject: [Gluster-devel] Proposal: Changes in Gluster Community meetings Message-ID: All, We currently have 3 meetings which are public: 1. Maintainer's Meeting - Runs once in 2 weeks (on Mondays), and current attendance is around 3-5 on an avg, and not much is discussed. - Without majority attendance, we can't take any decisions too. 2. Community meeting - Supposed to happen on #gluster-meeting, every 2 weeks, and is the only meeting which is for 'Community/Users'. Others are for developers as of now. Sadly attendance is getting closer to 0 in recent times. 3. GCS meeting - We started it as an effort inside Red Hat gluster team, and opened it up for community from Jan 2019, but the attendance was always from RHT members, and haven't seen any traction from wider group. So, I have a proposal to call out for cancelling all these meeting, and keeping just 1 weekly 'Community' meeting, where even topics related to maintainers and GCS and other projects can be discussed. I have a template of a draft template @ https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g Please feel free to suggest improvements, both in agenda and in timings. So, we can have more participation from members of community, which allows more user - developer interactions, and hence quality of project. Waiting for feedbacks, Regards, Amar -------------- next part -------------- An HTML attachment was scrubbed... URL: From xiubli at redhat.com Mon Mar 25 10:32:25 2019 From: xiubli at redhat.com (Xiubo Li) Date: Mon, 25 Mar 2019 18:32:25 +0800 Subject: [Gluster-devel] [Gluster-users] Network Block device (NBD) on top of glusterfs In-Reply-To: References: Message-ID: On 2019/3/25 14:36, Vijay Bellur wrote: > > Hi Xiubo, > > On Fri, Mar 22, 2019 at 5:48 PM Xiubo Li > wrote: > > On 2019/3/21 11:29, Xiubo Li wrote: >> >> All, >> >> I am one of the contributor forgluster-block >> [1] project, and also I >> contribute to linux kernel andopen-iscsi >> project.[2] >> >> NBD was around for some time, but in recent time, linux kernel?s >> Network Block Device (NBD) is enhanced and made to work with more >> devices and also the option to integrate with netlink is added. >> So, I tried to provide a glusterfs client based NBD driver >> recently. Please refergithub issue #633 >> [3], and good >> news is I have a working code, with most basic things @nbd-runner >> project [4]. >> > > This is nice. Thank you for your work! > > As mentioned the nbd-runner(NBD proto) will work in the same layer > with tcmu-runner(iSCSI proto), this is not trying to replace the > gluster-block/ceph-iscsi-gateway great projects. > > It just provides the common library to do the low level stuff, > like the sysfs/netlink operations and the IOs from the nbd kernel > socket, and the great tcmu-runner project is doing the sysfs/uio > operations and IOs from the kernel SCSI/iSCSI. > > The nbd-cli tool will work like the iscsi-initiator-utils, and the > nbd-runner daemon will work like the tcmu-runner daemon, that's all. > > > Do you have thoughts on how nbd-runner currently differs or would > differ from tcmu-runner? It might be useful to document the > differences in github (or elsewhere) so that users can make an > informed choice between nbd-runner & tcmu-runner. Yeah, this makes sense and I will figure it out in the github. Currently for the open-iscsi/tcmu-runner, there are already many existing tools to help product it, and for NBD we may need to implement them, correct me if I am wrong here :-) > > In tcmu-runner for different backend storages, they have separate > handlers, glfs.c handler for Gluster, rbd.c handler for Ceph, etc. > And what the handlers here are doing the actual IOs with the > backend storage services once the IO paths setup are done by > ceph-iscsi-gateway/gluster-block.... > > Then we can support all the kind of backend storages, like the > Gluster/Ceph/Azure... as one separate handler in nbd-runner, which > no need to care about the NBD low level's stuff updates and changes. > > > Given that the charter for this project is to support multiple backend > storage projects, would not it be better to host the project in the > github repository associated with nbd [5]? Doing it that way could > provide a more neutral (as perceived by users) venue for hosting > nbd-runner and help you in getting more adoption for your work. > This is a good idea, I will try to push this forward. Thanks very much Vijay. BRs Xiubo Li > Thanks, > Vijay > > [5] https://github.com/NetworkBlockDevice/nbd > > > Thanks. > > >> While this email is about announcing the project, and asking for >> more collaboration, I would also like to discuss more about the >> placement of the project itself. Currently nbd-runner project is >> expected to be shared by our friends at Ceph project too, to >> provide NBD driver for Ceph. I have personally worked with some >> of them closely while contributing to open-iSCSI project, and we >> would like to take this project to great success. >> >> Now few questions: >> >> 1. Can I continue to usehttp://github.com/gluster/nbd-runneras >> home for this project, even if its shared by other filesystem >> projects? >> >> * I personally am fine with this. >> >> 2. Should there be a separate organization for this repo? >> >> * While it may make sense in future, for now, I am not planning >> to start any new thing? >> >> It would be great if we have some consensus on this soon as >> nbd-runner is a new repository. If there are no concerns, I will >> continue to contribute to the existing repository. >> >> Regards, >> Xiubo Li (@lxbsz) >> >> [1] -https://github.com/gluster/gluster-block >> [2] -https://github.com/open-iscsi >> [3] -https://github.com/gluster/glusterfs/issues/633 >> [4] -https://github.com/gluster/nbd-runner >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srangana at redhat.com Mon Mar 25 14:28:40 2019 From: srangana at redhat.com (Shyam Ranganathan) Date: Mon, 25 Mar 2019 10:28:40 -0400 Subject: [Gluster-devel] Announcing Gluster Release 6 Message-ID: The Gluster community is pleased to announce the release of 6.0, our latest release. This is a major release that includes a range of code improvements and stability fixes along with a few features as noted below. A selection of the key features and bugs addressed are documented in this [1] page. Announcements: 1. Releases that receive maintenance updates post release 6 are, 4.1 and 5 [2] 2. Release 6 will receive maintenance updates around the 10th of every month for the first 3 months post release (i.e Apr'19, May'19, Jun'19). Post the initial 3 months, it will receive maintenance updates every 2 months till EOL. [3] A series of features/xlators have been deprecated in release 6 as follows, for upgrade procedures from volumes that use these features to release 6 refer to the release 6 upgrade guide [4]. Features deprecated: - Block device (bd) xlator - Decompounder feature - Crypt xlator - Symlink-cache xlator - Stripe feature - Tiering support (tier xlator and changetimerecorder) Highlights of this release are: - Several stability fixes addressing, coverity, clang-scan, address sanitizer and valgrind reported issues - Removal of unused and hence, deprecated code and features - Client side inode garbage collection - This release addresses one of the major concerns regarding FUSE mount process memory footprint, by introducing client side inode garbage collection - Performance Improvements - "--auto-invalidation" on FUSE mounts to leverage kernel page cache more effectively Bugs addressed are provided towards the end, in the release notes [1] Thank you, Gluster community References: [1] Release notes: https://docs.gluster.org/en/latest/release-notes/6.0/ [2] Release schedule: https://www.gluster.org/release-schedule/ [3] Gluster release cadence and version changes: https://lists.gluster.org/pipermail/announce/2018-July/000103.html [4] Upgrade guide to release-6: https://docs.gluster.org/en/latest/Upgrade-Guide/upgrade_to_6/ From atumball at redhat.com Mon Mar 25 15:23:43 2019 From: atumball at redhat.com (Amar Tumballi Suryanarayan) Date: Mon, 25 Mar 2019 20:53:43 +0530 Subject: [Gluster-devel] [Gluster-users] Proposal: Changes in Gluster Community meetings In-Reply-To: <62104B6F-99CF-4C22-80FC-9C177F73E897@onholyground.com> References: <62104B6F-99CF-4C22-80FC-9C177F73E897@onholyground.com> Message-ID: Thanks for the feedback Darrell, The new proposal is to have one in North America 'morning' time. (10AM PST), And another in ASIA day time, which is evening 7pm/6pm in Australia, 9pm Newzealand, 5pm Tokyo, 4pm Beijing. For example, if we choose Every other Tuesday for meeting, and 1st of the month is Tuesday, we would have North America time for 1st, and on 15th it would be ASIA/Pacific time. Hopefully, this way, we can cover all the timezones, and meeting minutes would be committed to github repo, so that way, it will be easier for everyone to be aware of what is happening. Regards, Amar On Mon, Mar 25, 2019 at 8:40 PM Darrell Budic wrote: > As a user, I?d like to visit more of these, but the time slot is my 3AM. > Any possibility for a rolling schedule (move meeting +6 hours each week > with rolling attendance from maintainers?) or an occasional regional > meeting 12 hours opposed to the one you?re proposing? > > -Darrell > > On Mar 25, 2019, at 4:25 AM, Amar Tumballi Suryanarayan < > atumball at redhat.com> wrote: > > All, > > We currently have 3 meetings which are public: > > 1. Maintainer's Meeting > > - Runs once in 2 weeks (on Mondays), and current attendance is around 3-5 > on an avg, and not much is discussed. > - Without majority attendance, we can't take any decisions too. > > 2. Community meeting > > - Supposed to happen on #gluster-meeting, every 2 weeks, and is the only > meeting which is for 'Community/Users'. Others are for developers as of > now. > Sadly attendance is getting closer to 0 in recent times. > > 3. GCS meeting > > - We started it as an effort inside Red Hat gluster team, and opened it up > for community from Jan 2019, but the attendance was always from RHT > members, and haven't seen any traction from wider group. > > So, I have a proposal to call out for cancelling all these meeting, and > keeping just 1 weekly 'Community' meeting, where even topics related to > maintainers and GCS and other projects can be discussed. > > I have a template of a draft template @ > https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g > > Please feel free to suggest improvements, both in agenda and in timings. > So, we can have more participation from members of community, which allows > more user - developer interactions, and hence quality of project. > > Waiting for feedbacks, > > Regards, > Amar > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > > -- Amar Tumballi (amarts) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ravishankar at redhat.com Tue Mar 26 04:12:30 2019 From: ravishankar at redhat.com (Ravishankar N) Date: Tue, 26 Mar 2019 09:42:30 +0530 Subject: [Gluster-devel] gfapi: add function to set client-pid In-Reply-To: <20190312105041.GI3535@ndevos-x270> References: <20190312105041.GI3535@ndevos-x270> Message-ID: On 12/03/19 4:20 PM, Niels de Vos wrote: > On Tue, Mar 12, 2019 at 03:00:27PM +0530, Ravishankar N wrote: >> Hello, >> >> I'm planning to expose setting client-pid for gfapi clients via a new api, >> something like `glfs_set_client_pid (fs, pid)`. >> The functionality already exists for fuse mounts via the --client-pid=$PID >> option, where the value is captured in >> glusterfs_ctx_t->cmd_args->client_pid. >> >> Background: >> >> If the glusterfs eventing framework is enabled, AFR sends child-up/child >> down events (via the gf_event() call) in the notify code path whenever there >> is a connect/disconnect at AFR level. While this is okay for normal client >> processes, it does not make much sense if the event is coming from say >> glfsheal, which is a gfapi based program (having the AFR xlator) that is >> invoked when you run the heal info set of commands. Many applications >> periodically run heal info to monitor the heals and display it on the >> dashboard (like tendryl), leading to a flood of child up/ down messages to >> the application monitoring these events. >> >> We need to add a unique key=value to all such gf_event() calls in AFR, based >> on which the consumer of the events can decide to ignore them if needed. >> This key-value can be client-pid=$PID, where PID can be >> GF_CLIENT_PID_SELF_HEALD for selfheal daemon, GF_CLIENT_PID_GLFS_HEAL for >> glfsheal etc (these values are already defined in the code). This is why we >> need a way to set the client-pid for gfapi clients as well. >> >> Another approach would be to add an xlator option (say 'client-name') >> specific to AFR? and use that as the key-value pair but it seems to be an >> overkill to do that just for the sake of eventing purposes. Besides, the pid >> approach can also be extended to other gluster processes like rebalance, shd >> and other daemons where AFR is loaded but AFR child-up/down events from it >> are not of any particular interest.These daemons will now have to be spawned >> by glusterd with the --client-pid option. > Sounds good to me. This probably should be a function that is not > available to all gfapi consumers, so please use api/src/glfs-internal.h > for that. With clear documentation as written in the email, it should be > obvious that only Gluster internal processes may use it. > > Adding the integration mailinglist on CC, as that is where all > discussions around gfapi should be archived. Hi Niels, The gfapi patch [1] is awaiting your review. Thanks, Ravi [1] https://review.gluster.org/#/c/glusterfs/+/22368/ > > Thanks, > Niels From nux at li.nux.ro Tue Mar 26 11:21:54 2019 From: nux at li.nux.ro (Nux!) Date: Tue, 26 Mar 2019 11:21:54 +0000 (GMT) Subject: [Gluster-devel] Prioritise local bricks for IO? Message-ID: <29221907.583.1553599314586.JavaMail.zimbra@li.nux.ro> Hello, I'm trying to set up a distributed backup storage (no replicas), but I'd like to prioritise the local bricks for any IO done on the volume. This will be a backup stor, so in other words, I'd like the files to be written locally if there is space, so as to save the NICs for other traffic. Anyone knows how this might be achievable, if at all? -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro From vbellur at redhat.com Tue Mar 26 21:52:33 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Tue, 26 Mar 2019 14:52:33 -0700 Subject: [Gluster-devel] [Gluster-Maintainers] GF_CALLOC to GF_MALLOC conversion - is it safe? In-Reply-To: References: Message-ID: On Thu, Mar 21, 2019 at 8:44 AM Yaniv Kaul wrote: > > > On Thu, Mar 21, 2019 at 5:23 PM Nithya Balachandran > wrote: > >> >> >> On Thu, 21 Mar 2019 at 16:16, Atin Mukherjee wrote: >> >>> All, >>> >>> In the last few releases of glusterfs, with stability as a primary theme >>> of the releases, there has been lots of changes done on the code >>> optimization with an expectation that such changes will have gluster to >>> provide better performance. While many of these changes do help, but off >>> late we have started seeing some diverse effects of them, one especially >>> being the calloc to malloc conversions. While I do understand that malloc >>> syscall will eliminate the extra memset bottleneck which calloc bears, but >>> with recent kernels having in-built strong compiler optimizations I am not >>> sure whether that makes any significant difference, but as I mentioned >>> earlier certainly if this isn't done carefully it can potentially introduce >>> lot of bugs and I'm writing this email to share one of such experiences. >>> >>> Sanju & I were having troubles for last two days to figure out why >>> https://review.gluster.org/#/c/glusterfs/+/22388/ wasn't working in >>> Sanju's system but it had no problems running the same fix in my gluster >>> containers. After spending a significant amount of time, what we now >>> figured out is that a malloc call [1] (which was a calloc earlier) is the >>> culprit here. As you all can see, in this function we allocate txn_id and >>> copy the event->txn_id into it through gf_uuid_copy () . But when we were >>> debugging this step wise through gdb, txn_id wasn't exactly copied with the >>> exact event->txn_id and it had some junk values which made the >>> glusterd_clear_txn_opinfo to be invoked with a wrong txn_id later on >>> resulting the leaks to remain the same which was the original intention of >>> the fix. >>> >>> This was quite painful to debug and we had to spend some time to figure >>> this out. Considering we have converted many such calls in past, I'd urge >>> that we review all such conversions and see if there're any side effects to >>> it. Otherwise we might end up running into many potential memory related >>> bugs later on. OTOH, going forward I'd request every patch >>> owners/maintainers to pay some special attention to these conversions and >>> see they are really beneficial and error free. IMO, general guideline >>> should be - for bigger buffers, malloc would make better sense but has to >>> be done carefully, for smaller size, we stick to calloc. >>> >> >>> What do others think about it? >>> >> >> I believe that replacing calloc with malloc everywhere without adequate >> testing and review is not safe and am against doing so for the following >> reasons: >> > > No patch should get in without adequate testing and thorough review. > There are lots of interesting points to glean in this thread. However, this particular one caught my attention. How about we introduce a policy that no patch gets merged unless it is thoroughly tested? The onus would be on the developer to provide a .t test case to show completeness in the testing of that patch. If the developer does not or cannot for any reason, we could have the maintainer run tests and add a note in gerrit explaining the tests run. This would provide more assurance about the patches being tested before getting merged. Obviously, patches that fix typos or that cannot affect any functionality need not be subject to this policy. As far as review thoroughness is concerned, it might be better to mandate acks from respective maintainers before merging a patch that affects several components. More eyeballs that specialize in particular component(s) will hopefully catch some of these issues during the review phase. Thanks, Vijay -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Wed Mar 27 01:48:07 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Wed, 27 Mar 2019 07:18:07 +0530 Subject: [Gluster-devel] POSIX locks and disconnections between clients and bricks Message-ID: All, Glusterfs cleans up POSIX locks held on an fd when the client/mount through which those locks are held disconnects from bricks/server. This helps Glusterfs to not run into a stale lock problem later (For eg., if application unlocks while the connection was still down). However, this means the lock is no longer exclusive as other applications/clients can acquire the same lock. To communicate that locks are no longer valid, we are planning to mark the fd (which has POSIX locks) bad on a disconnect so that any future operations on that fd will fail, forcing the application to re-open the fd and re-acquire locks it needs [1]. Note that with AFR/replicate in picture we can prevent errors to application as long as Quorum number of children "never ever" lost connection with bricks after locks have been acquired. I am using the term "never ever" as locks are not healed back after re-connection and hence first disconnect would've marked the fd bad and the fd remains so even after re-connection happens. So, its not just Quorum number of children "currently online", but Quorum number of children "never having disconnected with bricks after locks are acquired". However, this use case is not affected if the application don't acquire any POSIX locks. So, I am interested in knowing * whether your use cases use POSIX locks? * Is it feasible for your application to re-open fds and re-acquire locks on seeing EBADFD errors? [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 regards, Raghavendra -------------- next part -------------- An HTML attachment was scrubbed... URL: From nux at li.nux.ro Wed Mar 27 07:01:58 2019 From: nux at li.nux.ro (Lucian) Date: Wed, 27 Mar 2019 07:01:58 +0000 Subject: [Gluster-devel] [Gluster-users] Prioritise local bricks for IO? In-Reply-To: References: <29221907.583.1553599314586.JavaMail.zimbra@li.nux.ro> Message-ID: Oh, that's just what the doctor ordered! Hope it works, thanks On 27 March 2019 03:15:57 GMT, Vlad Kopylov wrote: >I don't remember if it still in works >NUFA >https://github.com/gluster/glusterfs-specs/blob/master/done/Features/nufa.md > >v > >On Tue, Mar 26, 2019 at 7:27 AM Nux! wrote: > >> Hello, >> >> I'm trying to set up a distributed backup storage (no replicas), but >I'd >> like to prioritise the local bricks for any IO done on the volume. >> This will be a backup stor, so in other words, I'd like the files to >be >> written locally if there is space, so as to save the NICs for other >traffic. >> >> Anyone knows how this might be achievable, if at all? >> >> -- >> Sent from the Delta quadrant using Borg technology! >> >> Nux! >> www.nux.ro >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users >> -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jahernan at redhat.com Wed Mar 27 07:25:57 2019 From: jahernan at redhat.com (Xavi Hernandez) Date: Wed, 27 Mar 2019 08:25:57 +0100 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: Hi Raghavendra, On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa wrote: > All, > > Glusterfs cleans up POSIX locks held on an fd when the client/mount > through which those locks are held disconnects from bricks/server. This > helps Glusterfs to not run into a stale lock problem later (For eg., if > application unlocks while the connection was still down). However, this > means the lock is no longer exclusive as other applications/clients can > acquire the same lock. To communicate that locks are no longer valid, we > are planning to mark the fd (which has POSIX locks) bad on a disconnect so > that any future operations on that fd will fail, forcing the application to > re-open the fd and re-acquire locks it needs [1]. > Wouldn't it be better to retake the locks when the brick is reconnected if the lock is still in use ? BTW, the referenced bug is not public. Should we open another bug to track this ? > > Note that with AFR/replicate in picture we can prevent errors to > application as long as Quorum number of children "never ever" lost > connection with bricks after locks have been acquired. I am using the term > "never ever" as locks are not healed back after re-connection and hence > first disconnect would've marked the fd bad and the fd remains so even > after re-connection happens. So, its not just Quorum number of children > "currently online", but Quorum number of children "never having > disconnected with bricks after locks are acquired". > I think this requisite is not feasible. In a distributed file system, sooner or later all bricks will be disconnected. It could be because of failures or because an upgrade is done, but it will happen. The difference here is how long are fd's kept open. If applications open and close files frequently enough (i.e. the fd is not kept open more time than it takes to have more than Quorum bricks disconnected) then there's no problem. The problem can only appear on applications that open files for a long time and also use posix locks. In this case, the only good solution I see is to retake the locks on brick reconnection. > However, this use case is not affected if the application don't acquire > any POSIX locks. So, I am interested in knowing > * whether your use cases use POSIX locks? > * Is it feasible for your application to re-open fds and re-acquire locks > on seeing EBADFD errors? > I think that many applications are not prepared to handle that. Xavi > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 > > regards, > Raghavendra > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndevos at redhat.com Wed Mar 27 08:54:21 2019 From: ndevos at redhat.com (Niels de Vos) Date: Wed, 27 Mar 2019 09:54:21 +0100 Subject: [Gluster-devel] [Gluster-Maintainers] GF_CALLOC to GF_MALLOC conversion - is it safe? In-Reply-To: References: Message-ID: <20190327085421.GG2684@ndevos-x270.lan.nixpanic.net> On Tue, Mar 26, 2019 at 02:52:33PM -0700, Vijay Bellur wrote: > On Thu, Mar 21, 2019 at 8:44 AM Yaniv Kaul wrote: > > > > > > > On Thu, Mar 21, 2019 at 5:23 PM Nithya Balachandran > > wrote: > > > >> > >> > >> On Thu, 21 Mar 2019 at 16:16, Atin Mukherjee wrote: > >> > >>> All, > >>> > >>> In the last few releases of glusterfs, with stability as a primary theme > >>> of the releases, there has been lots of changes done on the code > >>> optimization with an expectation that such changes will have gluster to > >>> provide better performance. While many of these changes do help, but off > >>> late we have started seeing some diverse effects of them, one especially > >>> being the calloc to malloc conversions. While I do understand that malloc > >>> syscall will eliminate the extra memset bottleneck which calloc bears, but > >>> with recent kernels having in-built strong compiler optimizations I am not > >>> sure whether that makes any significant difference, but as I mentioned > >>> earlier certainly if this isn't done carefully it can potentially introduce > >>> lot of bugs and I'm writing this email to share one of such experiences. > >>> > >>> Sanju & I were having troubles for last two days to figure out why > >>> https://review.gluster.org/#/c/glusterfs/+/22388/ wasn't working in > >>> Sanju's system but it had no problems running the same fix in my gluster > >>> containers. After spending a significant amount of time, what we now > >>> figured out is that a malloc call [1] (which was a calloc earlier) is the > >>> culprit here. As you all can see, in this function we allocate txn_id and > >>> copy the event->txn_id into it through gf_uuid_copy () . But when we were > >>> debugging this step wise through gdb, txn_id wasn't exactly copied with the > >>> exact event->txn_id and it had some junk values which made the > >>> glusterd_clear_txn_opinfo to be invoked with a wrong txn_id later on > >>> resulting the leaks to remain the same which was the original intention of > >>> the fix. > >>> > >>> This was quite painful to debug and we had to spend some time to figure > >>> this out. Considering we have converted many such calls in past, I'd urge > >>> that we review all such conversions and see if there're any side effects to > >>> it. Otherwise we might end up running into many potential memory related > >>> bugs later on. OTOH, going forward I'd request every patch > >>> owners/maintainers to pay some special attention to these conversions and > >>> see they are really beneficial and error free. IMO, general guideline > >>> should be - for bigger buffers, malloc would make better sense but has to > >>> be done carefully, for smaller size, we stick to calloc. > >>> > >> > >>> What do others think about it? > >>> > >> > >> I believe that replacing calloc with malloc everywhere without adequate > >> testing and review is not safe and am against doing so for the following > >> reasons: > >> > > > > No patch should get in without adequate testing and thorough review. > > > > > There are lots of interesting points to glean in this thread. However, this > particular one caught my attention. How about we introduce a policy that no > patch gets merged unless it is thoroughly tested? The onus would be on the > developer to provide a .t test case to show completeness in the testing of > that patch. If the developer does not or cannot for any reason, we could > have the maintainer run tests and add a note in gerrit explaining the tests > run. This would provide more assurance about the patches being tested > before getting merged. Obviously, patches that fix typos or that cannot > affect any functionality need not be subject to this policy. > > As far as review thoroughness is concerned, it might be better to mandate > acks from respective maintainers before merging a patch that affects > several components. More eyeballs that specialize in particular > component(s) will hopefully catch some of these issues during the review > phase. Both of these points have always been strongly encouraged. They are also documented in the https://docs.gluster.org/en/latest/Contributors-Guide/Guidelines-For-Maintainers/ https://github.com/gluster/glusterdocs/blob/master/docs/Contributors-Guide/Guidelines-For-Maintainers.md (formatting is broken in the 1st link, but I dont know how to fix it) We probably need to apply our own guidelines a little better, and remember developers that > 90% of the patch(series) should come with a .t file or added test in an existing one. And a big +1 for getting reviews or at least some involvement of the component maintainers. Niels From skoduri at redhat.com Wed Mar 27 09:53:35 2019 From: skoduri at redhat.com (Soumya Koduri) Date: Wed, 27 Mar 2019 15:23:35 +0530 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On 3/27/19 12:55 PM, Xavi Hernandez wrote: > Hi?Raghavendra, > > On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa > > wrote: > > All, > > Glusterfs cleans up POSIX locks held on an fd when the client/mount > through which those locks are held disconnects from bricks/server. > This helps Glusterfs to not run into a stale lock problem later (For > eg., if application unlocks while the connection was still down). > However, this means the lock is no longer exclusive as other > applications/clients can acquire the same lock. To communicate that > locks are no longer valid, we are planning to mark the fd (which has > POSIX locks) bad on a disconnect so that any future operations on > that fd will fail, forcing the application to re-open the fd and > re-acquire locks it needs [1]. > > > Wouldn't it be better to retake the locks when the brick is reconnected > if the lock is still in use ? > > BTW, the referenced bug is not public. Should we open another bug to > track this ? > > > Note that with AFR/replicate in picture we can prevent errors to > application as long as Quorum number of children "never ever" lost > connection with bricks after locks have been acquired. I am using > the term "never ever" as locks are not healed back after > re-connection and hence first disconnect would've marked the fd bad > and the fd remains so even after re-connection happens. So, its not > just Quorum number of children "currently online", but Quorum number > of children "never having disconnected with bricks after locks are > acquired". > > > I think this requisite is not feasible. In a distributed file system, > sooner or later all bricks will be disconnected. It could be because of > failures or because an upgrade is done, but it will happen. > > The difference here is how long are fd's kept open. If applications open > and close files frequently enough (i.e. the fd is not kept open more > time than it takes to have more than Quorum bricks disconnected) then > there's no problem. The problem can only appear on applications that > open files for a long time and also use posix locks. In this case, the > only good solution I see is to retake the locks on brick reconnection. > > > However, this use case is not affected if the application don't > acquire any POSIX locks. So, I am interested in knowing > * whether your use cases use POSIX locks? > * Is it feasible for your application to re-open fds and re-acquire > locks on seeing EBADFD errors? > > > I think that many applications are not prepared to handle that. +1 to all the points mentioned by Xavi. This has been day-1 issue for all the applications using locks (like NFS-Ganesha and Samba). Not many applications re-open and re-acquire the locks. On receiving EBADFD, that error is most likely propagated to application clients. Agree with Xavi that its better to heal/re-acquire the locks on brick reconnects before it accepts any fresh requests. I also suggest to have this healing mechanism generic enough (if possible) to heal any server-side state (like upcall, leases etc). Thanks, Soumya > > Xavi > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 > > regards, > Raghavendra > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > From rgowdapp at redhat.com Wed Mar 27 10:52:45 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Wed, 27 Mar 2019 16:22:45 +0530 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez wrote: > Hi Raghavendra, > > On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa > wrote: > >> All, >> >> Glusterfs cleans up POSIX locks held on an fd when the client/mount >> through which those locks are held disconnects from bricks/server. This >> helps Glusterfs to not run into a stale lock problem later (For eg., if >> application unlocks while the connection was still down). However, this >> means the lock is no longer exclusive as other applications/clients can >> acquire the same lock. To communicate that locks are no longer valid, we >> are planning to mark the fd (which has POSIX locks) bad on a disconnect so >> that any future operations on that fd will fail, forcing the application to >> re-open the fd and re-acquire locks it needs [1]. >> > > Wouldn't it be better to retake the locks when the brick is reconnected if > the lock is still in use ? > There is also a possibility that clients may never reconnect. That's the primary reason why bricks assume the worst (client will not reconnect) and cleanup the locks. > BTW, the referenced bug is not public. Should we open another bug to track > this ? > I've just opened up the comment to give enough context. I'll open a bug upstream too. > > >> >> Note that with AFR/replicate in picture we can prevent errors to >> application as long as Quorum number of children "never ever" lost >> connection with bricks after locks have been acquired. I am using the term >> "never ever" as locks are not healed back after re-connection and hence >> first disconnect would've marked the fd bad and the fd remains so even >> after re-connection happens. So, its not just Quorum number of children >> "currently online", but Quorum number of children "never having >> disconnected with bricks after locks are acquired". >> > > I think this requisite is not feasible. In a distributed file system, > sooner or later all bricks will be disconnected. It could be because of > failures or because an upgrade is done, but it will happen. > > The difference here is how long are fd's kept open. If applications open > and close files frequently enough (i.e. the fd is not kept open more time > than it takes to have more than Quorum bricks disconnected) then there's no > problem. The problem can only appear on applications that open files for a > long time and also use posix locks. In this case, the only good solution I > see is to retake the locks on brick reconnection. > Agree. But lock-healing should be done only by HA layers like AFR/EC as only they know whether there are enough online bricks to have prevented any conflicting lock. Protocol/client itself doesn't have enough information to do that. If its a plain distribute, I don't see a way to heal locks without loosing the property of exclusivity of locks. What I proposed is a short term solution. mid to long term solution should be lock healing feature implemented in AFR/EC. In fact I had this conversation with +Karampuri, Pranith before posting this msg to ML. > >> However, this use case is not affected if the application don't acquire >> any POSIX locks. So, I am interested in knowing >> * whether your use cases use POSIX locks? >> * Is it feasible for your application to re-open fds and re-acquire locks >> on seeing EBADFD errors? >> > > I think that many applications are not prepared to handle that. > I too suspected that and in fact not too happy with the solution. But went ahead with this mail as I heard implementing lock-heal in AFR will take time and hence there are no alternative short term solutions. > Xavi > > >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >> >> regards, >> Raghavendra >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Wed Mar 27 10:54:12 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Wed, 27 Mar 2019 16:24:12 +0530 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Wed, Mar 27, 2019 at 4:22 PM Raghavendra Gowdappa wrote: > > > On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez > wrote: > >> Hi Raghavendra, >> >> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa >> wrote: >> >>> All, >>> >>> Glusterfs cleans up POSIX locks held on an fd when the client/mount >>> through which those locks are held disconnects from bricks/server. This >>> helps Glusterfs to not run into a stale lock problem later (For eg., if >>> application unlocks while the connection was still down). However, this >>> means the lock is no longer exclusive as other applications/clients can >>> acquire the same lock. To communicate that locks are no longer valid, we >>> are planning to mark the fd (which has POSIX locks) bad on a disconnect so >>> that any future operations on that fd will fail, forcing the application to >>> re-open the fd and re-acquire locks it needs [1]. >>> >> >> Wouldn't it be better to retake the locks when the brick is reconnected >> if the lock is still in use ? >> > > There is also a possibility that clients may never reconnect. That's the > primary reason why bricks assume the worst (client will not reconnect) and > cleanup the locks. > > >> BTW, the referenced bug is not public. Should we open another bug to >> track this ? >> > > I've just opened up the comment to give enough context. I'll open a bug > upstream too. > > >> >> >>> >>> Note that with AFR/replicate in picture we can prevent errors to >>> application as long as Quorum number of children "never ever" lost >>> connection with bricks after locks have been acquired. I am using the term >>> "never ever" as locks are not healed back after re-connection and hence >>> first disconnect would've marked the fd bad and the fd remains so even >>> after re-connection happens. So, its not just Quorum number of children >>> "currently online", but Quorum number of children "never having >>> disconnected with bricks after locks are acquired". >>> >> >> I think this requisite is not feasible. In a distributed file system, >> sooner or later all bricks will be disconnected. It could be because of >> failures or because an upgrade is done, but it will happen. >> >> The difference here is how long are fd's kept open. If applications open >> and close files frequently enough (i.e. the fd is not kept open more time >> than it takes to have more than Quorum bricks disconnected) then there's no >> problem. The problem can only appear on applications that open files for a >> long time and also use posix locks. In this case, the only good solution I >> see is to retake the locks on brick reconnection. >> > > Agree. But lock-healing should be done only by HA layers like AFR/EC as > only they know whether there are enough online bricks to have prevented any > conflicting lock. Protocol/client itself doesn't have enough information to > do that. If its a plain distribute, I don't see a way to heal locks without > loosing the property of exclusivity of locks. > > What I proposed is a short term solution. mid to long term solution should > be lock healing feature implemented in AFR/EC. In fact I had this > conversation with +Karampuri, Pranith before > posting this msg to ML. > > >> >>> However, this use case is not affected if the application don't acquire >>> any POSIX locks. So, I am interested in knowing >>> * whether your use cases use POSIX locks? >>> * Is it feasible for your application to re-open fds and re-acquire >>> locks on seeing EBADFD errors? >>> >> >> I think that many applications are not prepared to handle that. >> > > I too suspected that and in fact not too happy with the solution. But went > ahead with this mail as I heard implementing lock-heal in AFR will take > time and hence there are no alternative short term solutions. > Also failing loudly is preferred to silently dropping locks. > > >> Xavi >> >> >>> >>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>> >>> regards, >>> Raghavendra >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jahernan at redhat.com Wed Mar 27 11:43:11 2019 From: jahernan at redhat.com (Xavi Hernandez) Date: Wed, 27 Mar 2019 12:43:11 +0100 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa wrote: > > > On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez > wrote: > >> Hi Raghavendra, >> >> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa >> wrote: >> >>> All, >>> >>> Glusterfs cleans up POSIX locks held on an fd when the client/mount >>> through which those locks are held disconnects from bricks/server. This >>> helps Glusterfs to not run into a stale lock problem later (For eg., if >>> application unlocks while the connection was still down). However, this >>> means the lock is no longer exclusive as other applications/clients can >>> acquire the same lock. To communicate that locks are no longer valid, we >>> are planning to mark the fd (which has POSIX locks) bad on a disconnect so >>> that any future operations on that fd will fail, forcing the application to >>> re-open the fd and re-acquire locks it needs [1]. >>> >> >> Wouldn't it be better to retake the locks when the brick is reconnected >> if the lock is still in use ? >> > > There is also a possibility that clients may never reconnect. That's the > primary reason why bricks assume the worst (client will not reconnect) and > cleanup the locks. > True, so it's fine to cleanup the locks. I'm not saying that locks shouldn't be released on disconnect. The assumption is that if the client has really died, it will also disconnect from other bricks, who will release the locks. So, eventually, another client will have enough quorum to attempt a lock that will succeed. In other words, if a client gets disconnected from too many bricks simultaneously (loses Quorum), then that client can be considered as bad and can return errors to the application. This should also cause to release the locks on the remaining connected bricks. On the other hand, if the disconnection is very short and the client has not died, it will keep enough locked files (it has quorum) to avoid other clients to successfully acquire a lock. In this case, if the brick is reconnected, all existing locks should be reacquired to recover the original state before the disconnection. > >> BTW, the referenced bug is not public. Should we open another bug to >> track this ? >> > > I've just opened up the comment to give enough context. I'll open a bug > upstream too. > > >> >> >>> >>> Note that with AFR/replicate in picture we can prevent errors to >>> application as long as Quorum number of children "never ever" lost >>> connection with bricks after locks have been acquired. I am using the term >>> "never ever" as locks are not healed back after re-connection and hence >>> first disconnect would've marked the fd bad and the fd remains so even >>> after re-connection happens. So, its not just Quorum number of children >>> "currently online", but Quorum number of children "never having >>> disconnected with bricks after locks are acquired". >>> >> >> I think this requisite is not feasible. In a distributed file system, >> sooner or later all bricks will be disconnected. It could be because of >> failures or because an upgrade is done, but it will happen. >> >> The difference here is how long are fd's kept open. If applications open >> and close files frequently enough (i.e. the fd is not kept open more time >> than it takes to have more than Quorum bricks disconnected) then there's no >> problem. The problem can only appear on applications that open files for a >> long time and also use posix locks. In this case, the only good solution I >> see is to retake the locks on brick reconnection. >> > > Agree. But lock-healing should be done only by HA layers like AFR/EC as > only they know whether there are enough online bricks to have prevented any > conflicting lock. Protocol/client itself doesn't have enough information to > do that. If its a plain distribute, I don't see a way to heal locks without > loosing the property of exclusivity of locks. > Lock-healing of locks acquired while a brick was disconnected need to be handled by AFR/EC. However, locks already present at the moment of disconnection could be recovered by client xlator itself as long as the file has not been closed (which client xlator already knows). Xavi > What I proposed is a short term solution. mid to long term solution should > be lock healing feature implemented in AFR/EC. In fact I had this > conversation with +Karampuri, Pranith before > posting this msg to ML. > > >> >>> However, this use case is not affected if the application don't acquire >>> any POSIX locks. So, I am interested in knowing >>> * whether your use cases use POSIX locks? >>> * Is it feasible for your application to re-open fds and re-acquire >>> locks on seeing EBADFD errors? >>> >> >> I think that many applications are not prepared to handle that. >> > > I too suspected that and in fact not too happy with the solution. But went > ahead with this mail as I heard implementing lock-heal in AFR will take > time and hence there are no alternative short term solutions. > > >> Xavi >> >> >>> >>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>> >>> regards, >>> Raghavendra >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jahernan at redhat.com Wed Mar 27 11:51:41 2019 From: jahernan at redhat.com (Xavi Hernandez) Date: Wed, 27 Mar 2019 12:51:41 +0100 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Wed, Mar 27, 2019 at 11:54 AM Raghavendra Gowdappa wrote: > > > On Wed, Mar 27, 2019 at 4:22 PM Raghavendra Gowdappa > wrote: > >> >> >> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez >> wrote: >> >>> Hi Raghavendra, >>> >>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa < >>> rgowdapp at redhat.com> wrote: >>> >>>> All, >>>> >>>> Glusterfs cleans up POSIX locks held on an fd when the client/mount >>>> through which those locks are held disconnects from bricks/server. This >>>> helps Glusterfs to not run into a stale lock problem later (For eg., if >>>> application unlocks while the connection was still down). However, this >>>> means the lock is no longer exclusive as other applications/clients can >>>> acquire the same lock. To communicate that locks are no longer valid, we >>>> are planning to mark the fd (which has POSIX locks) bad on a disconnect so >>>> that any future operations on that fd will fail, forcing the application to >>>> re-open the fd and re-acquire locks it needs [1]. >>>> >>> >>> Wouldn't it be better to retake the locks when the brick is reconnected >>> if the lock is still in use ? >>> >> >> There is also a possibility that clients may never reconnect. That's the >> primary reason why bricks assume the worst (client will not reconnect) and >> cleanup the locks. >> >> >>> BTW, the referenced bug is not public. Should we open another bug to >>> track this ? >>> >> >> I've just opened up the comment to give enough context. I'll open a bug >> upstream too. >> >> >>> >>> >>>> >>>> Note that with AFR/replicate in picture we can prevent errors to >>>> application as long as Quorum number of children "never ever" lost >>>> connection with bricks after locks have been acquired. I am using the term >>>> "never ever" as locks are not healed back after re-connection and hence >>>> first disconnect would've marked the fd bad and the fd remains so even >>>> after re-connection happens. So, its not just Quorum number of children >>>> "currently online", but Quorum number of children "never having >>>> disconnected with bricks after locks are acquired". >>>> >>> >>> I think this requisite is not feasible. In a distributed file system, >>> sooner or later all bricks will be disconnected. It could be because of >>> failures or because an upgrade is done, but it will happen. >>> >>> The difference here is how long are fd's kept open. If applications open >>> and close files frequently enough (i.e. the fd is not kept open more time >>> than it takes to have more than Quorum bricks disconnected) then there's no >>> problem. The problem can only appear on applications that open files for a >>> long time and also use posix locks. In this case, the only good solution I >>> see is to retake the locks on brick reconnection. >>> >> >> Agree. But lock-healing should be done only by HA layers like AFR/EC as >> only they know whether there are enough online bricks to have prevented any >> conflicting lock. Protocol/client itself doesn't have enough information to >> do that. If its a plain distribute, I don't see a way to heal locks without >> loosing the property of exclusivity of locks. >> >> What I proposed is a short term solution. mid to long term solution >> should be lock healing feature implemented in AFR/EC. In fact I had this >> conversation with +Karampuri, Pranith before >> posting this msg to ML. >> >> >>> >>>> However, this use case is not affected if the application don't acquire >>>> any POSIX locks. So, I am interested in knowing >>>> * whether your use cases use POSIX locks? >>>> * Is it feasible for your application to re-open fds and re-acquire >>>> locks on seeing EBADFD errors? >>>> >>> >>> I think that many applications are not prepared to handle that. >>> >> >> I too suspected that and in fact not too happy with the solution. But >> went ahead with this mail as I heard implementing lock-heal in AFR will >> take time and hence there are no alternative short term solutions. >> > > Also failing loudly is preferred to silently dropping locks. > Yes. Silently dropping locks can cause corruption, which is worse. However causing application failures doesn't improve user experience either. Unfortunately I'm not aware of any other short term solution right now. > >> >> >>> Xavi >>> >>> >>>> >>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>>> >>>> regards, >>>> Raghavendra >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From pkarampu at redhat.com Wed Mar 27 12:13:06 2019 From: pkarampu at redhat.com (Pranith Kumar Karampuri) Date: Wed, 27 Mar 2019 17:43:06 +0530 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez wrote: > On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa > wrote: > >> >> >> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez >> wrote: >> >>> Hi Raghavendra, >>> >>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa < >>> rgowdapp at redhat.com> wrote: >>> >>>> All, >>>> >>>> Glusterfs cleans up POSIX locks held on an fd when the client/mount >>>> through which those locks are held disconnects from bricks/server. This >>>> helps Glusterfs to not run into a stale lock problem later (For eg., if >>>> application unlocks while the connection was still down). However, this >>>> means the lock is no longer exclusive as other applications/clients can >>>> acquire the same lock. To communicate that locks are no longer valid, we >>>> are planning to mark the fd (which has POSIX locks) bad on a disconnect so >>>> that any future operations on that fd will fail, forcing the application to >>>> re-open the fd and re-acquire locks it needs [1]. >>>> >>> >>> Wouldn't it be better to retake the locks when the brick is reconnected >>> if the lock is still in use ? >>> >> >> There is also a possibility that clients may never reconnect. That's the >> primary reason why bricks assume the worst (client will not reconnect) and >> cleanup the locks. >> > > True, so it's fine to cleanup the locks. I'm not saying that locks > shouldn't be released on disconnect. The assumption is that if the client > has really died, it will also disconnect from other bricks, who will > release the locks. So, eventually, another client will have enough quorum > to attempt a lock that will succeed. In other words, if a client gets > disconnected from too many bricks simultaneously (loses Quorum), then that > client can be considered as bad and can return errors to the application. > This should also cause to release the locks on the remaining connected > bricks. > > On the other hand, if the disconnection is very short and the client has > not died, it will keep enough locked files (it has quorum) to avoid other > clients to successfully acquire a lock. In this case, if the brick is > reconnected, all existing locks should be reacquired to recover the > original state before the disconnection. > > >> >>> BTW, the referenced bug is not public. Should we open another bug to >>> track this ? >>> >> >> I've just opened up the comment to give enough context. I'll open a bug >> upstream too. >> >> >>> >>> >>>> >>>> Note that with AFR/replicate in picture we can prevent errors to >>>> application as long as Quorum number of children "never ever" lost >>>> connection with bricks after locks have been acquired. I am using the term >>>> "never ever" as locks are not healed back after re-connection and hence >>>> first disconnect would've marked the fd bad and the fd remains so even >>>> after re-connection happens. So, its not just Quorum number of children >>>> "currently online", but Quorum number of children "never having >>>> disconnected with bricks after locks are acquired". >>>> >>> >>> I think this requisite is not feasible. In a distributed file system, >>> sooner or later all bricks will be disconnected. It could be because of >>> failures or because an upgrade is done, but it will happen. >>> >>> The difference here is how long are fd's kept open. If applications open >>> and close files frequently enough (i.e. the fd is not kept open more time >>> than it takes to have more than Quorum bricks disconnected) then there's no >>> problem. The problem can only appear on applications that open files for a >>> long time and also use posix locks. In this case, the only good solution I >>> see is to retake the locks on brick reconnection. >>> >> >> Agree. But lock-healing should be done only by HA layers like AFR/EC as >> only they know whether there are enough online bricks to have prevented any >> conflicting lock. Protocol/client itself doesn't have enough information to >> do that. If its a plain distribute, I don't see a way to heal locks without >> loosing the property of exclusivity of locks. >> > > Lock-healing of locks acquired while a brick was disconnected need to be > handled by AFR/EC. However, locks already present at the moment of > disconnection could be recovered by client xlator itself as long as the > file has not been closed (which client xlator already knows). > What if another client (say mount-2) took locks at the time of disconnect from mount-1 and modified the file and unlocked? client xlator doing the heal may not be a good idea. > > Xavi > > >> What I proposed is a short term solution. mid to long term solution >> should be lock healing feature implemented in AFR/EC. In fact I had this >> conversation with +Karampuri, Pranith before >> posting this msg to ML. >> >> >>> >>>> However, this use case is not affected if the application don't acquire >>>> any POSIX locks. So, I am interested in knowing >>>> * whether your use cases use POSIX locks? >>>> * Is it feasible for your application to re-open fds and re-acquire >>>> locks on seeing EBADFD errors? >>>> >>> >>> I think that many applications are not prepared to handle that. >>> >> >> I too suspected that and in fact not too happy with the solution. But >> went ahead with this mail as I heard implementing lock-heal in AFR will >> take time and hence there are no alternative short term solutions. >> > >> >>> Xavi >>> >>> >>>> >>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>>> >>>> regards, >>>> Raghavendra >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >>> -- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: From jahernan at redhat.com Wed Mar 27 13:08:08 2019 From: jahernan at redhat.com (Xavi Hernandez) Date: Wed, 27 Mar 2019 14:08:08 +0100 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Wed, Mar 27, 2019 at 1:13 PM Pranith Kumar Karampuri wrote: > > > On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez > wrote: > >> On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa < >> rgowdapp at redhat.com> wrote: >> >>> >>> >>> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez >>> wrote: >>> >>>> Hi Raghavendra, >>>> >>>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa < >>>> rgowdapp at redhat.com> wrote: >>>> >>>>> All, >>>>> >>>>> Glusterfs cleans up POSIX locks held on an fd when the client/mount >>>>> through which those locks are held disconnects from bricks/server. This >>>>> helps Glusterfs to not run into a stale lock problem later (For eg., if >>>>> application unlocks while the connection was still down). However, this >>>>> means the lock is no longer exclusive as other applications/clients can >>>>> acquire the same lock. To communicate that locks are no longer valid, we >>>>> are planning to mark the fd (which has POSIX locks) bad on a disconnect so >>>>> that any future operations on that fd will fail, forcing the application to >>>>> re-open the fd and re-acquire locks it needs [1]. >>>>> >>>> >>>> Wouldn't it be better to retake the locks when the brick is reconnected >>>> if the lock is still in use ? >>>> >>> >>> There is also a possibility that clients may never reconnect. That's >>> the primary reason why bricks assume the worst (client will not reconnect) >>> and cleanup the locks. >>> >> >> True, so it's fine to cleanup the locks. I'm not saying that locks >> shouldn't be released on disconnect. The assumption is that if the client >> has really died, it will also disconnect from other bricks, who will >> release the locks. So, eventually, another client will have enough quorum >> to attempt a lock that will succeed. In other words, if a client gets >> disconnected from too many bricks simultaneously (loses Quorum), then that >> client can be considered as bad and can return errors to the application. >> This should also cause to release the locks on the remaining connected >> bricks. >> >> On the other hand, if the disconnection is very short and the client has >> not died, it will keep enough locked files (it has quorum) to avoid other >> clients to successfully acquire a lock. In this case, if the brick is >> reconnected, all existing locks should be reacquired to recover the >> original state before the disconnection. >> >> >>> >>>> BTW, the referenced bug is not public. Should we open another bug to >>>> track this ? >>>> >>> >>> I've just opened up the comment to give enough context. I'll open a bug >>> upstream too. >>> >>> >>>> >>>> >>>>> >>>>> Note that with AFR/replicate in picture we can prevent errors to >>>>> application as long as Quorum number of children "never ever" lost >>>>> connection with bricks after locks have been acquired. I am using the term >>>>> "never ever" as locks are not healed back after re-connection and hence >>>>> first disconnect would've marked the fd bad and the fd remains so even >>>>> after re-connection happens. So, its not just Quorum number of children >>>>> "currently online", but Quorum number of children "never having >>>>> disconnected with bricks after locks are acquired". >>>>> >>>> >>>> I think this requisite is not feasible. In a distributed file system, >>>> sooner or later all bricks will be disconnected. It could be because of >>>> failures or because an upgrade is done, but it will happen. >>>> >>>> The difference here is how long are fd's kept open. If applications >>>> open and close files frequently enough (i.e. the fd is not kept open more >>>> time than it takes to have more than Quorum bricks disconnected) then >>>> there's no problem. The problem can only appear on applications that open >>>> files for a long time and also use posix locks. In this case, the only good >>>> solution I see is to retake the locks on brick reconnection. >>>> >>> >>> Agree. But lock-healing should be done only by HA layers like AFR/EC as >>> only they know whether there are enough online bricks to have prevented any >>> conflicting lock. Protocol/client itself doesn't have enough information to >>> do that. If its a plain distribute, I don't see a way to heal locks without >>> loosing the property of exclusivity of locks. >>> >> >> Lock-healing of locks acquired while a brick was disconnected need to be >> handled by AFR/EC. However, locks already present at the moment of >> disconnection could be recovered by client xlator itself as long as the >> file has not been closed (which client xlator already knows). >> > > What if another client (say mount-2) took locks at the time of disconnect > from mount-1 and modified the file and unlocked? client xlator doing the > heal may not be a good idea. > To avoid that we should ensure that any lock/unlocks are sent to the client, even if we know it's disconnected, so that client xlator can track them. The alternative is to duplicate and maintain code both on AFR and EC (and not sure if even in DHT depending on how we want to handle some cases). A similar thing could be done for open fd, since the current solution duplicates code in AFR and EC, but this is another topic... > >> >> Xavi >> >> >>> What I proposed is a short term solution. mid to long term solution >>> should be lock healing feature implemented in AFR/EC. In fact I had this >>> conversation with +Karampuri, Pranith before >>> posting this msg to ML. >>> >>> >>>> >>>>> However, this use case is not affected if the application don't >>>>> acquire any POSIX locks. So, I am interested in knowing >>>>> * whether your use cases use POSIX locks? >>>>> * Is it feasible for your application to re-open fds and re-acquire >>>>> locks on seeing EBADFD errors? >>>>> >>>> >>>> I think that many applications are not prepared to handle that. >>>> >>> >>> I too suspected that and in fact not too happy with the solution. But >>> went ahead with this mail as I heard implementing lock-heal in AFR will >>> take time and hence there are no alternative short term solutions. >>> >> >>> >>>> Xavi >>>> >>>> >>>>> >>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>>>> >>>>> regards, >>>>> Raghavendra >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>>> > > -- > Pranith > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pkarampu at redhat.com Wed Mar 27 13:19:52 2019 From: pkarampu at redhat.com (Pranith Kumar Karampuri) Date: Wed, 27 Mar 2019 18:49:52 +0530 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Wed, Mar 27, 2019 at 6:38 PM Xavi Hernandez wrote: > On Wed, Mar 27, 2019 at 1:13 PM Pranith Kumar Karampuri < > pkarampu at redhat.com> wrote: > >> >> >> On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez >> wrote: >> >>> On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa < >>> rgowdapp at redhat.com> wrote: >>> >>>> >>>> >>>> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez >>>> wrote: >>>> >>>>> Hi Raghavendra, >>>>> >>>>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa < >>>>> rgowdapp at redhat.com> wrote: >>>>> >>>>>> All, >>>>>> >>>>>> Glusterfs cleans up POSIX locks held on an fd when the client/mount >>>>>> through which those locks are held disconnects from bricks/server. This >>>>>> helps Glusterfs to not run into a stale lock problem later (For eg., if >>>>>> application unlocks while the connection was still down). However, this >>>>>> means the lock is no longer exclusive as other applications/clients can >>>>>> acquire the same lock. To communicate that locks are no longer valid, we >>>>>> are planning to mark the fd (which has POSIX locks) bad on a disconnect so >>>>>> that any future operations on that fd will fail, forcing the application to >>>>>> re-open the fd and re-acquire locks it needs [1]. >>>>>> >>>>> >>>>> Wouldn't it be better to retake the locks when the brick is >>>>> reconnected if the lock is still in use ? >>>>> >>>> >>>> There is also a possibility that clients may never reconnect. That's >>>> the primary reason why bricks assume the worst (client will not reconnect) >>>> and cleanup the locks. >>>> >>> >>> True, so it's fine to cleanup the locks. I'm not saying that locks >>> shouldn't be released on disconnect. The assumption is that if the client >>> has really died, it will also disconnect from other bricks, who will >>> release the locks. So, eventually, another client will have enough quorum >>> to attempt a lock that will succeed. In other words, if a client gets >>> disconnected from too many bricks simultaneously (loses Quorum), then that >>> client can be considered as bad and can return errors to the application. >>> This should also cause to release the locks on the remaining connected >>> bricks. >>> >>> On the other hand, if the disconnection is very short and the client has >>> not died, it will keep enough locked files (it has quorum) to avoid other >>> clients to successfully acquire a lock. In this case, if the brick is >>> reconnected, all existing locks should be reacquired to recover the >>> original state before the disconnection. >>> >>> >>>> >>>>> BTW, the referenced bug is not public. Should we open another bug to >>>>> track this ? >>>>> >>>> >>>> I've just opened up the comment to give enough context. I'll open a bug >>>> upstream too. >>>> >>>> >>>>> >>>>> >>>>>> >>>>>> Note that with AFR/replicate in picture we can prevent errors to >>>>>> application as long as Quorum number of children "never ever" lost >>>>>> connection with bricks after locks have been acquired. I am using the term >>>>>> "never ever" as locks are not healed back after re-connection and hence >>>>>> first disconnect would've marked the fd bad and the fd remains so even >>>>>> after re-connection happens. So, its not just Quorum number of children >>>>>> "currently online", but Quorum number of children "never having >>>>>> disconnected with bricks after locks are acquired". >>>>>> >>>>> >>>>> I think this requisite is not feasible. In a distributed file system, >>>>> sooner or later all bricks will be disconnected. It could be because of >>>>> failures or because an upgrade is done, but it will happen. >>>>> >>>>> The difference here is how long are fd's kept open. If applications >>>>> open and close files frequently enough (i.e. the fd is not kept open more >>>>> time than it takes to have more than Quorum bricks disconnected) then >>>>> there's no problem. The problem can only appear on applications that open >>>>> files for a long time and also use posix locks. In this case, the only good >>>>> solution I see is to retake the locks on brick reconnection. >>>>> >>>> >>>> Agree. But lock-healing should be done only by HA layers like AFR/EC as >>>> only they know whether there are enough online bricks to have prevented any >>>> conflicting lock. Protocol/client itself doesn't have enough information to >>>> do that. If its a plain distribute, I don't see a way to heal locks without >>>> loosing the property of exclusivity of locks. >>>> >>> >>> Lock-healing of locks acquired while a brick was disconnected need to be >>> handled by AFR/EC. However, locks already present at the moment of >>> disconnection could be recovered by client xlator itself as long as the >>> file has not been closed (which client xlator already knows). >>> >> >> What if another client (say mount-2) took locks at the time of disconnect >> from mount-1 and modified the file and unlocked? client xlator doing the >> heal may not be a good idea. >> > > To avoid that we should ensure that any lock/unlocks are sent to the > client, even if we know it's disconnected, so that client xlator can track > them. The alternative is to duplicate and maintain code both on AFR and EC > (and not sure if even in DHT depending on how we want to handle some > cases). > Didn't understand the solution. I wanted to highlight that client xlator by itself can't make a decision about healing locks because it doesn't know what happened on other replicas. If we have replica-3 volume and all 3 bricks get disconnected to their respective bricks. Now another mount process can take a lock on that file modify it and unlock. Now upon reconnection, the old mount process which had locks would think it always had the lock if client xlator independently tries to heal its own locks because file is not closed on it so far. But that is wrong. Let me know if it makes sense.... > A similar thing could be done for open fd, since the current solution > duplicates code in AFR and EC, but this is another topic... > > >> >>> >>> Xavi >>> >>> >>>> What I proposed is a short term solution. mid to long term solution >>>> should be lock healing feature implemented in AFR/EC. In fact I had this >>>> conversation with +Karampuri, Pranith before >>>> posting this msg to ML. >>>> >>>> >>>>> >>>>>> However, this use case is not affected if the application don't >>>>>> acquire any POSIX locks. So, I am interested in knowing >>>>>> * whether your use cases use POSIX locks? >>>>>> * Is it feasible for your application to re-open fds and re-acquire >>>>>> locks on seeing EBADFD errors? >>>>>> >>>>> >>>>> I think that many applications are not prepared to handle that. >>>>> >>>> >>>> I too suspected that and in fact not too happy with the solution. But >>>> went ahead with this mail as I heard implementing lock-heal in AFR will >>>> take time and hence there are no alternative short term solutions. >>>> >>> >>>> >>>>> Xavi >>>>> >>>>> >>>>>> >>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>>>>> >>>>>> regards, >>>>>> Raghavendra >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>>> >> >> -- >> Pranith >> > -- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: From pgurusid at redhat.com Wed Mar 27 14:56:13 2019 From: pgurusid at redhat.com (Poornima Gurusiddaiah) Date: Wed, 27 Mar 2019 20:26:13 +0530 Subject: [Gluster-devel] [Gluster-users] Prioritise local bricks for IO? In-Reply-To: References: <29221907.583.1553599314586.JavaMail.zimbra@li.nux.ro> Message-ID: This feature is not under active development as it was not used widely. AFAIK its not supported feature. +Nithya +Raghavendra for further clarifications. Regards, Poornima On Wed, Mar 27, 2019 at 12:33 PM Lucian wrote: > Oh, that's just what the doctor ordered! > Hope it works, thanks > > On 27 March 2019 03:15:57 GMT, Vlad Kopylov wrote: >> >> I don't remember if it still in works >> NUFA >> >> https://github.com/gluster/glusterfs-specs/blob/master/done/Features/nufa.md >> >> v >> >> On Tue, Mar 26, 2019 at 7:27 AM Nux! wrote: >> >>> Hello, >>> >>> I'm trying to set up a distributed backup storage (no replicas), but I'd >>> like to prioritise the local bricks for any IO done on the volume. >>> This will be a backup stor, so in other words, I'd like the files to be >>> written locally if there is space, so as to save the NICs for other traffic. >>> >>> Anyone knows how this might be achievable, if at all? >>> >>> -- >>> Sent from the Delta quadrant using Borg technology! >>> >>> Nux! >>> www.nux.ro >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >>> >> > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From jahernan at redhat.com Wed Mar 27 15:08:34 2019 From: jahernan at redhat.com (Xavi Hernandez) Date: Wed, 27 Mar 2019 16:08:34 +0100 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Wed, Mar 27, 2019 at 2:20 PM Pranith Kumar Karampuri wrote: > > > On Wed, Mar 27, 2019 at 6:38 PM Xavi Hernandez > wrote: > >> On Wed, Mar 27, 2019 at 1:13 PM Pranith Kumar Karampuri < >> pkarampu at redhat.com> wrote: >> >>> >>> >>> On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez >>> wrote: >>> >>>> On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa < >>>> rgowdapp at redhat.com> wrote: >>>> >>>>> >>>>> >>>>> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez >>>>> wrote: >>>>> >>>>>> Hi Raghavendra, >>>>>> >>>>>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa < >>>>>> rgowdapp at redhat.com> wrote: >>>>>> >>>>>>> All, >>>>>>> >>>>>>> Glusterfs cleans up POSIX locks held on an fd when the client/mount >>>>>>> through which those locks are held disconnects from bricks/server. This >>>>>>> helps Glusterfs to not run into a stale lock problem later (For eg., if >>>>>>> application unlocks while the connection was still down). However, this >>>>>>> means the lock is no longer exclusive as other applications/clients can >>>>>>> acquire the same lock. To communicate that locks are no longer valid, we >>>>>>> are planning to mark the fd (which has POSIX locks) bad on a disconnect so >>>>>>> that any future operations on that fd will fail, forcing the application to >>>>>>> re-open the fd and re-acquire locks it needs [1]. >>>>>>> >>>>>> >>>>>> Wouldn't it be better to retake the locks when the brick is >>>>>> reconnected if the lock is still in use ? >>>>>> >>>>> >>>>> There is also a possibility that clients may never reconnect. That's >>>>> the primary reason why bricks assume the worst (client will not reconnect) >>>>> and cleanup the locks. >>>>> >>>> >>>> True, so it's fine to cleanup the locks. I'm not saying that locks >>>> shouldn't be released on disconnect. The assumption is that if the client >>>> has really died, it will also disconnect from other bricks, who will >>>> release the locks. So, eventually, another client will have enough quorum >>>> to attempt a lock that will succeed. In other words, if a client gets >>>> disconnected from too many bricks simultaneously (loses Quorum), then that >>>> client can be considered as bad and can return errors to the application. >>>> This should also cause to release the locks on the remaining connected >>>> bricks. >>>> >>>> On the other hand, if the disconnection is very short and the client >>>> has not died, it will keep enough locked files (it has quorum) to avoid >>>> other clients to successfully acquire a lock. In this case, if the brick is >>>> reconnected, all existing locks should be reacquired to recover the >>>> original state before the disconnection. >>>> >>>> >>>>> >>>>>> BTW, the referenced bug is not public. Should we open another bug to >>>>>> track this ? >>>>>> >>>>> >>>>> I've just opened up the comment to give enough context. I'll open a >>>>> bug upstream too. >>>>> >>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Note that with AFR/replicate in picture we can prevent errors to >>>>>>> application as long as Quorum number of children "never ever" lost >>>>>>> connection with bricks after locks have been acquired. I am using the term >>>>>>> "never ever" as locks are not healed back after re-connection and hence >>>>>>> first disconnect would've marked the fd bad and the fd remains so even >>>>>>> after re-connection happens. So, its not just Quorum number of children >>>>>>> "currently online", but Quorum number of children "never having >>>>>>> disconnected with bricks after locks are acquired". >>>>>>> >>>>>> >>>>>> I think this requisite is not feasible. In a distributed file system, >>>>>> sooner or later all bricks will be disconnected. It could be because of >>>>>> failures or because an upgrade is done, but it will happen. >>>>>> >>>>>> The difference here is how long are fd's kept open. If applications >>>>>> open and close files frequently enough (i.e. the fd is not kept open more >>>>>> time than it takes to have more than Quorum bricks disconnected) then >>>>>> there's no problem. The problem can only appear on applications that open >>>>>> files for a long time and also use posix locks. In this case, the only good >>>>>> solution I see is to retake the locks on brick reconnection. >>>>>> >>>>> >>>>> Agree. But lock-healing should be done only by HA layers like AFR/EC >>>>> as only they know whether there are enough online bricks to have prevented >>>>> any conflicting lock. Protocol/client itself doesn't have enough >>>>> information to do that. If its a plain distribute, I don't see a way to >>>>> heal locks without loosing the property of exclusivity of locks. >>>>> >>>> >>>> Lock-healing of locks acquired while a brick was disconnected need to >>>> be handled by AFR/EC. However, locks already present at the moment of >>>> disconnection could be recovered by client xlator itself as long as the >>>> file has not been closed (which client xlator already knows). >>>> >>> >>> What if another client (say mount-2) took locks at the time of >>> disconnect from mount-1 and modified the file and unlocked? client xlator >>> doing the heal may not be a good idea. >>> >> >> To avoid that we should ensure that any lock/unlocks are sent to the >> client, even if we know it's disconnected, so that client xlator can track >> them. The alternative is to duplicate and maintain code both on AFR and EC >> (and not sure if even in DHT depending on how we want to handle some >> cases). >> > > Didn't understand the solution. I wanted to highlight that client xlator > by itself can't make a decision about healing locks because it doesn't know > what happened on other replicas. If we have replica-3 volume and all 3 > bricks get disconnected to their respective bricks. Now another mount > process can take a lock on that file modify it and unlock. Now upon > reconnection, the old mount process which had locks would think it always > had the lock if client xlator independently tries to heal its own locks > because file is not closed on it so far. But that is wrong. Let me know if > it makes sense.... > My point of view is that any configuration with these requirements will have an appropriate quorum value so that it's impossible to have two or more partitions of the nodes working at the same time. So, under this assumptions, mount-1 can be in two situations: 1. It has lost a single brick and it's still operational. The other bricks will continue locked and everything should work fine from the point of view of the application. Any other application trying to get a lock will fail due to lack of quorum. When the lost brick comes back and is reconnected, client xlator will still have the fd reference and locks taken (unless the application has released the lock or closed the fd, in which case client xlator should get notified and clear that information), so it should be able to recover the previous state. 2. It has lost 2 or 3 bricks. In this case mount-1 has lost quorum and any operation going to that file should fail with EIO. AFR should send a special request to client xlator so that it forgets any fd's and locks for that file. If bricks reconnect after that, no fd reopen or lock recovery will happen. Eventually the application should close the fd and retry later. This may succeed to not, depending on whether mount-2 has taken the lock already or not. So, it's true that client xlator doesn't know the state of the other bricks, but it doesn't need to as long as AFR/EC strictly enforces quorum and updates client xlator when quorum is lost. I haven't worked out all the details of this approach, but I think it should work and it's simpler to maintain than trying to do the same for AFR and EC. Xavi > >> A similar thing could be done for open fd, since the current solution >> duplicates code in AFR and EC, but this is another topic... >> >> >>> >>>> >>>> Xavi >>>> >>>> >>>>> What I proposed is a short term solution. mid to long term solution >>>>> should be lock healing feature implemented in AFR/EC. In fact I had this >>>>> conversation with +Karampuri, Pranith before >>>>> posting this msg to ML. >>>>> >>>>> >>>>>> >>>>>>> However, this use case is not affected if the application don't >>>>>>> acquire any POSIX locks. So, I am interested in knowing >>>>>>> * whether your use cases use POSIX locks? >>>>>>> * Is it feasible for your application to re-open fds and re-acquire >>>>>>> locks on seeing EBADFD errors? >>>>>>> >>>>>> >>>>>> I think that many applications are not prepared to handle that. >>>>>> >>>>> >>>>> I too suspected that and in fact not too happy with the solution. But >>>>> went ahead with this mail as I heard implementing lock-heal in AFR will >>>>> take time and hence there are no alternative short term solutions. >>>>> >>>> >>>>> >>>>>> Xavi >>>>>> >>>>>> >>>>>>> >>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>>>>>> >>>>>>> regards, >>>>>>> Raghavendra >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> Gluster-users at gluster.org >>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>>> >>> >>> -- >>> Pranith >>> >> > > -- > Pranith > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pkarampu at redhat.com Wed Mar 27 17:26:41 2019 From: pkarampu at redhat.com (Pranith Kumar Karampuri) Date: Wed, 27 Mar 2019 22:56:41 +0530 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Wed, Mar 27, 2019 at 8:38 PM Xavi Hernandez wrote: > On Wed, Mar 27, 2019 at 2:20 PM Pranith Kumar Karampuri < > pkarampu at redhat.com> wrote: > >> >> >> On Wed, Mar 27, 2019 at 6:38 PM Xavi Hernandez >> wrote: >> >>> On Wed, Mar 27, 2019 at 1:13 PM Pranith Kumar Karampuri < >>> pkarampu at redhat.com> wrote: >>> >>>> >>>> >>>> On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez >>>> wrote: >>>> >>>>> On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa < >>>>> rgowdapp at redhat.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez >>>>>> wrote: >>>>>> >>>>>>> Hi Raghavendra, >>>>>>> >>>>>>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa < >>>>>>> rgowdapp at redhat.com> wrote: >>>>>>> >>>>>>>> All, >>>>>>>> >>>>>>>> Glusterfs cleans up POSIX locks held on an fd when the client/mount >>>>>>>> through which those locks are held disconnects from bricks/server. This >>>>>>>> helps Glusterfs to not run into a stale lock problem later (For eg., if >>>>>>>> application unlocks while the connection was still down). However, this >>>>>>>> means the lock is no longer exclusive as other applications/clients can >>>>>>>> acquire the same lock. To communicate that locks are no longer valid, we >>>>>>>> are planning to mark the fd (which has POSIX locks) bad on a disconnect so >>>>>>>> that any future operations on that fd will fail, forcing the application to >>>>>>>> re-open the fd and re-acquire locks it needs [1]. >>>>>>>> >>>>>>> >>>>>>> Wouldn't it be better to retake the locks when the brick is >>>>>>> reconnected if the lock is still in use ? >>>>>>> >>>>>> >>>>>> There is also a possibility that clients may never reconnect. That's >>>>>> the primary reason why bricks assume the worst (client will not reconnect) >>>>>> and cleanup the locks. >>>>>> >>>>> >>>>> True, so it's fine to cleanup the locks. I'm not saying that locks >>>>> shouldn't be released on disconnect. The assumption is that if the client >>>>> has really died, it will also disconnect from other bricks, who will >>>>> release the locks. So, eventually, another client will have enough quorum >>>>> to attempt a lock that will succeed. In other words, if a client gets >>>>> disconnected from too many bricks simultaneously (loses Quorum), then that >>>>> client can be considered as bad and can return errors to the application. >>>>> This should also cause to release the locks on the remaining connected >>>>> bricks. >>>>> >>>>> On the other hand, if the disconnection is very short and the client >>>>> has not died, it will keep enough locked files (it has quorum) to avoid >>>>> other clients to successfully acquire a lock. In this case, if the brick is >>>>> reconnected, all existing locks should be reacquired to recover the >>>>> original state before the disconnection. >>>>> >>>>> >>>>>> >>>>>>> BTW, the referenced bug is not public. Should we open another bug to >>>>>>> track this ? >>>>>>> >>>>>> >>>>>> I've just opened up the comment to give enough context. I'll open a >>>>>> bug upstream too. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Note that with AFR/replicate in picture we can prevent errors to >>>>>>>> application as long as Quorum number of children "never ever" lost >>>>>>>> connection with bricks after locks have been acquired. I am using the term >>>>>>>> "never ever" as locks are not healed back after re-connection and hence >>>>>>>> first disconnect would've marked the fd bad and the fd remains so even >>>>>>>> after re-connection happens. So, its not just Quorum number of children >>>>>>>> "currently online", but Quorum number of children "never having >>>>>>>> disconnected with bricks after locks are acquired". >>>>>>>> >>>>>>> >>>>>>> I think this requisite is not feasible. In a distributed file >>>>>>> system, sooner or later all bricks will be disconnected. It could be >>>>>>> because of failures or because an upgrade is done, but it will happen. >>>>>>> >>>>>>> The difference here is how long are fd's kept open. If applications >>>>>>> open and close files frequently enough (i.e. the fd is not kept open more >>>>>>> time than it takes to have more than Quorum bricks disconnected) then >>>>>>> there's no problem. The problem can only appear on applications that open >>>>>>> files for a long time and also use posix locks. In this case, the only good >>>>>>> solution I see is to retake the locks on brick reconnection. >>>>>>> >>>>>> >>>>>> Agree. But lock-healing should be done only by HA layers like AFR/EC >>>>>> as only they know whether there are enough online bricks to have prevented >>>>>> any conflicting lock. Protocol/client itself doesn't have enough >>>>>> information to do that. If its a plain distribute, I don't see a way to >>>>>> heal locks without loosing the property of exclusivity of locks. >>>>>> >>>>> >>>>> Lock-healing of locks acquired while a brick was disconnected need to >>>>> be handled by AFR/EC. However, locks already present at the moment of >>>>> disconnection could be recovered by client xlator itself as long as the >>>>> file has not been closed (which client xlator already knows). >>>>> >>>> >>>> What if another client (say mount-2) took locks at the time of >>>> disconnect from mount-1 and modified the file and unlocked? client xlator >>>> doing the heal may not be a good idea. >>>> >>> >>> To avoid that we should ensure that any lock/unlocks are sent to the >>> client, even if we know it's disconnected, so that client xlator can track >>> them. The alternative is to duplicate and maintain code both on AFR and EC >>> (and not sure if even in DHT depending on how we want to handle some >>> cases). >>> >> >> Didn't understand the solution. I wanted to highlight that client xlator >> by itself can't make a decision about healing locks because it doesn't know >> what happened on other replicas. If we have replica-3 volume and all 3 >> bricks get disconnected to their respective bricks. Now another mount >> process can take a lock on that file modify it and unlock. Now upon >> reconnection, the old mount process which had locks would think it always >> had the lock if client xlator independently tries to heal its own locks >> because file is not closed on it so far. But that is wrong. Let me know if >> it makes sense.... >> > > My point of view is that any configuration with these requirements will > have an appropriate quorum value so that it's impossible to have two or > more partitions of the nodes working at the same time. So, under this > assumptions, mount-1 can be in two situations: > > 1. It has lost a single brick and it's still operational. The other bricks > will continue locked and everything should work fine from the point of view > of the application. Any other application trying to get a lock will fail > due to lack of quorum. When the lost brick comes back and is reconnected, > client xlator will still have the fd reference and locks taken (unless the > application has released the lock or closed the fd, in which case client > xlator should get notified and clear that information), so it should be > able to recover the previous state. > Application could be in blocked state as well if it tries to get blocking lock. So as soon as a disconnect happens, the lock will be granted on that brick to one of the blocked locks. On the other two bricks it would still be blocked. Trying to heal that will require a new operation that is not already present in locks code, which should be able to tell client as well about either changing the lock state to blocked on that brick or to retry lock operation. > > 2. It has lost 2 or 3 bricks. In this case mount-1 has lost quorum and any > operation going to that file should fail with EIO. AFR should send a > special request to client xlator so that it forgets any fd's and locks for > that file. If bricks reconnect after that, no fd reopen or lock recovery > will happen. Eventually the application should close the fd and retry > later. This may succeed to not, depending on whether mount-2 has taken the > lock already or not. > > So, it's true that client xlator doesn't know the state of the other > bricks, but it doesn't need to as long as AFR/EC strictly enforces quorum > and updates client xlator when quorum is lost. > This part seems good. > > I haven't worked out all the details of this approach, but I think it > should work and it's simpler to maintain than trying to do the same for AFR > and EC. > Let us spend some time on this on #gluster-dev when you get some time tomorrow to figure out the complete solution which handles the corner cases too. > > Xavi > > >> >>> A similar thing could be done for open fd, since the current solution >>> duplicates code in AFR and EC, but this is another topic... >>> >>> >>>> >>>>> >>>>> Xavi >>>>> >>>>> >>>>>> What I proposed is a short term solution. mid to long term solution >>>>>> should be lock healing feature implemented in AFR/EC. In fact I had this >>>>>> conversation with +Karampuri, Pranith before >>>>>> posting this msg to ML. >>>>>> >>>>>> >>>>>>> >>>>>>>> However, this use case is not affected if the application don't >>>>>>>> acquire any POSIX locks. So, I am interested in knowing >>>>>>>> * whether your use cases use POSIX locks? >>>>>>>> * Is it feasible for your application to re-open fds and re-acquire >>>>>>>> locks on seeing EBADFD errors? >>>>>>>> >>>>>>> >>>>>>> I think that many applications are not prepared to handle that. >>>>>>> >>>>>> >>>>>> I too suspected that and in fact not too happy with the solution. But >>>>>> went ahead with this mail as I heard implementing lock-heal in AFR will >>>>>> take time and hence there are no alternative short term solutions. >>>>>> >>>>> >>>>>> >>>>>>> Xavi >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>>>>>>> >>>>>>>> regards, >>>>>>>> Raghavendra >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>>> >>>> >>>> -- >>>> Pranith >>>> >>> >> >> -- >> Pranith >> > -- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: From jahernan at redhat.com Wed Mar 27 18:24:57 2019 From: jahernan at redhat.com (Xavi Hernandez) Date: Wed, 27 Mar 2019 19:24:57 +0100 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Wed, 27 Mar 2019, 18:26 Pranith Kumar Karampuri, wrote: > > > On Wed, Mar 27, 2019 at 8:38 PM Xavi Hernandez > wrote: > >> On Wed, Mar 27, 2019 at 2:20 PM Pranith Kumar Karampuri < >> pkarampu at redhat.com> wrote: >> >>> >>> >>> On Wed, Mar 27, 2019 at 6:38 PM Xavi Hernandez >>> wrote: >>> >>>> On Wed, Mar 27, 2019 at 1:13 PM Pranith Kumar Karampuri < >>>> pkarampu at redhat.com> wrote: >>>> >>>>> >>>>> >>>>> On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez >>>>> wrote: >>>>> >>>>>> On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa < >>>>>> rgowdapp at redhat.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Raghavendra, >>>>>>>> >>>>>>>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa < >>>>>>>> rgowdapp at redhat.com> wrote: >>>>>>>> >>>>>>>>> All, >>>>>>>>> >>>>>>>>> Glusterfs cleans up POSIX locks held on an fd when the >>>>>>>>> client/mount through which those locks are held disconnects from >>>>>>>>> bricks/server. This helps Glusterfs to not run into a stale lock problem >>>>>>>>> later (For eg., if application unlocks while the connection was still >>>>>>>>> down). However, this means the lock is no longer exclusive as other >>>>>>>>> applications/clients can acquire the same lock. To communicate that locks >>>>>>>>> are no longer valid, we are planning to mark the fd (which has POSIX locks) >>>>>>>>> bad on a disconnect so that any future operations on that fd will fail, >>>>>>>>> forcing the application to re-open the fd and re-acquire locks it needs [1]. >>>>>>>>> >>>>>>>> >>>>>>>> Wouldn't it be better to retake the locks when the brick is >>>>>>>> reconnected if the lock is still in use ? >>>>>>>> >>>>>>> >>>>>>> There is also a possibility that clients may never reconnect. >>>>>>> That's the primary reason why bricks assume the worst (client will not >>>>>>> reconnect) and cleanup the locks. >>>>>>> >>>>>> >>>>>> True, so it's fine to cleanup the locks. I'm not saying that locks >>>>>> shouldn't be released on disconnect. The assumption is that if the client >>>>>> has really died, it will also disconnect from other bricks, who will >>>>>> release the locks. So, eventually, another client will have enough quorum >>>>>> to attempt a lock that will succeed. In other words, if a client gets >>>>>> disconnected from too many bricks simultaneously (loses Quorum), then that >>>>>> client can be considered as bad and can return errors to the application. >>>>>> This should also cause to release the locks on the remaining connected >>>>>> bricks. >>>>>> >>>>>> On the other hand, if the disconnection is very short and the client >>>>>> has not died, it will keep enough locked files (it has quorum) to avoid >>>>>> other clients to successfully acquire a lock. In this case, if the brick is >>>>>> reconnected, all existing locks should be reacquired to recover the >>>>>> original state before the disconnection. >>>>>> >>>>>> >>>>>>> >>>>>>>> BTW, the referenced bug is not public. Should we open another bug >>>>>>>> to track this ? >>>>>>>> >>>>>>> >>>>>>> I've just opened up the comment to give enough context. I'll open a >>>>>>> bug upstream too. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Note that with AFR/replicate in picture we can prevent errors to >>>>>>>>> application as long as Quorum number of children "never ever" lost >>>>>>>>> connection with bricks after locks have been acquired. I am using the term >>>>>>>>> "never ever" as locks are not healed back after re-connection and hence >>>>>>>>> first disconnect would've marked the fd bad and the fd remains so even >>>>>>>>> after re-connection happens. So, its not just Quorum number of children >>>>>>>>> "currently online", but Quorum number of children "never having >>>>>>>>> disconnected with bricks after locks are acquired". >>>>>>>>> >>>>>>>> >>>>>>>> I think this requisite is not feasible. In a distributed file >>>>>>>> system, sooner or later all bricks will be disconnected. It could be >>>>>>>> because of failures or because an upgrade is done, but it will happen. >>>>>>>> >>>>>>>> The difference here is how long are fd's kept open. If applications >>>>>>>> open and close files frequently enough (i.e. the fd is not kept open more >>>>>>>> time than it takes to have more than Quorum bricks disconnected) then >>>>>>>> there's no problem. The problem can only appear on applications that open >>>>>>>> files for a long time and also use posix locks. In this case, the only good >>>>>>>> solution I see is to retake the locks on brick reconnection. >>>>>>>> >>>>>>> >>>>>>> Agree. But lock-healing should be done only by HA layers like AFR/EC >>>>>>> as only they know whether there are enough online bricks to have prevented >>>>>>> any conflicting lock. Protocol/client itself doesn't have enough >>>>>>> information to do that. If its a plain distribute, I don't see a way to >>>>>>> heal locks without loosing the property of exclusivity of locks. >>>>>>> >>>>>> >>>>>> Lock-healing of locks acquired while a brick was disconnected need to >>>>>> be handled by AFR/EC. However, locks already present at the moment of >>>>>> disconnection could be recovered by client xlator itself as long as the >>>>>> file has not been closed (which client xlator already knows). >>>>>> >>>>> >>>>> What if another client (say mount-2) took locks at the time of >>>>> disconnect from mount-1 and modified the file and unlocked? client xlator >>>>> doing the heal may not be a good idea. >>>>> >>>> >>>> To avoid that we should ensure that any lock/unlocks are sent to the >>>> client, even if we know it's disconnected, so that client xlator can track >>>> them. The alternative is to duplicate and maintain code both on AFR and EC >>>> (and not sure if even in DHT depending on how we want to handle some >>>> cases). >>>> >>> >>> Didn't understand the solution. I wanted to highlight that client xlator >>> by itself can't make a decision about healing locks because it doesn't know >>> what happened on other replicas. If we have replica-3 volume and all 3 >>> bricks get disconnected to their respective bricks. Now another mount >>> process can take a lock on that file modify it and unlock. Now upon >>> reconnection, the old mount process which had locks would think it always >>> had the lock if client xlator independently tries to heal its own locks >>> because file is not closed on it so far. But that is wrong. Let me know if >>> it makes sense.... >>> >> >> My point of view is that any configuration with these requirements will >> have an appropriate quorum value so that it's impossible to have two or >> more partitions of the nodes working at the same time. So, under this >> assumptions, mount-1 can be in two situations: >> >> 1. It has lost a single brick and it's still operational. The other >> bricks will continue locked and everything should work fine from the point >> of view of the application. Any other application trying to get a lock will >> fail due to lack of quorum. When the lost brick comes back and is >> reconnected, client xlator will still have the fd reference and locks taken >> (unless the application has released the lock or closed the fd, in which >> case client xlator should get notified and clear that information), so it >> should be able to recover the previous state. >> > > Application could be in blocked state as well if it tries to get blocking > lock. So as soon as a disconnect happens, the lock will be granted on that > brick to one of the blocked locks. On the other two bricks it would still > be blocked. Trying to heal that will require a new operation that is not > already present in locks code, which should be able to tell client as well > about either changing the lock state to blocked on that brick or to retry > lock operation. > Yes, but this problem exists even if the lock-heal is done by AFR/EC. This is something that needs to be solved anyway, but it's independent of who does the lock-heal. > >> >> 2. It has lost 2 or 3 bricks. In this case mount-1 has lost quorum and >> any operation going to that file should fail with EIO. AFR should send a >> special request to client xlator so that it forgets any fd's and locks for >> that file. If bricks reconnect after that, no fd reopen or lock recovery >> will happen. Eventually the application should close the fd and retry >> later. This may succeed to not, depending on whether mount-2 has taken the >> lock already or not. >> >> So, it's true that client xlator doesn't know the state of the other >> bricks, but it doesn't need to as long as AFR/EC strictly enforces quorum >> and updates client xlator when quorum is lost. >> > > This part seems good. > > >> >> I haven't worked out all the details of this approach, but I think it >> should work and it's simpler to maintain than trying to do the same for AFR >> and EC. >> > > Let us spend some time on this on #gluster-dev when you get some time > tomorrow to figure out the complete solution which handles the corner cases > too. > > >> >> Xavi >> >> >>> >>>> A similar thing could be done for open fd, since the current solution >>>> duplicates code in AFR and EC, but this is another topic... >>>> >>>> >>>>> >>>>>> >>>>>> Xavi >>>>>> >>>>>> >>>>>>> What I proposed is a short term solution. mid to long term solution >>>>>>> should be lock healing feature implemented in AFR/EC. In fact I had this >>>>>>> conversation with +Karampuri, Pranith before >>>>>>> posting this msg to ML. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> However, this use case is not affected if the application don't >>>>>>>>> acquire any POSIX locks. So, I am interested in knowing >>>>>>>>> * whether your use cases use POSIX locks? >>>>>>>>> * Is it feasible for your application to re-open fds and >>>>>>>>> re-acquire locks on seeing EBADFD errors? >>>>>>>>> >>>>>>>> >>>>>>>> I think that many applications are not prepared to handle that. >>>>>>>> >>>>>>> >>>>>>> I too suspected that and in fact not too happy with the solution. >>>>>>> But went ahead with this mail as I heard implementing lock-heal in AFR >>>>>>> will take time and hence there are no alternative short term solutions. >>>>>>> >>>>>> >>>>>>> >>>>>>>> Xavi >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>>>>>>>> >>>>>>>>> regards, >>>>>>>>> Raghavendra >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>>> >>>>> >>>>> -- >>>>> Pranith >>>>> >>>> >>> >>> -- >>> Pranith >>> >> > > -- > Pranith > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Thu Mar 28 02:04:48 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Thu, 28 Mar 2019 07:34:48 +0530 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Wed, Mar 27, 2019 at 8:38 PM Xavi Hernandez wrote: > On Wed, Mar 27, 2019 at 2:20 PM Pranith Kumar Karampuri < > pkarampu at redhat.com> wrote: > >> >> >> On Wed, Mar 27, 2019 at 6:38 PM Xavi Hernandez >> wrote: >> >>> On Wed, Mar 27, 2019 at 1:13 PM Pranith Kumar Karampuri < >>> pkarampu at redhat.com> wrote: >>> >>>> >>>> >>>> On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez >>>> wrote: >>>> >>>>> On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa < >>>>> rgowdapp at redhat.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez >>>>>> wrote: >>>>>> >>>>>>> Hi Raghavendra, >>>>>>> >>>>>>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa < >>>>>>> rgowdapp at redhat.com> wrote: >>>>>>> >>>>>>>> All, >>>>>>>> >>>>>>>> Glusterfs cleans up POSIX locks held on an fd when the client/mount >>>>>>>> through which those locks are held disconnects from bricks/server. This >>>>>>>> helps Glusterfs to not run into a stale lock problem later (For eg., if >>>>>>>> application unlocks while the connection was still down). However, this >>>>>>>> means the lock is no longer exclusive as other applications/clients can >>>>>>>> acquire the same lock. To communicate that locks are no longer valid, we >>>>>>>> are planning to mark the fd (which has POSIX locks) bad on a disconnect so >>>>>>>> that any future operations on that fd will fail, forcing the application to >>>>>>>> re-open the fd and re-acquire locks it needs [1]. >>>>>>>> >>>>>>> >>>>>>> Wouldn't it be better to retake the locks when the brick is >>>>>>> reconnected if the lock is still in use ? >>>>>>> >>>>>> >>>>>> There is also a possibility that clients may never reconnect. That's >>>>>> the primary reason why bricks assume the worst (client will not reconnect) >>>>>> and cleanup the locks. >>>>>> >>>>> >>>>> True, so it's fine to cleanup the locks. I'm not saying that locks >>>>> shouldn't be released on disconnect. The assumption is that if the client >>>>> has really died, it will also disconnect from other bricks, who will >>>>> release the locks. So, eventually, another client will have enough quorum >>>>> to attempt a lock that will succeed. In other words, if a client gets >>>>> disconnected from too many bricks simultaneously (loses Quorum), then that >>>>> client can be considered as bad and can return errors to the application. >>>>> This should also cause to release the locks on the remaining connected >>>>> bricks. >>>>> >>>>> On the other hand, if the disconnection is very short and the client >>>>> has not died, it will keep enough locked files (it has quorum) to avoid >>>>> other clients to successfully acquire a lock. In this case, if the brick is >>>>> reconnected, all existing locks should be reacquired to recover the >>>>> original state before the disconnection. >>>>> >>>>> >>>>>> >>>>>>> BTW, the referenced bug is not public. Should we open another bug to >>>>>>> track this ? >>>>>>> >>>>>> >>>>>> I've just opened up the comment to give enough context. I'll open a >>>>>> bug upstream too. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Note that with AFR/replicate in picture we can prevent errors to >>>>>>>> application as long as Quorum number of children "never ever" lost >>>>>>>> connection with bricks after locks have been acquired. I am using the term >>>>>>>> "never ever" as locks are not healed back after re-connection and hence >>>>>>>> first disconnect would've marked the fd bad and the fd remains so even >>>>>>>> after re-connection happens. So, its not just Quorum number of children >>>>>>>> "currently online", but Quorum number of children "never having >>>>>>>> disconnected with bricks after locks are acquired". >>>>>>>> >>>>>>> >>>>>>> I think this requisite is not feasible. In a distributed file >>>>>>> system, sooner or later all bricks will be disconnected. It could be >>>>>>> because of failures or because an upgrade is done, but it will happen. >>>>>>> >>>>>>> The difference here is how long are fd's kept open. If applications >>>>>>> open and close files frequently enough (i.e. the fd is not kept open more >>>>>>> time than it takes to have more than Quorum bricks disconnected) then >>>>>>> there's no problem. The problem can only appear on applications that open >>>>>>> files for a long time and also use posix locks. In this case, the only good >>>>>>> solution I see is to retake the locks on brick reconnection. >>>>>>> >>>>>> >>>>>> Agree. But lock-healing should be done only by HA layers like AFR/EC >>>>>> as only they know whether there are enough online bricks to have prevented >>>>>> any conflicting lock. Protocol/client itself doesn't have enough >>>>>> information to do that. If its a plain distribute, I don't see a way to >>>>>> heal locks without loosing the property of exclusivity of locks. >>>>>> >>>>> >>>>> Lock-healing of locks acquired while a brick was disconnected need to >>>>> be handled by AFR/EC. However, locks already present at the moment of >>>>> disconnection could be recovered by client xlator itself as long as the >>>>> file has not been closed (which client xlator already knows). >>>>> >>>> >>>> What if another client (say mount-2) took locks at the time of >>>> disconnect from mount-1 and modified the file and unlocked? client xlator >>>> doing the heal may not be a good idea. >>>> >>> >>> To avoid that we should ensure that any lock/unlocks are sent to the >>> client, even if we know it's disconnected, so that client xlator can track >>> them. The alternative is to duplicate and maintain code both on AFR and EC >>> (and not sure if even in DHT depending on how we want to handle some >>> cases). >>> >> >> Didn't understand the solution. I wanted to highlight that client xlator >> by itself can't make a decision about healing locks because it doesn't know >> what happened on other replicas. If we have replica-3 volume and all 3 >> bricks get disconnected to their respective bricks. Now another mount >> process can take a lock on that file modify it and unlock. Now upon >> reconnection, the old mount process which had locks would think it always >> had the lock if client xlator independently tries to heal its own locks >> because file is not closed on it so far. But that is wrong. Let me know if >> it makes sense.... >> > > My point of view is that any configuration with these requirements will > have an appropriate quorum value so that it's impossible to have two or > more partitions of the nodes working at the same time. So, under this > assumptions, mount-1 can be in two situations: > > 1. It has lost a single brick and it's still operational. The other bricks > will continue locked and everything should work fine from the point of view > of the application. Any other application trying to get a lock will fail > due to lack of quorum. When the lost brick comes back and is reconnected, > client xlator will still have the fd reference and locks taken (unless the > application has released the lock or closed the fd, in which case client > xlator should get notified and clear that information), so it should be > able to recover the previous state. > > 2. It has lost 2 or 3 bricks. In this case mount-1 has lost quorum and any > operation going to that file should fail with EIO. AFR should send a > special request to client xlator so that it forgets any fd's and locks for > that file. If bricks reconnect after that, no fd reopen or lock recovery > will happen. Eventually the application should close the fd and retry > later. This may succeed to not, depending on whether mount-2 has taken the > lock already or not. > > So, it's true that client xlator doesn't know the state of the other > bricks, but it doesn't need to as long as AFR/EC strictly enforces quorum > and updates client xlator when quorum is lost. > Just curious. Is there any reason why you think delegating the actual responsibility of re-opening or forgetting the locks to protocol/client is better when compared to AFR/EC doing the actual work of re-opening files and reacquiring locks? Asking this because, in the case of plain distribute, DHT will also have to indicate Quorum loss on every disconnect (as Quorum consisted of just 1 brick). >From what I understand, the design is the same one which me, Pranith, Anoop and Vijay had discussed (in essence) but varies in implementation details. > I haven't worked out all the details of this approach, but I think it > should work and it's simpler to maintain than trying to do the same for AFR > and EC. > > Xavi > > >> >>> A similar thing could be done for open fd, since the current solution >>> duplicates code in AFR and EC, but this is another topic... >>> >>> >>>> >>>>> >>>>> Xavi >>>>> >>>>> >>>>>> What I proposed is a short term solution. mid to long term solution >>>>>> should be lock healing feature implemented in AFR/EC. In fact I had this >>>>>> conversation with +Karampuri, Pranith before >>>>>> posting this msg to ML. >>>>>> >>>>>> >>>>>>> >>>>>>>> However, this use case is not affected if the application don't >>>>>>>> acquire any POSIX locks. So, I am interested in knowing >>>>>>>> * whether your use cases use POSIX locks? >>>>>>>> * Is it feasible for your application to re-open fds and re-acquire >>>>>>>> locks on seeing EBADFD errors? >>>>>>>> >>>>>>> >>>>>>> I think that many applications are not prepared to handle that. >>>>>>> >>>>>> >>>>>> I too suspected that and in fact not too happy with the solution. But >>>>>> went ahead with this mail as I heard implementing lock-heal in AFR will >>>>>> take time and hence there are no alternative short term solutions. >>>>>> >>>>> >>>>>> >>>>>>> Xavi >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>>>>>>> >>>>>>>> regards, >>>>>>>> Raghavendra >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Gluster-users mailing list >>>>>>>> Gluster-users at gluster.org >>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>> >>>>>>> >>>> >>>> -- >>>> Pranith >>>> >>> >> >> -- >> Pranith >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Thu Mar 28 02:20:00 2019 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Thu, 28 Mar 2019 07:50:00 +0530 Subject: [Gluster-devel] Following up on the "Github teams/repo cleanup" Message-ID: The one at I am not sure if the proposal from Michael was agreed to separately and it was done. Also, do we want to do this periodically? Feedback is appreciated. From pgurusid at redhat.com Thu Mar 28 05:18:42 2019 From: pgurusid at redhat.com (Poornima Gurusiddaiah) Date: Thu, 28 Mar 2019 10:48:42 +0530 Subject: [Gluster-devel] [Gluster-Maintainers] GF_CALLOC to GF_MALLOC conversion - is it safe? In-Reply-To: <20190327085421.GG2684@ndevos-x270.lan.nixpanic.net> References: <20190327085421.GG2684@ndevos-x270.lan.nixpanic.net> Message-ID: Regarding the use of malloc vs calloc, i had tried this a while back, though theoretically it should give performance improvement, i was not able to see this in practice for small memory allocations, may be due to the optimizations in the glibc and kernel memory management itself. When i ran a sample program for random size millions of malloc vs calloc and also use it(so that it results in page fault) resulted in no conclusive results on which is more performant, in some runs calloc was slower in some runs malloc was slower. In glibc the calloc is not exactly malloc + memset, after browsing and looking through __libc_calloc(), the kernel always gives zeroed pages for security purpose(with COW), when calloc implementation in libc reuses the memory allocated by its own process then there is a difference in perf of malloc and calloc, but even these are optimised to some extent in glibc. I would personally prefer keeping calloc and use malloc only if its large buffers, and trade off this one nano perf improvement, for the ease of coding and uniformity across the project. As a thumb rule we have been using calloc for smaller memory, we should discuss and conclude on this so it can go in developer-guide(./doc/developer-guide/coding-standard.md) Also, while at the memory alloc optimization discussions, would like to mention about one another observation, long back had tried doing some changes to gf_malloc and gf_calloc to use from the per thread mem pool if available, rather than send it to libc. We saw that there were millions of small mallocs/callocs happening so we tried to modify the code to lookup the mempool for availability for pre allocated memory, with this change we saw no improvement! But there were improvements wrt using iobuf_pool and other pre allocated poo for dict, inode etcl. something we could explore. Regards, Poornima On Wed, Mar 27, 2019 at 2:25 PM Niels de Vos wrote: > On Tue, Mar 26, 2019 at 02:52:33PM -0700, Vijay Bellur wrote: > > On Thu, Mar 21, 2019 at 8:44 AM Yaniv Kaul wrote: > > > > > > > > > > > On Thu, Mar 21, 2019 at 5:23 PM Nithya Balachandran < > nbalacha at redhat.com> > > > wrote: > > > > > >> > > >> > > >> On Thu, 21 Mar 2019 at 16:16, Atin Mukherjee > wrote: > > >> > > >>> All, > > >>> > > >>> In the last few releases of glusterfs, with stability as a primary > theme > > >>> of the releases, there has been lots of changes done on the code > > >>> optimization with an expectation that such changes will have gluster > to > > >>> provide better performance. While many of these changes do help, but > off > > >>> late we have started seeing some diverse effects of them, one > especially > > >>> being the calloc to malloc conversions. While I do understand that > malloc > > >>> syscall will eliminate the extra memset bottleneck which calloc > bears, but > > >>> with recent kernels having in-built strong compiler optimizations I > am not > > >>> sure whether that makes any significant difference, but as I > mentioned > > >>> earlier certainly if this isn't done carefully it can potentially > introduce > > >>> lot of bugs and I'm writing this email to share one of such > experiences. > > >>> > > >>> Sanju & I were having troubles for last two days to figure out why > > >>> https://review.gluster.org/#/c/glusterfs/+/22388/ wasn't working in > > >>> Sanju's system but it had no problems running the same fix in my > gluster > > >>> containers. After spending a significant amount of time, what we now > > >>> figured out is that a malloc call [1] (which was a calloc earlier) > is the > > >>> culprit here. As you all can see, in this function we allocate > txn_id and > > >>> copy the event->txn_id into it through gf_uuid_copy () . But when we > were > > >>> debugging this step wise through gdb, txn_id wasn't exactly copied > with the > > >>> exact event->txn_id and it had some junk values which made the > > >>> glusterd_clear_txn_opinfo to be invoked with a wrong txn_id later on > > >>> resulting the leaks to remain the same which was the original > intention of > > >>> the fix. > > >>> > > >>> This was quite painful to debug and we had to spend some time to > figure > > >>> this out. Considering we have converted many such calls in past, I'd > urge > > >>> that we review all such conversions and see if there're any side > effects to > > >>> it. Otherwise we might end up running into many potential memory > related > > >>> bugs later on. OTOH, going forward I'd request every patch > > >>> owners/maintainers to pay some special attention to these > conversions and > > >>> see they are really beneficial and error free. IMO, general guideline > > >>> should be - for bigger buffers, malloc would make better sense but > has to > > >>> be done carefully, for smaller size, we stick to calloc. > > >>> > > >> > > >>> What do others think about it? > > >>> > > >> > > >> I believe that replacing calloc with malloc everywhere without > adequate > > >> testing and review is not safe and am against doing so for the > following > > >> reasons: > > >> > > > > > > No patch should get in without adequate testing and thorough review. > > > > > > > > > There are lots of interesting points to glean in this thread. However, > this > > particular one caught my attention. How about we introduce a policy that > no > > patch gets merged unless it is thoroughly tested? The onus would be on > the > > developer to provide a .t test case to show completeness in the testing > of > > that patch. If the developer does not or cannot for any reason, we could > > have the maintainer run tests and add a note in gerrit explaining the > tests > > run. This would provide more assurance about the patches being tested > > before getting merged. Obviously, patches that fix typos or that cannot > > affect any functionality need not be subject to this policy. > > > > As far as review thoroughness is concerned, it might be better to mandate > > acks from respective maintainers before merging a patch that affects > > several components. More eyeballs that specialize in particular > > component(s) will hopefully catch some of these issues during the review > > phase. > > Both of these points have always been strongly encouraged. They are also > documented in the > > https://docs.gluster.org/en/latest/Contributors-Guide/Guidelines-For-Maintainers/ > > https://github.com/gluster/glusterdocs/blob/master/docs/Contributors-Guide/Guidelines-For-Maintainers.md > (formatting is broken in the 1st link, but I dont know how to fix it) > > We probably need to apply our own guidelines a little better, and > remember developers that > 90% of the patch(series) should come with a > .t file or added test in an existing one. > > And a big +1 for getting reviews or at least some involvement of the > component maintainers. > > Niels > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jahernan at redhat.com Thu Mar 28 09:07:33 2019 From: jahernan at redhat.com (Xavi Hernandez) Date: Thu, 28 Mar 2019 10:07:33 +0100 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Thu, Mar 28, 2019 at 3:05 AM Raghavendra Gowdappa wrote: > > > On Wed, Mar 27, 2019 at 8:38 PM Xavi Hernandez > wrote: > >> On Wed, Mar 27, 2019 at 2:20 PM Pranith Kumar Karampuri < >> pkarampu at redhat.com> wrote: >> >>> >>> >>> On Wed, Mar 27, 2019 at 6:38 PM Xavi Hernandez >>> wrote: >>> >>>> On Wed, Mar 27, 2019 at 1:13 PM Pranith Kumar Karampuri < >>>> pkarampu at redhat.com> wrote: >>>> >>>>> >>>>> >>>>> On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez >>>>> wrote: >>>>> >>>>>> On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa < >>>>>> rgowdapp at redhat.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Raghavendra, >>>>>>>> >>>>>>>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa < >>>>>>>> rgowdapp at redhat.com> wrote: >>>>>>>> >>>>>>>>> All, >>>>>>>>> >>>>>>>>> Glusterfs cleans up POSIX locks held on an fd when the >>>>>>>>> client/mount through which those locks are held disconnects from >>>>>>>>> bricks/server. This helps Glusterfs to not run into a stale lock problem >>>>>>>>> later (For eg., if application unlocks while the connection was still >>>>>>>>> down). However, this means the lock is no longer exclusive as other >>>>>>>>> applications/clients can acquire the same lock. To communicate that locks >>>>>>>>> are no longer valid, we are planning to mark the fd (which has POSIX locks) >>>>>>>>> bad on a disconnect so that any future operations on that fd will fail, >>>>>>>>> forcing the application to re-open the fd and re-acquire locks it needs [1]. >>>>>>>>> >>>>>>>> >>>>>>>> Wouldn't it be better to retake the locks when the brick is >>>>>>>> reconnected if the lock is still in use ? >>>>>>>> >>>>>>> >>>>>>> There is also a possibility that clients may never reconnect. >>>>>>> That's the primary reason why bricks assume the worst (client will not >>>>>>> reconnect) and cleanup the locks. >>>>>>> >>>>>> >>>>>> True, so it's fine to cleanup the locks. I'm not saying that locks >>>>>> shouldn't be released on disconnect. The assumption is that if the client >>>>>> has really died, it will also disconnect from other bricks, who will >>>>>> release the locks. So, eventually, another client will have enough quorum >>>>>> to attempt a lock that will succeed. In other words, if a client gets >>>>>> disconnected from too many bricks simultaneously (loses Quorum), then that >>>>>> client can be considered as bad and can return errors to the application. >>>>>> This should also cause to release the locks on the remaining connected >>>>>> bricks. >>>>>> >>>>>> On the other hand, if the disconnection is very short and the client >>>>>> has not died, it will keep enough locked files (it has quorum) to avoid >>>>>> other clients to successfully acquire a lock. In this case, if the brick is >>>>>> reconnected, all existing locks should be reacquired to recover the >>>>>> original state before the disconnection. >>>>>> >>>>>> >>>>>>> >>>>>>>> BTW, the referenced bug is not public. Should we open another bug >>>>>>>> to track this ? >>>>>>>> >>>>>>> >>>>>>> I've just opened up the comment to give enough context. I'll open a >>>>>>> bug upstream too. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Note that with AFR/replicate in picture we can prevent errors to >>>>>>>>> application as long as Quorum number of children "never ever" lost >>>>>>>>> connection with bricks after locks have been acquired. I am using the term >>>>>>>>> "never ever" as locks are not healed back after re-connection and hence >>>>>>>>> first disconnect would've marked the fd bad and the fd remains so even >>>>>>>>> after re-connection happens. So, its not just Quorum number of children >>>>>>>>> "currently online", but Quorum number of children "never having >>>>>>>>> disconnected with bricks after locks are acquired". >>>>>>>>> >>>>>>>> >>>>>>>> I think this requisite is not feasible. In a distributed file >>>>>>>> system, sooner or later all bricks will be disconnected. It could be >>>>>>>> because of failures or because an upgrade is done, but it will happen. >>>>>>>> >>>>>>>> The difference here is how long are fd's kept open. If applications >>>>>>>> open and close files frequently enough (i.e. the fd is not kept open more >>>>>>>> time than it takes to have more than Quorum bricks disconnected) then >>>>>>>> there's no problem. The problem can only appear on applications that open >>>>>>>> files for a long time and also use posix locks. In this case, the only good >>>>>>>> solution I see is to retake the locks on brick reconnection. >>>>>>>> >>>>>>> >>>>>>> Agree. But lock-healing should be done only by HA layers like AFR/EC >>>>>>> as only they know whether there are enough online bricks to have prevented >>>>>>> any conflicting lock. Protocol/client itself doesn't have enough >>>>>>> information to do that. If its a plain distribute, I don't see a way to >>>>>>> heal locks without loosing the property of exclusivity of locks. >>>>>>> >>>>>> >>>>>> Lock-healing of locks acquired while a brick was disconnected need to >>>>>> be handled by AFR/EC. However, locks already present at the moment of >>>>>> disconnection could be recovered by client xlator itself as long as the >>>>>> file has not been closed (which client xlator already knows). >>>>>> >>>>> >>>>> What if another client (say mount-2) took locks at the time of >>>>> disconnect from mount-1 and modified the file and unlocked? client xlator >>>>> doing the heal may not be a good idea. >>>>> >>>> >>>> To avoid that we should ensure that any lock/unlocks are sent to the >>>> client, even if we know it's disconnected, so that client xlator can track >>>> them. The alternative is to duplicate and maintain code both on AFR and EC >>>> (and not sure if even in DHT depending on how we want to handle some >>>> cases). >>>> >>> >>> Didn't understand the solution. I wanted to highlight that client xlator >>> by itself can't make a decision about healing locks because it doesn't know >>> what happened on other replicas. If we have replica-3 volume and all 3 >>> bricks get disconnected to their respective bricks. Now another mount >>> process can take a lock on that file modify it and unlock. Now upon >>> reconnection, the old mount process which had locks would think it always >>> had the lock if client xlator independently tries to heal its own locks >>> because file is not closed on it so far. But that is wrong. Let me know if >>> it makes sense.... >>> >> >> My point of view is that any configuration with these requirements will >> have an appropriate quorum value so that it's impossible to have two or >> more partitions of the nodes working at the same time. So, under this >> assumptions, mount-1 can be in two situations: >> >> 1. It has lost a single brick and it's still operational. The other >> bricks will continue locked and everything should work fine from the point >> of view of the application. Any other application trying to get a lock will >> fail due to lack of quorum. When the lost brick comes back and is >> reconnected, client xlator will still have the fd reference and locks taken >> (unless the application has released the lock or closed the fd, in which >> case client xlator should get notified and clear that information), so it >> should be able to recover the previous state. >> >> 2. It has lost 2 or 3 bricks. In this case mount-1 has lost quorum and >> any operation going to that file should fail with EIO. AFR should send a >> special request to client xlator so that it forgets any fd's and locks for >> that file. If bricks reconnect after that, no fd reopen or lock recovery >> will happen. Eventually the application should close the fd and retry >> later. This may succeed to not, depending on whether mount-2 has taken the >> lock already or not. >> >> So, it's true that client xlator doesn't know the state of the other >> bricks, but it doesn't need to as long as AFR/EC strictly enforces quorum >> and updates client xlator when quorum is lost. >> > > Just curious. Is there any reason why you think delegating the actual > responsibility of re-opening or forgetting the locks to protocol/client is > better when compared to AFR/EC doing the actual work of re-opening files > and reacquiring locks? Asking this because, in the case of plain > distribute, DHT will also have to indicate Quorum loss on every disconnect > (as Quorum consisted of just 1 brick). > The basic reason is that doing that on AFR and EC requires code duplication. The code is not expected to be simple either, so it can contain bugs or it could require improvements eventually. Every time we want to do a change, we should fix both AFR and EC, but this has not happened in many cases in the past on features that are already duplicated in AFR and EC, so it's quite unlikely that this will happen in the future. Regarding the requirement of sending a quorum loss notification from DHT, I agree it's a new thing, but it's way simpler to do than the fd and lock heal logic. Xavi > From what I understand, the design is the same one which me, Pranith, > Anoop and Vijay had discussed (in essence) but varies in implementation > details. > > >> I haven't worked out all the details of this approach, but I think it >> should work and it's simpler to maintain than trying to do the same for AFR >> and EC. >> >> Xavi >> >> >>> >>>> A similar thing could be done for open fd, since the current solution >>>> duplicates code in AFR and EC, but this is another topic... >>>> >>>> >>>>> >>>>>> >>>>>> Xavi >>>>>> >>>>>> >>>>>>> What I proposed is a short term solution. mid to long term solution >>>>>>> should be lock healing feature implemented in AFR/EC. In fact I had this >>>>>>> conversation with +Karampuri, Pranith before >>>>>>> posting this msg to ML. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> However, this use case is not affected if the application don't >>>>>>>>> acquire any POSIX locks. So, I am interested in knowing >>>>>>>>> * whether your use cases use POSIX locks? >>>>>>>>> * Is it feasible for your application to re-open fds and >>>>>>>>> re-acquire locks on seeing EBADFD errors? >>>>>>>>> >>>>>>>> >>>>>>>> I think that many applications are not prepared to handle that. >>>>>>>> >>>>>>> >>>>>>> I too suspected that and in fact not too happy with the solution. >>>>>>> But went ahead with this mail as I heard implementing lock-heal in AFR >>>>>>> will take time and hence there are no alternative short term solutions. >>>>>>> >>>>>> >>>>>>> >>>>>>>> Xavi >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>>>>>>>> >>>>>>>>> regards, >>>>>>>>> Raghavendra >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Gluster-users mailing list >>>>>>>>> Gluster-users at gluster.org >>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>> >>>>>>>> >>>>> >>>>> -- >>>>> Pranith >>>>> >>>> >>> >>> -- >>> Pranith >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgowdapp at redhat.com Thu Mar 28 09:18:12 2019 From: rgowdapp at redhat.com (Raghavendra Gowdappa) Date: Thu, 28 Mar 2019 14:48:12 +0530 Subject: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks In-Reply-To: References: Message-ID: On Thu, Mar 28, 2019 at 2:37 PM Xavi Hernandez wrote: > On Thu, Mar 28, 2019 at 3:05 AM Raghavendra Gowdappa > wrote: > >> >> >> On Wed, Mar 27, 2019 at 8:38 PM Xavi Hernandez >> wrote: >> >>> On Wed, Mar 27, 2019 at 2:20 PM Pranith Kumar Karampuri < >>> pkarampu at redhat.com> wrote: >>> >>>> >>>> >>>> On Wed, Mar 27, 2019 at 6:38 PM Xavi Hernandez >>>> wrote: >>>> >>>>> On Wed, Mar 27, 2019 at 1:13 PM Pranith Kumar Karampuri < >>>>> pkarampu at redhat.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez >>>>>> wrote: >>>>>> >>>>>>> On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa < >>>>>>> rgowdapp at redhat.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez < >>>>>>>> jahernan at redhat.com> wrote: >>>>>>>> >>>>>>>>> Hi Raghavendra, >>>>>>>>> >>>>>>>>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa < >>>>>>>>> rgowdapp at redhat.com> wrote: >>>>>>>>> >>>>>>>>>> All, >>>>>>>>>> >>>>>>>>>> Glusterfs cleans up POSIX locks held on an fd when the >>>>>>>>>> client/mount through which those locks are held disconnects from >>>>>>>>>> bricks/server. This helps Glusterfs to not run into a stale lock problem >>>>>>>>>> later (For eg., if application unlocks while the connection was still >>>>>>>>>> down). However, this means the lock is no longer exclusive as other >>>>>>>>>> applications/clients can acquire the same lock. To communicate that locks >>>>>>>>>> are no longer valid, we are planning to mark the fd (which has POSIX locks) >>>>>>>>>> bad on a disconnect so that any future operations on that fd will fail, >>>>>>>>>> forcing the application to re-open the fd and re-acquire locks it needs [1]. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Wouldn't it be better to retake the locks when the brick is >>>>>>>>> reconnected if the lock is still in use ? >>>>>>>>> >>>>>>>> >>>>>>>> There is also a possibility that clients may never reconnect. >>>>>>>> That's the primary reason why bricks assume the worst (client will not >>>>>>>> reconnect) and cleanup the locks. >>>>>>>> >>>>>>> >>>>>>> True, so it's fine to cleanup the locks. I'm not saying that locks >>>>>>> shouldn't be released on disconnect. The assumption is that if the client >>>>>>> has really died, it will also disconnect from other bricks, who will >>>>>>> release the locks. So, eventually, another client will have enough quorum >>>>>>> to attempt a lock that will succeed. In other words, if a client gets >>>>>>> disconnected from too many bricks simultaneously (loses Quorum), then that >>>>>>> client can be considered as bad and can return errors to the application. >>>>>>> This should also cause to release the locks on the remaining connected >>>>>>> bricks. >>>>>>> >>>>>>> On the other hand, if the disconnection is very short and the client >>>>>>> has not died, it will keep enough locked files (it has quorum) to avoid >>>>>>> other clients to successfully acquire a lock. In this case, if the brick is >>>>>>> reconnected, all existing locks should be reacquired to recover the >>>>>>> original state before the disconnection. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> BTW, the referenced bug is not public. Should we open another bug >>>>>>>>> to track this ? >>>>>>>>> >>>>>>>> >>>>>>>> I've just opened up the comment to give enough context. I'll open a >>>>>>>> bug upstream too. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Note that with AFR/replicate in picture we can prevent errors to >>>>>>>>>> application as long as Quorum number of children "never ever" lost >>>>>>>>>> connection with bricks after locks have been acquired. I am using the term >>>>>>>>>> "never ever" as locks are not healed back after re-connection and hence >>>>>>>>>> first disconnect would've marked the fd bad and the fd remains so even >>>>>>>>>> after re-connection happens. So, its not just Quorum number of children >>>>>>>>>> "currently online", but Quorum number of children "never having >>>>>>>>>> disconnected with bricks after locks are acquired". >>>>>>>>>> >>>>>>>>> >>>>>>>>> I think this requisite is not feasible. In a distributed file >>>>>>>>> system, sooner or later all bricks will be disconnected. It could be >>>>>>>>> because of failures or because an upgrade is done, but it will happen. >>>>>>>>> >>>>>>>>> The difference here is how long are fd's kept open. If >>>>>>>>> applications open and close files frequently enough (i.e. the fd is not >>>>>>>>> kept open more time than it takes to have more than Quorum bricks >>>>>>>>> disconnected) then there's no problem. The problem can only appear on >>>>>>>>> applications that open files for a long time and also use posix locks. In >>>>>>>>> this case, the only good solution I see is to retake the locks on brick >>>>>>>>> reconnection. >>>>>>>>> >>>>>>>> >>>>>>>> Agree. But lock-healing should be done only by HA layers like >>>>>>>> AFR/EC as only they know whether there are enough online bricks to have >>>>>>>> prevented any conflicting lock. Protocol/client itself doesn't have enough >>>>>>>> information to do that. If its a plain distribute, I don't see a way to >>>>>>>> heal locks without loosing the property of exclusivity of locks. >>>>>>>> >>>>>>> >>>>>>> Lock-healing of locks acquired while a brick was disconnected need >>>>>>> to be handled by AFR/EC. However, locks already present at the moment of >>>>>>> disconnection could be recovered by client xlator itself as long as the >>>>>>> file has not been closed (which client xlator already knows). >>>>>>> >>>>>> >>>>>> What if another client (say mount-2) took locks at the time of >>>>>> disconnect from mount-1 and modified the file and unlocked? client xlator >>>>>> doing the heal may not be a good idea. >>>>>> >>>>> >>>>> To avoid that we should ensure that any lock/unlocks are sent to the >>>>> client, even if we know it's disconnected, so that client xlator can track >>>>> them. The alternative is to duplicate and maintain code both on AFR and EC >>>>> (and not sure if even in DHT depending on how we want to handle some >>>>> cases). >>>>> >>>> >>>> Didn't understand the solution. I wanted to highlight that client >>>> xlator by itself can't make a decision about healing locks because it >>>> doesn't know what happened on other replicas. If we have replica-3 volume >>>> and all 3 bricks get disconnected to their respective bricks. Now another >>>> mount process can take a lock on that file modify it and unlock. Now upon >>>> reconnection, the old mount process which had locks would think it always >>>> had the lock if client xlator independently tries to heal its own locks >>>> because file is not closed on it so far. But that is wrong. Let me know if >>>> it makes sense.... >>>> >>> >>> My point of view is that any configuration with these requirements will >>> have an appropriate quorum value so that it's impossible to have two or >>> more partitions of the nodes working at the same time. So, under this >>> assumptions, mount-1 can be in two situations: >>> >>> 1. It has lost a single brick and it's still operational. The other >>> bricks will continue locked and everything should work fine from the point >>> of view of the application. Any other application trying to get a lock will >>> fail due to lack of quorum. When the lost brick comes back and is >>> reconnected, client xlator will still have the fd reference and locks taken >>> (unless the application has released the lock or closed the fd, in which >>> case client xlator should get notified and clear that information), so it >>> should be able to recover the previous state. >>> >>> 2. It has lost 2 or 3 bricks. In this case mount-1 has lost quorum and >>> any operation going to that file should fail with EIO. AFR should send a >>> special request to client xlator so that it forgets any fd's and locks for >>> that file. If bricks reconnect after that, no fd reopen or lock recovery >>> will happen. Eventually the application should close the fd and retry >>> later. This may succeed to not, depending on whether mount-2 has taken the >>> lock already or not. >>> >>> So, it's true that client xlator doesn't know the state of the other >>> bricks, but it doesn't need to as long as AFR/EC strictly enforces quorum >>> and updates client xlator when quorum is lost. >>> >> >> Just curious. Is there any reason why you think delegating the actual >> responsibility of re-opening or forgetting the locks to protocol/client is >> better when compared to AFR/EC doing the actual work of re-opening files >> and reacquiring locks? Asking this because, in the case of plain >> distribute, DHT will also have to indicate Quorum loss on every disconnect >> (as Quorum consisted of just 1 brick). >> > > The basic reason is that doing that on AFR and EC requires code > duplication. The code is not expected to be simple either, so it can > contain bugs or it could require improvements eventually. Every time we > want to do a change, we should fix both AFR and EC, but this has not > happened in many cases in the past on features that are already duplicated > in AFR and EC, so it's quite unlikely that this will happen in the future. > That's a good reason. +1. > > Regarding the requirement of sending a quorum loss notification from DHT, > I agree it's a new thing, but it's way simpler to do than the fd and lock > heal logic. > > Xavi > > >> From what I understand, the design is the same one which me, Pranith, >> Anoop and Vijay had discussed (in essence) but varies in implementation >> details. >> >> >>> I haven't worked out all the details of this approach, but I think it >>> should work and it's simpler to maintain than trying to do the same for AFR >>> and EC. >>> >>> Xavi >>> >>> >>>> >>>>> A similar thing could be done for open fd, since the current solution >>>>> duplicates code in AFR and EC, but this is another topic... >>>>> >>>>> >>>>>> >>>>>>> >>>>>>> Xavi >>>>>>> >>>>>>> >>>>>>>> What I proposed is a short term solution. mid to long term solution >>>>>>>> should be lock healing feature implemented in AFR/EC. In fact I had this >>>>>>>> conversation with +Karampuri, Pranith before >>>>>>>> posting this msg to ML. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>> However, this use case is not affected if the application don't >>>>>>>>>> acquire any POSIX locks. So, I am interested in knowing >>>>>>>>>> * whether your use cases use POSIX locks? >>>>>>>>>> * Is it feasible for your application to re-open fds and >>>>>>>>>> re-acquire locks on seeing EBADFD errors? >>>>>>>>>> >>>>>>>>> >>>>>>>>> I think that many applications are not prepared to handle that. >>>>>>>>> >>>>>>>> >>>>>>>> I too suspected that and in fact not too happy with the solution. >>>>>>>> But went ahead with this mail as I heard implementing lock-heal in AFR >>>>>>>> will take time and hence there are no alternative short term solutions. >>>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> Xavi >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1689375#c7 >>>>>>>>>> >>>>>>>>>> regards, >>>>>>>>>> Raghavendra >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Gluster-users mailing list >>>>>>>>>> Gluster-users at gluster.org >>>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>>>>> >>>>>>>>> >>>>>> >>>>>> -- >>>>>> Pranith >>>>>> >>>>> >>>> >>>> -- >>>> Pranith >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From nbalacha at redhat.com Thu Mar 28 09:38:16 2019 From: nbalacha at redhat.com (Nithya Balachandran) Date: Thu, 28 Mar 2019 15:08:16 +0530 Subject: [Gluster-devel] [Gluster-users] Prioritise local bricks for IO? In-Reply-To: References: <29221907.583.1553599314586.JavaMail.zimbra@li.nux.ro> Message-ID: On Wed, 27 Mar 2019 at 20:27, Poornima Gurusiddaiah wrote: > This feature is not under active development as it was not used widely. > AFAIK its not supported feature. > +Nithya +Raghavendra for further clarifications. > This is not actively supported - there has been no work done on this feature for a long time. Regards, Nithya > > Regards, > Poornima > > On Wed, Mar 27, 2019 at 12:33 PM Lucian wrote: > >> Oh, that's just what the doctor ordered! >> Hope it works, thanks >> >> On 27 March 2019 03:15:57 GMT, Vlad Kopylov wrote: >>> >>> I don't remember if it still in works >>> NUFA >>> >>> https://github.com/gluster/glusterfs-specs/blob/master/done/Features/nufa.md >>> >>> v >>> >>> On Tue, Mar 26, 2019 at 7:27 AM Nux! wrote: >>> >>>> Hello, >>>> >>>> I'm trying to set up a distributed backup storage (no replicas), but >>>> I'd like to prioritise the local bricks for any IO done on the volume. >>>> This will be a backup stor, so in other words, I'd like the files to be >>>> written locally if there is space, so as to save the NICs for other traffic. >>>> >>>> Anyone knows how this might be achievable, if at all? >>>> >>>> -- >>>> Sent from the Delta quadrant using Borg technology! >>>> >>>> Nux! >>>> www.nux.ro >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>> >> -- >> Sent from my Android device with K-9 Mail. Please excuse my brevity. >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From jstrunk at redhat.com Thu Mar 28 18:03:59 2019 From: jstrunk at redhat.com (John Strunk) Date: Thu, 28 Mar 2019 14:03:59 -0400 Subject: [Gluster-devel] Following up on the "Github teams/repo cleanup" In-Reply-To: References: Message-ID: Thanks for bringing this to the list. I think this is a good set of guidelines, and we should publicly post and enforce them once agreement is reached. The integrity of the gluster github org is important for the future of the project. -John On Wed, Mar 27, 2019 at 10:21 PM Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > The one at < > https://lists.gluster.org/pipermail/gluster-infra/2018-June/004589.html> > I am not sure if the proposal from Michael was agreed to separately > and it was done. Also, do we want to do this periodically? > > Feedback is appreciated. > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Thu Mar 28 18:38:18 2019 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Fri, 29 Mar 2019 00:08:18 +0530 Subject: [Gluster-devel] Following up on the "Github teams/repo cleanup" In-Reply-To: References: Message-ID: On Thu, Mar 28, 2019 at 11:34 PM John Strunk wrote: > > Thanks for bringing this to the list. > > I think this is a good set of guidelines, and we should publicly post and enforce them once agreement is reached. > The integrity of the gluster github org is important for the future of the project. > I agree. And so, I am looking forward to additional individuals/maintainers agreeing to this so that we can codify it under the Gluster.org Github org too. > > On Wed, Mar 27, 2019 at 10:21 PM Sankarshan Mukhopadhyay wrote: >> >> The one at >> I am not sure if the proposal from Michael was agreed to separately >> and it was done. Also, do we want to do this periodically? >> >> Feedback is appreciated. From vbellur at redhat.com Thu Mar 28 19:33:55 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Thu, 28 Mar 2019 12:33:55 -0700 Subject: [Gluster-devel] Following up on the "Github teams/repo cleanup" In-Reply-To: References: Message-ID: On Thu, Mar 28, 2019 at 11:39 AM Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > On Thu, Mar 28, 2019 at 11:34 PM John Strunk wrote: > > > > Thanks for bringing this to the list. > > > > I think this is a good set of guidelines, and we should publicly post > and enforce them once agreement is reached. > > The integrity of the gluster github org is important for the future of > the project. > > > > I agree. And so, I am looking forward to additional > individuals/maintainers agreeing to this so that we can codify it > under the Gluster.org Github org too. > Looks good to me. While at this, I would also like us to think about evolving some guidelines for creating a new repository in the gluster github organization. Right now, a bug report does get a new repository created and I feel that the process could be a bit more involved to ensure that we host projects with the right content in github. Thanks, Vijay > > > > > On Wed, Mar 27, 2019 at 10:21 PM Sankarshan Mukhopadhyay < > sankarshan.mukhopadhyay at gmail.com> wrote: > >> > >> The one at < > https://lists.gluster.org/pipermail/gluster-infra/2018-June/004589.html> > >> I am not sure if the proposal from Michael was agreed to separately > >> and it was done. Also, do we want to do this periodically? > >> > >> Feedback is appreciated. > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sankarshan.mukhopadhyay at gmail.com Thu Mar 28 19:58:53 2019 From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay) Date: Fri, 29 Mar 2019 01:28:53 +0530 Subject: [Gluster-devel] Following up on the "Github teams/repo cleanup" In-Reply-To: References: Message-ID: On Fri 29 Mar, 2019, 01:04 Vijay Bellur, wrote: > > > On Thu, Mar 28, 2019 at 11:39 AM Sankarshan Mukhopadhyay < > sankarshan.mukhopadhyay at gmail.com> wrote: > >> On Thu, Mar 28, 2019 at 11:34 PM John Strunk wrote: >> > >> > Thanks for bringing this to the list. >> > >> > I think this is a good set of guidelines, and we should publicly post >> and enforce them once agreement is reached. >> > The integrity of the gluster github org is important for the future of >> the project. >> > >> >> I agree. And so, I am looking forward to additional >> individuals/maintainers agreeing to this so that we can codify it >> under the Gluster.org Github org too. >> > > > Looks good to me. > > While at this, I would also like us to think about evolving some > guidelines for creating a new repository in the gluster github > organization. Right now, a bug report does get a new repository created and > I feel that the process could be a bit more involved to ensure that we > host projects with the right content in github. > The bug ensures that there is a recorded trail for the request. What might be the additional detail required which can emphasize on greater involvement? At this point, I don't see many fly-by-night projects. Just inactive ones and those too for myriad reasons. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vbellur at redhat.com Thu Mar 28 21:40:27 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Thu, 28 Mar 2019 14:40:27 -0700 Subject: [Gluster-devel] Following up on the "Github teams/repo cleanup" In-Reply-To: References: Message-ID: On Thu, Mar 28, 2019 at 12:59 PM Sankarshan Mukhopadhyay < sankarshan.mukhopadhyay at gmail.com> wrote: > > > On Fri 29 Mar, 2019, 01:04 Vijay Bellur, wrote: > >> >> >> On Thu, Mar 28, 2019 at 11:39 AM Sankarshan Mukhopadhyay < >> sankarshan.mukhopadhyay at gmail.com> wrote: >> >>> On Thu, Mar 28, 2019 at 11:34 PM John Strunk wrote: >>> > >>> > Thanks for bringing this to the list. >>> > >>> > I think this is a good set of guidelines, and we should publicly post >>> and enforce them once agreement is reached. >>> > The integrity of the gluster github org is important for the future of >>> the project. >>> > >>> >>> I agree. And so, I am looking forward to additional >>> individuals/maintainers agreeing to this so that we can codify it >>> under the Gluster.org Github org too. >>> >> >> >> Looks good to me. >> >> While at this, I would also like us to think about evolving some >> guidelines for creating a new repository in the gluster github >> organization. Right now, a bug report does get a new repository created and >> I feel that the process could be a bit more involved to ensure that we >> host projects with the right content in github. >> > > The bug ensures that there is a recorded trail for the request. What might > be the additional detail required which can emphasize on greater > involvement? At this point, I don't see many fly-by-night projects. Just > inactive ones and those too for myriad reasons. > I certainly did not intend fly-by-night projects. Having a tracking mechanism is indeed desirable. Of the 98 projects we host in github, there are projects without commits, without licenses and some whose relevance to Gluster is not easily understood. We could evaluate factors like these before hosting a repo under the gluster umbrella. Thanks, Vijay > > >> _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From avishwan at redhat.com Fri Mar 29 04:29:33 2019 From: avishwan at redhat.com (Aravinda) Date: Fri, 29 Mar 2019 09:59:33 +0530 Subject: [Gluster-devel] Following up on the "Github teams/repo cleanup" In-Reply-To: References: Message-ID: On Fri, 2019-03-29 at 00:08 +0530, Sankarshan Mukhopadhyay wrote: > On Thu, Mar 28, 2019 at 11:34 PM John Strunk > wrote: > > Thanks for bringing this to the list. > > > > I think this is a good set of guidelines, and we should publicly > > post and enforce them once agreement is reached. > > The integrity of the gluster github org is important for the future > > of the project. > > > > I agree. And so, I am looking forward to additional > individuals/maintainers agreeing to this so that we can codify it > under the Gluster.org Github org too. We can create new Github organization similar to facebookarchive( https://github.com/facebookarchive), and move all unmaintained repositories to that organization. I strongly feel separate github organization is required to host new projects. Once the project becomes mature we can promote to Gluster organization. All new project should be created under this organization instead of creating under main Gluster organization. Inspirations: - https://github.com/rust-lang-nursery - https://github.com/facebookincubator - https://github.com/cloudfoundry-incubator > > > On Wed, Mar 27, 2019 at 10:21 PM Sankarshan Mukhopadhyay < > > sankarshan.mukhopadhyay at gmail.com> wrote: > > > The one at < > > > https://lists.gluster.org/pipermail/gluster-infra/2018-June/004589.html > > > > > > > I am not sure if the proposal from Michael was agreed to > > > separately > > > and it was done. Also, do we want to do this periodically? > > > > > > Feedback is appreciated. > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel -- regards Aravinda From hgowtham at redhat.com Fri Mar 29 06:12:33 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Fri, 29 Mar 2019 11:42:33 +0530 Subject: [Gluster-devel] Upgrade testing to gluster 6 Message-ID: Hello Gluster users, As you all aware that glusterfs-6 is out, we would like to inform you that, we have spent a significant amount of time in testing glusterfs-6 in upgrade scenarios. We have done upgrade testing to glusterfs-6 from various releases like 3.12, 4.1 and 5.3. As glusterfs-6 has got in a lot of changes, we wanted to test those portions. There were xlators (and respective options to enable/disable them) added and deprecated in glusterfs-6 from various versions [1]. We had to check the following upgrade scenarios for all such options Identified in [1]: 1) option never enabled and upgraded 2) option enabled and then upgraded 3) option enabled and then disabled and then upgraded We weren't manually able to check all the combinations for all the options. So the options involving enabling and disabling xlators were prioritized. The below are the result of the ones tested. Never enabled and upgraded: checked from 3.12, 4.1, 5.3 to 6 the upgrade works. Enabled and upgraded: Tested for tier which is deprecated, It is not a recommended upgrade. As expected the volume won't be consumable and will have a few more issues as well. Tested with 3.12, 4.1 and 5.3 to 6 upgrade. Enabled, disabled before upgrade. Tested for tier with 3.12 and the upgrade went fine. There is one common issue to note in every upgrade. The node being upgraded is going into disconnected state. You have to flush the iptables and the restart glusterd on all nodes to fix this. The testing for enabling new options is still pending. The new options won't cause as much issues as the deprecated ones so this was put at the end of the priority list. It would be nice to get contributions for this. For the disable testing, tier was used as it covers most of the xlator that was removed. And all of these tests were done on a replica 3 volume. Note: This is only for upgrade testing of the newly added and removed xlators. Does not involve the normal tests for the xlator. If you have any questions, please feel free to reach us. [1] https://docs.google.com/spreadsheets/d/1nh7T5AXaV6kc5KgILOy2pEqjzC3t_R47f1XUXSVFetI/edit?usp=sharing Regards, Hari and Sanju. From suprasam at crossmeta.org Fri Mar 29 07:04:49 2019 From: suprasam at crossmeta.org (Supra Sammandam) Date: Fri, 29 Mar 2019 00:04:49 -0700 Subject: [Gluster-devel] FUSE client work on Windows Message-ID: Hi, Is there any client work on Windows happening? I would like to do this port with Crossmeta FUSE github.com/crossmeta/cxfuse I am looking for relevant source directories that is lean and mean for the Client FUSE . Thanks in advance Sam -------------- next part -------------- An HTML attachment was scrubbed... URL: From hgowtham at redhat.com Fri Mar 29 10:07:54 2019 From: hgowtham at redhat.com (Hari Gowtham) Date: Fri, 29 Mar 2019 15:37:54 +0530 Subject: [Gluster-devel] Upgrade testing to gluster 6 In-Reply-To: References: Message-ID: Hi, Have added a few more info that was missed earlier. The disconnect issue being minor we are working on it with a lower priority. But yes, it will be fixed soon. The bug to track this is: https://bugzilla.redhat.com/show_bug.cgi?id=1694010 The workaround to get over this if it happens is to, upgrade the nodes one after other to the latest version. Once the upgrade is done, 1) kill the glusterd process alone in all the nodes using the command "pkill glusterd" 2) then do a "iptables -F" to flush the iptables. 3) start glusterd using "glusterd" Note: users can use systemctl stop/start glusterd.service command as well instead of the above to kill and start glusterd. On Fri, Mar 29, 2019 at 11:42 AM Hari Gowtham wrote: > > Hello Gluster users, > > As you all aware that glusterfs-6 is out, we would like to inform you > that, we have spent a significant amount of time in testing > glusterfs-6 in upgrade scenarios. We have done upgrade testing to > glusterfs-6 from various releases like 3.12, 4.1 and 5.3. > > As glusterfs-6 has got in a lot of changes, we wanted to test those portions. > There were xlators (and respective options to enable/disable them) > added and deprecated in glusterfs-6 from various versions [1]. > > We had to check the following upgrade scenarios for all such options > Identified in [1]: > 1) option never enabled and upgraded > 2) option enabled and then upgraded > 3) option enabled and then disabled and then upgraded > > We weren't manually able to check all the combinations for all the options. > So the options involving enabling and disabling xlators were prioritized. > The below are the result of the ones tested. > > Never enabled and upgraded: > checked from 3.12, 4.1, 5.3 to 6 the upgrade works. > > Enabled and upgraded: > Tested for tier which is deprecated, It is not a recommended upgrade. > As expected the volume won't be consumable and will have a few more > issues as well. > Tested with 3.12, 4.1 and 5.3 to 6 upgrade. > > Enabled, disabled before upgrade. > Tested for tier with 3.12 and the upgrade went fine. > > There is one common issue to note in every upgrade. The node being > upgraded is going into disconnected state. You have to flush the iptables > and the restart glusterd on all nodes to fix this. > > The testing for enabling new options is still pending. The new options > won't cause as much issues as the deprecated ones so this was put at > the end of the priority list. It would be nice to get contributions > for this. > > For the disable testing, tier was used as it covers most of the xlator > that was removed. And all of these tests were done on a replica 3 volume. > > Note: This is only for upgrade testing of the newly added and removed > xlators. Does not involve the normal tests for the xlator. > > If you have any questions, please feel free to reach us. > > [1] https://docs.google.com/spreadsheets/d/1nh7T5AXaV6kc5KgILOy2pEqjzC3t_R47f1XUXSVFetI/edit?usp=sharing > > Regards, > Hari and Sanju. -- Regards, Hari Gowtham. From amukherj at redhat.com Fri Mar 29 13:41:35 2019 From: amukherj at redhat.com (Atin Mukherjee) Date: Fri, 29 Mar 2019 19:11:35 +0530 Subject: [Gluster-devel] Quick update on glusterd's volume scalability improvements Message-ID: All, As many of you already know that the design logic with which GlusterD (here on to be referred as GD1) was implemented has some fundamental scalability bottlenecks at design level, especially around it's way of handshaking configuration meta data and replicating them across all the peers. While the initial design was adopted with a factor in mind that GD1 will have to deal with just few tens of nodes/peers and volumes, the magnitude of the scaling bottleneck this design can bring in was never realized and estimated. Ever since Gluster has been adopted in container storage land as one of the storage backends, the business needs have changed. From tens of volumes, the requirements have translated to hundreds and now to thousands. We introduced brick multiplexing which had given some relief to have a better control on the memory footprint when having many number of bricks/volumes hosted in the node, but this wasn't enough. In one of our (I represent Red Hat) customer's deployment we had seen on a 3 nodes cluster, whenever the number of volumes go beyond ~1500 and for some reason if one of the storage pods get rebooted, the overall time it takes to complete the overall handshaking (not only in a factor of n X n peer handshaking but also the number of volume iterations, building up the dictionary and sending it over the write) consumes a huge time as part of the handshaking process, the hard timeout of an rpc request which is 10 minutes gets expired and we see cluster going into a state where none of the cli commands go through and get stuck. With such problem being around and more demand of volume scalability, we started looking into these areas in GD1 to focus on improving (a) volume scalability (b) node scalability. While (b) is a separate topic for some other day we're going to focus on more on (a) today. While looking into this volume scalability problem with a deep dive, we realized that most of the bottleneck which was causing the overall delay in the friend handshaking and exchanging handshake packets between peers in the cluster was iterating over the in-memory data structures of the volumes, putting them into the dictionary sequentially. With 2k like volumes the function glusterd_add_volumes_to_export_dict () was quite costly and most time consuming. From pstack output when glusterd instance was restarted in one of the pods, we could always see that control was iterating in this function. Based on our testing on a 16 vCPU, 32 GB RAM 3 nodes cluster, this function itself took almost *7.5 minutes . *The bottleneck is primarily because of sequential iteration of volumes, sequentially updating the dictionary with lot of (un)necessary keys. So what we tried out was making this loop to work on a worker thread model so that multiple threads can process a range of volume list and not all of them so that we can get more parallelism within glusterd. But with that we still didn't see any improvement and the primary reason for that was our dictionary APIs need locking. So the next idea was to actually make threads work on multiple dictionaries and then once all the volumes are iterated the subsequent dictionaries to be merged into a single one. Along with these changes there are few other improvements done on skipping comparison of snapshots if there's no snap available, excluding tiering keys if the volume type is not tier. With this enhancement [1] we see the overall time it took to complete building up the dictionary from the in-memory structure is *2 minutes 18 seconds* which is close* ~3x* improvement. We firmly believe that with this improvement, we should be able to scale up to 2000 volumes on a 3 node cluster and that'd help our users to get benefited with supporting more PVCs/volumes. Patch [1] is still in testing and might undergo few minor changes. But we welcome you for review and comment on it. We plan to get this work completed, tested and release in glusterfs-7. Last but not the least, I'd like to give a shout to Mohit Agrawal (In cc) for all the work done on this for last few days. Thank you Mohit! [1] https://review.gluster.org/#/c/glusterfs/+/22445/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From xhernandez at redhat.com Fri Mar 29 18:25:52 2019 From: xhernandez at redhat.com (Xavi Hernandez) Date: Fri, 29 Mar 2019 19:25:52 +0100 Subject: [Gluster-devel] Issue with posix locks Message-ID: Hi all, there is one potential problem with posix locks when used in a replicated or dispersed volume. Some background: Posix locks allow any process to lock a region of a file multiple times, but a single unlock on a given region will release all previous locks. Locked regions can be different for each lock request and they can overlap. The resulting lock will cover the union of all locked regions. A single unlock (the region doesn't necessarily need to match any of the ranges used for locking) will create a "hole" in the currently locked region, independently of how many times a lock request covered that region. For this reason, the locks xlator simply combines the locked regions that are requested, but it doesn't track each individual lock range. Under normal circumstances this works fine. But there are some cases where this behavior is not sufficient. For example, suppose we have a replica 3 volume with quorum = 2. Given the special nature of posix locks, AFR sends the lock request sequentially to each one of the bricks, to avoid that conflicting lock requests from other clients could require to unlock an already locked region on the client that has not got enough successful locks (i.e. quorum). An unlock here not only would cancel the current lock request. It would also cancel any previously acquired lock. However, when something goes wrong (a brick dies during a lock request, or there's a network partition or some other weird situation), it could happen that even using sequential locking, only one brick succeeds the lock request. In this case, AFR should cancel the previous lock (and it does), but this also cancels any previously acquired lock on that region, which is not good. A similar thing can happen if we try to recover (heal) posix locks that were active after a brick has been disconnected (for any reason) and then reconnected. To fix all these situations we need to change the way posix locks are managed by locks xlator. One possibility would be to embed the lock request inside an inode transaction using inodelk. Since inodelks do not suffer this problem, the follwing posix lock could be sent safely. However this implies an additional network request, which could cause some performance impact. Eager-locking could minimize the impact in some cases. However this approach won't work for lock recovery after a disconnect. Another possibility is to send a special partial posix lock request which won't be immediately merged with already existing locks once granted. An additional confirmation request of the partial posix lock will be required to fully grant the current lock and merge it with the existing ones. This requires a new network request, which will add latency, and makes everything more complex since there would be more combinations of states in which something could fail. So I think one possible solution would be the following: 1. Keep each posix lock as an independent object in locks xlator. This will make it possible to "invalidate" any already granted lock without affecting already established locks. 2. Additionally, we'll keep a sorted list of non-overlapping segments of locked regions. And we'll count, for each region, how many locks are referencing it. One lock can reference multiple segments, and each segment can be referenced by multiple locks. 3. An additional lock request that overlaps with an existing segment, can cause this segment to be split to satisfy the non-overlapping property. 4. When an unlock request is received, all segments intersecting with the region are eliminated (it may require some segment splits on the edges), and the unlocked region is subtracted from each lock associated to the segment. If a lock gets an empty region, it's removed. 5. We'll create a special "remove lock" request that doesn't unlock a region but removes an already granted lock. This will decrease the number of references to each of the segments this lock was covering. If some segment reaches 0, it's removed. Otherwise it remains there. This special request will only be used internally to cancel already acquired locks that cannot be fully granted due to quorum issues or any other problem. In some weird cases, the list of segments can be huge (many locks overlapping only on a single byte, so each segment represents only one byte). We can try to find some smarter structure that minimizes this problem or limit the number of segments (for example returning ENOLCK when there are too many). What do you think ? Xavi -------------- next part -------------- An HTML attachment was scrubbed... URL: From vbellur at redhat.com Sat Mar 30 02:36:21 2019 From: vbellur at redhat.com (Vijay Bellur) Date: Fri, 29 Mar 2019 19:36:21 -0700 Subject: [Gluster-devel] [Gluster-users] Quick update on glusterd's volume scalability improvements In-Reply-To: References: Message-ID: On Fri, Mar 29, 2019 at 6:42 AM Atin Mukherjee wrote: > All, > > As many of you already know that the design logic with which GlusterD > (here on to be referred as GD1) was implemented has some fundamental > scalability bottlenecks at design level, especially around it's way of > handshaking configuration meta data and replicating them across all the > peers. While the initial design was adopted with a factor in mind that GD1 > will have to deal with just few tens of nodes/peers and volumes, the > magnitude of the scaling bottleneck this design can bring in was never > realized and estimated. > > Ever since Gluster has been adopted in container storage land as one of > the storage backends, the business needs have changed. From tens of > volumes, the requirements have translated to hundreds and now to thousands. > We introduced brick multiplexing which had given some relief to have a > better control on the memory footprint when having many number of > bricks/volumes hosted in the node, but this wasn't enough. In one of our (I > represent Red Hat) customer's deployment we had seen on a 3 nodes cluster, > whenever the number of volumes go beyond ~1500 and for some reason if one > of the storage pods get rebooted, the overall time it takes to complete the > overall handshaking (not only in a factor of n X n peer handshaking but > also the number of volume iterations, building up the dictionary and > sending it over the write) consumes a huge time as part of the handshaking > process, the hard timeout of an rpc request which is 10 minutes gets > expired and we see cluster going into a state where none of the cli > commands go through and get stuck. > > With such problem being around and more demand of volume scalability, we > started looking into these areas in GD1 to focus on improving (a) volume > scalability (b) node scalability. While (b) is a separate topic for some > other day we're going to focus on more on (a) today. > > While looking into this volume scalability problem with a deep dive, we > realized that most of the bottleneck which was causing the overall delay in > the friend handshaking and exchanging handshake packets between peers in > the cluster was iterating over the in-memory data structures of the > volumes, putting them into the dictionary sequentially. With 2k like > volumes the function glusterd_add_volumes_to_export_dict () was quite > costly and most time consuming. From pstack output when glusterd instance > was restarted in one of the pods, we could always see that control was > iterating in this function. Based on our testing on a 16 vCPU, 32 GB RAM 3 > nodes cluster, this function itself took almost *7.5 minutes . *The > bottleneck is primarily because of sequential iteration of volumes, > sequentially updating the dictionary with lot of (un)necessary keys. > > So what we tried out was making this loop to work on a worker thread model > so that multiple threads can process a range of volume list and not all of > them so that we can get more parallelism within glusterd. But with that we > still didn't see any improvement and the primary reason for that was our > dictionary APIs need locking. So the next idea was to actually make threads > work on multiple dictionaries and then once all the volumes are iterated > the subsequent dictionaries to be merged into a single one. Along with > these changes there are few other improvements done on skipping comparison > of snapshots if there's no snap available, excluding tiering keys if the > volume type is not tier. With this enhancement [1] we see the overall time > it took to complete building up the dictionary from the in-memory structure > is *2 minutes 18 seconds* which is close* ~3x* improvement. We firmly > believe that with this improvement, we should be able to scale up to 2000 > volumes on a 3 node cluster and that'd help our users to get benefited with > supporting more PVCs/volumes. > > Patch [1] is still in testing and might undergo few minor changes. But we > welcome you for review and comment on it. We plan to get this work > completed, tested and release in glusterfs-7. > > Last but not the least, I'd like to give a shout to Mohit Agrawal (In cc) > for all the work done on this for last few days. Thank you Mohit! > > This sounds good! Thank you for the update on this work. Did you ever consider using etcd with GD1 (like as it is used with GD2)? Having etcd as a backing store for configuration could remove expensive handshaking as well as persistence of configuration on every node. I am interested in understanding if you are aware of any drawbacks with that approach. If there haven't been any thoughts in that direction, it might be a fun experiment to try. Thanks, Vijay -------------- next part -------------- An HTML attachment was scrubbed... URL: From amukherj at redhat.com Sat Mar 30 04:16:29 2019 From: amukherj at redhat.com (Atin Mukherjee) Date: Sat, 30 Mar 2019 09:46:29 +0530 Subject: [Gluster-devel] [Gluster-users] Quick update on glusterd's volume scalability improvements In-Reply-To: References: Message-ID: On Sat, 30 Mar 2019 at 08:06, Vijay Bellur wrote: > > > On Fri, Mar 29, 2019 at 6:42 AM Atin Mukherjee > wrote: > >> All, >> >> As many of you already know that the design logic with which GlusterD >> (here on to be referred as GD1) was implemented has some fundamental >> scalability bottlenecks at design level, especially around it's way of >> handshaking configuration meta data and replicating them across all the >> peers. While the initial design was adopted with a factor in mind that GD1 >> will have to deal with just few tens of nodes/peers and volumes, the >> magnitude of the scaling bottleneck this design can bring in was never >> realized and estimated. >> >> Ever since Gluster has been adopted in container storage land as one of >> the storage backends, the business needs have changed. From tens of >> volumes, the requirements have translated to hundreds and now to thousands. >> We introduced brick multiplexing which had given some relief to have a >> better control on the memory footprint when having many number of >> bricks/volumes hosted in the node, but this wasn't enough. In one of our (I >> represent Red Hat) customer's deployment we had seen on a 3 nodes cluster, >> whenever the number of volumes go beyond ~1500 and for some reason if one >> of the storage pods get rebooted, the overall time it takes to complete the >> overall handshaking (not only in a factor of n X n peer handshaking but >> also the number of volume iterations, building up the dictionary and >> sending it over the write) consumes a huge time as part of the handshaking >> process, the hard timeout of an rpc request which is 10 minutes gets >> expired and we see cluster going into a state where none of the cli >> commands go through and get stuck. >> >> With such problem being around and more demand of volume scalability, we >> started looking into these areas in GD1 to focus on improving (a) volume >> scalability (b) node scalability. While (b) is a separate topic for some >> other day we're going to focus on more on (a) today. >> >> While looking into this volume scalability problem with a deep dive, we >> realized that most of the bottleneck which was causing the overall delay in >> the friend handshaking and exchanging handshake packets between peers in >> the cluster was iterating over the in-memory data structures of the >> volumes, putting them into the dictionary sequentially. With 2k like >> volumes the function glusterd_add_volumes_to_export_dict () was quite >> costly and most time consuming. From pstack output when glusterd instance >> was restarted in one of the pods, we could always see that control was >> iterating in this function. Based on our testing on a 16 vCPU, 32 GB RAM 3 >> nodes cluster, this function itself took almost *7.5 minutes . *The >> bottleneck is primarily because of sequential iteration of volumes, >> sequentially updating the dictionary with lot of (un)necessary keys. >> >> So what we tried out was making this loop to work on a worker thread >> model so that multiple threads can process a range of volume list and not >> all of them so that we can get more parallelism within glusterd. But with >> that we still didn't see any improvement and the primary reason for that >> was our dictionary APIs need locking. So the next idea was to actually make >> threads work on multiple dictionaries and then once all the volumes are >> iterated the subsequent dictionaries to be merged into a single one. Along >> with these changes there are few other improvements done on skipping >> comparison of snapshots if there's no snap available, excluding tiering >> keys if the volume type is not tier. With this enhancement [1] we see the >> overall time it took to complete building up the dictionary from the >> in-memory structure is *2 minutes 18 seconds* which is close* ~3x* >> improvement. We firmly believe that with this improvement, we should be >> able to scale up to 2000 volumes on a 3 node cluster and that'd help our >> users to get benefited with supporting more PVCs/volumes. >> >> Patch [1] is still in testing and might undergo few minor changes. But we >> welcome you for review and comment on it. We plan to get this work >> completed, tested and release in glusterfs-7. >> >> Last but not the least, I'd like to give a shout to Mohit Agrawal (In cc) >> for all the work done on this for last few days. Thank you Mohit! >> >> > > This sounds good! Thank you for the update on this work. > > Did you ever consider using etcd with GD1 (like as it is used with GD2)? > Honestly I had thought about it few times, but the primary reason was not to go forward with that direction was the bandwidth as such improvements isn?t a short term and tiny tasks, and also to keep in mind that GD2 tasks were in our plate too. If any other contributors are willing to take this up, I am more than happy to collaborate and provide guidance. Having etcd as a backing store for configuration could remove expensive > handshaking as well as persistence of configuration on every node. I am > interested in understanding if you are aware of any drawbacks with that > approach. If there haven't been any thoughts in that direction, it might be > a fun experiment to try. > > Thanks, > Vijay > -- - Atin (atinm) -------------- next part -------------- An HTML attachment was scrubbed... URL: From skoduri at redhat.com Sun Mar 31 17:58:58 2019 From: skoduri at redhat.com (Soumya Koduri) Date: Sun, 31 Mar 2019 23:28:58 +0530 Subject: [Gluster-devel] Issue with posix locks In-Reply-To: References: Message-ID: <7a1adf3a-bdc9-a1ef-6f3c-e69d17c37adc@redhat.com> On 3/29/19 11:55 PM, Xavi Hernandez wrote: > Hi all, > > there is one potential problem with posix locks when used in a > replicated or dispersed volume. > > Some background: > > Posix locks allow any process to lock a region of a file multiple times, > but a single unlock on a given region will release all previous locks. > Locked regions can be different for each lock request and they can > overlap. The resulting lock will cover the union of all locked regions. > A single unlock (the region doesn't necessarily need to match any of the > ranges used for locking) will create a "hole" in the currently locked > region, independently of how many times a lock request covered that region. > > For this reason, the locks xlator simply combines the locked regions > that are requested, but it doesn't track each individual lock range. > > Under normal circumstances this works fine. But there are some cases > where this behavior is not sufficient. For example, suppose we have a > replica 3 volume with quorum = 2. Given the special nature of posix > locks, AFR sends the lock request sequentially to each one of the > bricks, to avoid that conflicting lock requests from other clients could > require to unlock an already locked region on the client that has not > got enough successful locks (i.e. quorum). An unlock here not only would > cancel the current lock request. It would also cancel any previously > acquired lock. > I may not have fully understood, please correct me. AFAIU, lk xlator merges locks only if both the lk-owner and the client opaque matches. In the case which you have mentioned above, considering clientA acquired locks on majority of quorum (say nodeA and nodeB) and clientB on nodeC alone- clientB now has to unlock/cancel the lock it acquired on nodeC. You are saying the it could pose a problem if there were already successful locks taken by clientB for the same region which would get unlocked by this particular unlock request..right? Assuming the previous locks acquired by clientB are shared (otherwise clientA wouldn't have got granted lock for the same region on nodeA & nodeB), they would still hold true on nodeA & nodeB as the unlock request was sent to only nodeC. Since the majority of quorum nodes still hold the locks by clientB, this isn't serious issue IMO. I haven't looked into heal part but would like to understand if this is really an issue in normal scenarios as well. Thanks, Soumya > However, when something goes wrong (a brick dies during a lock request, > or there's a network partition or some other weird situation), it could > happen that even using sequential locking, only one brick succeeds the > lock request. In this case, AFR should cancel the previous lock (and it > does), but this also cancels any previously acquired lock on that > region, which is not good. > > A similar thing can happen if we try to recover (heal) posix locks that > were active after a brick has been disconnected (for any reason) and > then reconnected. > > To fix all these situations we need to change the way posix locks are > managed by locks xlator. One possibility would be to embed the lock > request inside an inode transaction using inodelk. Since inodelks do not > suffer this problem, the follwing posix lock could be sent safely. > However this implies an additional network request, which could cause > some performance impact. Eager-locking could minimize the impact in some > cases. However this approach won't work for lock recovery after a > disconnect. > > Another possibility is to send a special partial posix lock request > which won't be immediately merged with already existing locks once > granted. An additional confirmation request of the partial posix lock > will be required to fully grant the current lock and merge it with the > existing ones. This requires a new network request, which will add > latency, and makes everything more complex since there would be more > combinations of states in which something could fail. > > So I think one possible solution would be the following: > > 1. Keep each posix lock as an independent object in locks xlator. This > will make it possible to "invalidate" any already granted lock without > affecting already established locks. > > 2. Additionally, we'll keep a sorted list of non-overlapping segments of > locked regions. And we'll count, for each region, how many locks are > referencing it. One lock can reference multiple segments, and each > segment can be referenced by multiple locks. > > 3. An additional lock request that overlaps with an existing segment, > can cause this segment to be split to satisfy the non-overlapping property. > > 4. When an unlock request is received, all segments intersecting with > the region are eliminated (it may require some segment splits on the > edges), and the unlocked region is subtracted from each lock associated > to the segment. If a lock gets an empty region, it's removed. > > 5. We'll create a special "remove lock" request that doesn't unlock a > region but removes an already granted lock. This will decrease the > number of references to each of the segments this lock was covering. If > some segment reaches 0, it's removed. Otherwise it remains there. This > special request will only be used internally to cancel already acquired > locks that cannot be fully granted due to quorum issues or any other > problem. > > In some weird cases, the list of segments can be huge (many locks > overlapping only on a single byte, so each segment represents only one > byte). We can try to find some smarter structure that minimizes this > problem or limit the number of segments (for example returning ENOLCK > when there are too many). > > What do you think ? > > Xavi > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel > From budic at onholyground.com Mon Mar 25 15:21:17 2019 From: budic at onholyground.com (Darrell Budic) Date: Mon, 25 Mar 2019 15:21:17 -0000 Subject: [Gluster-devel] [Gluster-users] Proposal: Changes in Gluster Community meetings In-Reply-To: References: Message-ID: <62104B6F-99CF-4C22-80FC-9C177F73E897@onholyground.com> As a user, I?d like to visit more of these, but the time slot is my 3AM. Any possibility for a rolling schedule (move meeting +6 hours each week with rolling attendance from maintainers?) or an occasional regional meeting 12 hours opposed to the one you?re proposing? -Darrell > On Mar 25, 2019, at 4:25 AM, Amar Tumballi Suryanarayan wrote: > > All, > > We currently have 3 meetings which are public: > > 1. Maintainer's Meeting > > - Runs once in 2 weeks (on Mondays), and current attendance is around 3-5 on an avg, and not much is discussed. > - Without majority attendance, we can't take any decisions too. > > 2. Community meeting > > - Supposed to happen on #gluster-meeting, every 2 weeks, and is the only meeting which is for 'Community/Users'. Others are for developers as of now. > Sadly attendance is getting closer to 0 in recent times. > > 3. GCS meeting > > - We started it as an effort inside Red Hat gluster team, and opened it up for community from Jan 2019, but the attendance was always from RHT members, and haven't seen any traction from wider group. > > So, I have a proposal to call out for cancelling all these meeting, and keeping just 1 weekly 'Community' meeting, where even topics related to maintainers and GCS and other projects can be discussed. > > I have a template of a draft template @ https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g > > Please feel free to suggest improvements, both in agenda and in timings. So, we can have more participation from members of community, which allows more user - developer interactions, and hence quality of project. > > Waiting for feedbacks, > > Regards, > Amar > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.spisla at iternity.com Sat Mar 23 20:31:37 2019 From: david.spisla at iternity.com (David Spisla) Date: Sat, 23 Mar 2019 20:31:37 -0000 Subject: [Gluster-devel] Gluster version EOL date In-Reply-To: References: Message-ID: Hello Abhisek, take a look on this: https://lists.gluster.org/pipermail/announce/2018-July/000103.html and this https://www.gluster.org/release-schedule/ Gluster 5.5 is a patch release. Gluster 5.x will be spported until v8.0 Regards David Spisla David Spisla Software Engineer david.spisla at iternity.com +49 761 59034852 iTernity GmbH Heinrich-von-Stephan-Str. 21 79100 Freiburg Deutschland Website Newsletter Support Portal iTernity GmbH. Gesch?ftsf?hrer: Ralf Steinemann. ?Eingetragen beim Amtsgericht Freiburg: HRB-Nr. 701332. ?USt.Id DE242664311. [v01.023] Von: gluster-devel-bounces at gluster.org Im Auftrag von ABHISHEK PALIWAL Gesendet: Freitag, 22. M?rz 2019 11:45 An: Gluster Devel Betreff: [Gluster-devel] Gluster version EOL date Hi, As per gluster community seems the latest version is 5.5. Could any one tell me what would be the EOL date for version 5.5? is it after 12 month of release date or what? -- Regards Abhishek Paliwal -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image342382.png Type: image/png Size: 382 bytes Desc: image342382.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image385764.png Type: image/png Size: 412 bytes Desc: image385764.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image606047.png Type: image/png Size: 6545 bytes Desc: image606047.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image543558.png Type: image/png Size: 7734 bytes Desc: image543558.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image723657.png Type: image/png Size: 522 bytes Desc: image723657.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image532499.png Type: image/png Size: 591 bytes Desc: image532499.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image672369.png Type: image/png Size: 775 bytes Desc: image672369.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image720985.png Type: image/png Size: 508 bytes Desc: image720985.png URL: