[Gluster-users] Upgrade to 4.1.2 geo-replication does not work

Kotresh Hiremath Ravishankar khiremat at redhat.com
Mon Sep 3 09:47:05 UTC 2018


Hi krishna,

I see no error in the shared logs. The only errro messages I see are during
geo-rep stop. That is expected.
Could you share the steps you used to created geo-rep setup?

Thanks,
Kotresh HR

On Mon, Sep 3, 2018 at 1:02 PM, Krishna Verma <kverma at cadence.com> wrote:

> Hi Kotesh,
>
>
>
> Below is the cat output of gsyncd.log file generating on my master server.
>
>
>
> And I am using 4.1.3 version only all my gluster nodes.
>
> [root at gluster-poc-noida distvol]# gluster --version | grep glusterfs
>
> glusterfs 4.1.3
>
>
>
>
>
> [root at gluster-poc-noida distvol]# cat /var/log/glusterfs/geo-
> replication/glusterdist_gluster-poc-sj_glusterdist/gsyncd.log
>
> [2018-09-03 04:01:52.424609] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 04:01:52.526323] I [gsyncd(status):297:main] <top>: Using
> session config file       path=/var/lib/glusterd/geo-
> replication/glusterdist_gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 06:55:41.326411] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 06:55:49.676120] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 06:55:50.406042] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 06:56:52.847537] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 06:57:03.778448] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 06:57:25.86958] I [gsyncd(config-get):297:main] <top>: Using
> session config file    path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 06:57:25.855273] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 06:58:09.294239] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 06:59:39.255487] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 06:59:39.355753] I [gsyncd(status):297:main] <top>: Using
> session config file       path=/var/lib/glusterd/geo-
> replication/glusterdist_gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:00:26.311767] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:03:29.205226] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:03:30.131258] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:10:34.679677] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:10:35.653928] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:26:24.438854] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:26:25.495117] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:26.159113] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:26.216475] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:26.932451] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:26.988286] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:26.992789] E [syncdutils(worker
> /data/gluster-dist/distvol):305:log_raise_exception] <top>: connection to
> peer is broken
>
> [2018-09-03 07:27:26.994750] E [syncdutils(worker
> /data/gluster-dist/distvol):801:errlog] Popen: command returned error
> cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
> /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto
> -S /tmp/gsyncd-aux-ssh-X8iHv1/86bcbaf188167a3859c3267081671312.sock
> gluster-poc-sj /nonexistent/gsyncd slave glusterdist
> gluster-poc-sj::glusterdist --master-node gluster-poc-noida
> --master-node-id 098c16c6-8dff-490a-a2e8-c8cb328fcbb3 --master-brick
> /data/gluster-dist/distvol --local-node gluster-poc-sj --local-node-id
> e54f2759-4c56-40dd-89e1-e10c3037d48b --slave-timeout 120
> --slave-log-level INFO --slave-gluster-log-level INFO
> --slave-gluster-command-dir /usr/local/sbin/       error=255
>
> [2018-09-03 07:27:26.994971] E [syncdutils(worker
> /data/gluster-dist/distvol):805:logerr] Popen: ssh> Killed by signal 15.
>
> [2018-09-03 07:27:27.7174] I [repce(agent /data/gluster-dist/distvol):80:service_loop]
> RepceServer: terminating on reaching EOF.
>
> [2018-09-03 07:27:27.15156] I [gsyncdstatus(monitor):244:set_worker_status]
> GeorepStatus: Worker Status Change  status=Faulty
>
> [2018-09-03 07:27:28.52725] I [gsyncd(monitor-status):297:main] <top>:
> Using session config file        path=/var/lib/glusterd/geo-
> replication/glusterdist_gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:28.64521] I [subcmds(monitor-status):19:subcmd_monitor_status]
> <top>: Monitor Status Change   status=Stopped
>
> [2018-09-03 07:27:35.345937] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:35.444247] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:36.181122] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:36.281459] I [gsyncd(monitor):297:main] <top>: Using
> session config file      path=/var/lib/glusterd/geo-
> replication/glusterdist_gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:39.782480] I [gsyncdstatus(monitor):244:set_worker_status]
> GeorepStatus: Worker Status Change status=Initializing...
>
> [2018-09-03 07:27:40.321157] I [monitor(monitor):158:monitor] Monitor:
> starting gsyncd worker   brick=/data/gluster-dist/distvol
> slave_node=gluster-poc-sj
>
> [2018-09-03 07:27:40.376172] I [gsyncd(agent /data/gluster-dist/distvol):297:main]
> <top>: Using session config file     path=/var/lib/glusterd/geo-
> replication/glusterdist_gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:40.377144] I [changelogagent(agent
> /data/gluster-dist/distvol):72:__init__] ChangelogAgent: Agent
> listining...
>
> [2018-09-03 07:27:40.378150] I [gsyncd(worker /data/gluster-dist/distvol):297:main]
> <top>: Using session config file    path=/var/lib/glusterd/geo-
> replication/glusterdist_gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:40.391185] I [resource(worker
> /data/gluster-dist/distvol):1377:connect_remote] SSH: Initializing SSH
> connection between master and slave...
>
> [2018-09-03 07:27:43.752819] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:43.848619] I [gsyncd(status):297:main] <top>: Using
> session config file       path=/var/lib/glusterd/geo-
> replication/glusterdist_gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:45.365627] I [resource(worker
> /data/gluster-dist/distvol):1424:connect_remote] SSH: SSH connection
> between master and slave established.       duration=4.9743
>
> [2018-09-03 07:27:45.365866] I [resource(worker
> /data/gluster-dist/distvol):1096:connect] GLUSTER: Mounting gluster
> volume locally...
>
> [2018-09-03 07:27:46.388974] I [resource(worker
> /data/gluster-dist/distvol):1119:connect] GLUSTER: Mounted gluster
> volume       duration=1.0230
>
> [2018-09-03 07:27:46.389206] I [subcmds(worker /data/gluster-dist/distvol):70:subcmd_worker]
> <top>: Worker spawn successful. Acknowledging back to monitor
>
> [2018-09-03 07:27:48.401196] I [master(worker /data/gluster-dist/distvol):1593:register]
> _GMaster: Working dir  path=/var/lib/misc/gluster/
> gsyncd/glusterdist_gluster-poc-sj_glusterdist/data-gluster-dist-distvol
>
> [2018-09-03 07:27:48.401477] I [resource(worker
> /data/gluster-dist/distvol):1282:service_loop] GLUSTER: Register time
> time=1535959668
>
> [2018-09-03 07:27:49.176095] I [gsyncdstatus(worker
> /data/gluster-dist/distvol):277:set_active] GeorepStatus: Worker Status
> Change      status=Active
>
> [2018-09-03 07:27:49.177079] I [gsyncdstatus(worker
> /data/gluster-dist/distvol):249:set_worker_crawl_status] GeorepStatus:
> Crawl Status Change  status=History Crawl
>
> [2018-09-03 07:27:49.177339] I [master(worker /data/gluster-dist/distvol):1507:crawl]
> _GMaster: starting history crawl  turns=1 stime=(1535701378, 0)
> entry_stime=(1535701378, 0)     etime=1535959669
>
> [2018-09-03 07:27:50.179210] I [master(worker /data/gluster-dist/distvol):1536:crawl]
> _GMaster: slave's time    stime=(1535701378, 0)
>
> [2018-09-03 07:27:51.300096] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:51.399027] I [gsyncd(status):297:main] <top>: Using
> session config file       path=/var/lib/glusterd/geo-
> replication/glusterdist_gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:27:52.510271] I [master(worker /data/gluster-dist/distvol):1944:syncjob]
> Syncer: Sync Time Taken duration=1.6146 num_files=1     job=2
> return_code=0
>
> [2018-09-03 07:27:52.514487] I [master(worker /data/gluster-dist/distvol):1374:process]
> _GMaster: Entry Time Taken      MKD=0   MKN=0   LIN=0   SYM=0    REN=1
> RMD=0   CRE=0   duration=0.2745 UNL=0
>
> [2018-09-03 07:27:52.514615] I [master(worker /data/gluster-dist/distvol):1384:process]
> _GMaster: Data/Metadata Time Taken      SETA=1  SETX=0
> meta_duration=0.2691     data_duration=1.7883    DATA=1  XATT=0
>
> [2018-09-03 07:27:52.514844] I [master(worker /data/gluster-dist/distvol):1394:process]
> _GMaster: Batch Completed       changelog_end=1535701379entry_stime=(1535701378,
> 0)      changelog_start=1535701379      stime=(1535701378, 0)
> duration=2.3353 num_changelogs=1        mode=history_changelog
>
> [2018-09-03 07:27:52.515224] I [master(worker /data/gluster-dist/distvol):1552:crawl]
> _GMaster: finished history crawl  endtime=1535959662
> stime=(1535701378, 0)    entry_stime=(1535701378, 0)
>
> [2018-09-03 07:28:01.706876] I [gsyncd(config-get):297:main] <top>: Using
> session config file   path=/var/lib/glusterd/geo-replication/glusterdist_
> gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:28:01.803858] I [gsyncd(status):297:main] <top>: Using
> session config file       path=/var/lib/glusterd/geo-
> replication/glusterdist_gluster-poc-sj_glusterdist/gsyncd.conf
>
> [2018-09-03 07:28:03.521949] I [master(worker /data/gluster-dist/distvol):1507:crawl]
> _GMaster: starting history crawl  turns=2 stime=(1535701378, 0)
> entry_stime=(1535701378, 0)     etime=1535959683
>
> [2018-09-03 07:28:03.523086] I [master(worker /data/gluster-dist/distvol):1552:crawl]
> _GMaster: finished history crawl  endtime=1535959677
> stime=(1535701378, 0)    entry_stime=(1535701378, 0)
>
> [2018-09-03 07:28:04.62274] I [gsyncdstatus(worker
> /data/gluster-dist/distvol):249:set_worker_crawl_status] GeorepStatus:
> Crawl Status Change   status=Changelog Crawl
>
> [root at gluster-poc-noida distvol]#
>
>
>
> *From:* Kotresh Hiremath Ravishankar <khiremat at redhat.com>
> *Sent:* Monday, September 3, 2018 12:44 PM
>
> *To:* Krishna Verma <kverma at cadence.com>
> *Cc:* Sunny Kumar <sunkumar at redhat.com>; Gluster Users <
> gluster-users at gluster.org>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
> Hi Krishna,
>
> The log is not complete. If you are re-trying, could you please try it out
> on 4.1.3 and share the logs.
>
> Thanks,
>
> Kotresh HR
>
>
>
> On Mon, Sep 3, 2018 at 12:42 PM, Krishna Verma <kverma at cadence.com> wrote:
>
> Hi Kotresh,
>
>
>
> Please find the log files attached.
>
>
>
> Request you to please have a look.
>
>
>
> /Krishna
>
>
>
>
>
>
>
> *From:* Kotresh Hiremath Ravishankar <khiremat at redhat.com>
> *Sent:* Monday, September 3, 2018 10:19 AM
>
>
> *To:* Krishna Verma <kverma at cadence.com>
> *Cc:* Sunny Kumar <sunkumar at redhat.com>; Gluster Users <
> gluster-users at gluster.org>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
> Hi Krishna,
>
> Indexing is the feature used by Hybrid crawl which only makes crawl
> faster. It has nothing to do with missing data sync.
>
> Could you please share the complete log file of the session where the
> issue is encountered ?
>
> Thanks,
>
> Kotresh HR
>
>
>
> On Mon, Sep 3, 2018 at 9:33 AM, Krishna Verma <kverma at cadence.com> wrote:
>
> Hi Kotresh/Support,
>
>
>
> Request your help to get it fix. My slave is not getting sync with master.
> When I restart the session after doing the indexing off then only it shows
> the file at slave but that is also blank with zero size.
>
>
>
> At master: file size is 5.8 GB.
>
>
>
> [root at gluster-poc-noida distvol]# du -sh 17.10.v001.20171023-201021_
> 17020_GPLV3.tar.gz
>
> 5.8G    17.10.v001.20171023-201021_17020_GPLV3.tar.gz
>
> [root at gluster-poc-noida distvol]#
>
>
>
> But at slave, after doing the “indexing off” and restart the session and
> then wait for 2 days. It shows only 4.9 GB copied.
>
>
>
> [root at gluster-poc-sj distvol]# du -sh 17.10.v001.20171023-201021_
> 17020_GPLV3.tar.gz
>
> 4.9G    17.10.v001.20171023-201021_17020_GPLV3.tar.gz
>
> [root at gluster-poc-sj distvol]#
>
>
>
> Similarly, I tested for small file of size 1.2 GB only that is still
> showing “0” size at slave  after days waiting time.
>
>
>
> At Master:
>
>
>
> [root at gluster-poc-noida distvol]# du -sh rflowTestInt18.08-b001.t.Z
>
> 1.2G    rflowTestInt18.08-b001.t.Z
>
> [root at gluster-poc-noida distvol]#
>
>
>
> At Slave:
>
>
>
> [root at gluster-poc-sj distvol]# du -sh rflowTestInt18.08-b001.t.Z
>
> 0       rflowTestInt18.08-b001.t.Z
>
> [root at gluster-poc-sj distvol]#
>
>
>
> Below is my distributed volume info :
>
>
>
> [root at gluster-poc-noida distvol]# gluster volume info glusterdist
>
>
>
> Volume Name: glusterdist
>
> Type: Distribute
>
> Volume ID: af5b2915-7170-4b5e-aee8-7e68757b9bf1
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 2
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: gluster-poc-noida:/data/gluster-dist/distvol
>
> Brick2: noi-poc-gluster:/data/gluster-dist/distvol
>
> Options Reconfigured:
>
> changelog.changelog: on
>
> geo-replication.ignore-pid-check: on
>
> geo-replication.indexing: on
>
> transport.address-family: inet
>
> nfs.disable: on
>
> [root at gluster-poc-noida distvol]#
>
>
>
> Please help to fix, I believe its not a normal behavior of gluster rsync.
>
>
>
> /Krishna
>
> *From:* Krishna Verma
> *Sent:* Friday, August 31, 2018 12:42 PM
> *To:* 'Kotresh Hiremath Ravishankar' <khiremat at redhat.com>
> *Cc:* Sunny Kumar <sunkumar at redhat.com>; Gluster Users <
> gluster-users at gluster.org>
> *Subject:* RE: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> Hi Kotresh,
>
>
>
> I have tested the geo replication over distributed volumes with 2*2
> gluster setup.
>
>
>
> [root at gluster-poc-noida ~]# gluster volume geo-replication glusterdist
> gluster-poc-sj::glusterdist status
>
>
>
> MASTER NODE          MASTER VOL     MASTER BRICK                  SLAVE
> USER    SLAVE                          SLAVE NODE         STATUS    CRAWL
> STATUS       LAST_SYNCED
>
> ------------------------------------------------------------
> ------------------------------------------------------------
> ---------------------------------------------------------
>
> gluster-poc-noida    glusterdist    /data/gluster-dist/distvol
> root          gluster-poc-sj::glusterdist    gluster-poc-sj     Active
> Changelog Crawl    2018-08-31 10:28:19
>
> noi-poc-gluster      glusterdist    /data/gluster-dist/distvol
> root          gluster-poc-sj::glusterdist    gluster-poc-sj2    Active
> History Crawl      N/A
>
> [root at gluster-poc-noida ~]#
>
>
>
> Not at client I copied a 848MB file from local disk to master mounted
> volume and it took only 1 minute and 15 seconds. Its great….
>
>
>
> But even after waited for 2 hrs I was unable to see that file at slave
> site. Then I again erased the indexing by doing “gluster volume set
> glusterdist  indexing off” and restart the session. Magically I received
> the file instantly at slave after doing this.
>
>
>
> Why I need to do “indexing off” every time to reflect data at slave site?
> Is there any fix/workaround of it?
>
>
>
> /Krishna
>
>
>
>
>
> *From:* Kotresh Hiremath Ravishankar <khiremat at redhat.com>
> *Sent:* Friday, August 31, 2018 10:10 AM
> *To:* Krishna Verma <kverma at cadence.com>
> *Cc:* Sunny Kumar <sunkumar at redhat.com>; Gluster Users <
> gluster-users at gluster.org>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
>
>
>
>
> On Thu, Aug 30, 2018 at 3:51 PM, Krishna Verma <kverma at cadence.com> wrote:
>
> Hi Kotresh,
>
>
>
> Yes, this include the time take  to write 1GB file to master. geo-rep was
> not stopped while the data was copying to master.
>
>
>
> This way, you can't really measure how much time geo-rep took.
>
>
>
>
>
> But now I am trouble, My putty session was timed out while copying data to
> master and geo replication was active. After I restart putty session My
> Master data is not syncing with slave. Its Last_synced time is  1hrs behind
> the current time.
>
>
>
> I restart the geo rep and also delete and again create the session but its
>  “LAST_SYNCED” time is same.
>
>
>
> Unless, geo-rep is Faulty, it would be processing/syncing. You should
> check logs for any errors.
>
>
>
>
>
> Please help in this.
>
>
>
> …. It's better if gluster volume has more distribute count like  3*3 or
> 4*3 :- Are you refereeing to create a distributed volume with 3 master
> node and 3 slave node?
>
>
>
> Yes,  that's correct. Please do the test with this. I recommend you to run
> the actual workload for which you are planning to use gluster instead of
> copying 1GB file and testing.
>
>
>
>
>
>
>
> /krishna
>
>
>
> *From:* Kotresh Hiremath Ravishankar <khiremat at redhat.com>
> *Sent:* Thursday, August 30, 2018 3:20 PM
>
>
> *To:* Krishna Verma <kverma at cadence.com>
> *Cc:* Sunny Kumar <sunkumar at redhat.com>; Gluster Users <
> gluster-users at gluster.org>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
>
>
>
>
> On Thu, Aug 30, 2018 at 1:52 PM, Krishna Verma <kverma at cadence.com> wrote:
>
> Hi Kotresh,
>
>
>
> After fix the library link on node "noi-poc-gluster ", the status of one
> mater node is “Active” and another is “Passive”. Can I setup both the
> master as “Active” ?
>
>
>
> Nope, since it's replica, it's redundant to sync same files from two
> nodes. Both replicas can't be Active.
>
>
>
>
>
> Also, when I copy a 1GB size of file from gluster client to master gluster
> volume which is replicated with the slave volume, it tooks 35 minutes and
> 49 seconds. Is there any way to reduce its time taken to rsync data.
>
>
>
> How did you measure this time? Does this include the time take for you to
> write 1GB file to master?
>
> There are two aspects to consider while measuring this.
>
>
>
> 1. Time to write 1GB to master
>
> 2. Time for geo-rep to transfer 1GB to slave.
>
>
>
> In your case, since the setup is 1*2 and only one geo-rep worker is
> Active, Step2 above equals to time for step1 + network transfer time.
>
>
>
> You can measure time in two scenarios
>
> 1. If geo-rep is started while the data is still being written to master.
> It's one way.
>
> 2. Or stop geo-rep until the 1GB file is written to master and then start
> geo-rep to get actual geo-rep time.
>
>
>
> To improve replicating speed,
>
> 1. You can play around with rsync options depending on the kind of I/O
>
>     and configure the same for geo-rep as it also uses rsync internally.
>
> 2. It's better if gluster volume has more distribute count like  3*3 or 4*3
>
>     It will help in two ways.
>
>        1. The files gets distributed on master to multiple bricks
>
>        2. So above will help geo-rep as files on multiple bricks are
> synced in parallel (multiple Actives)
>
>
>
> NOTE: Gluster master server and one client is in Noida, India Location.
>
>              Gluster Slave server and one client is in USA.
>
>
>
> Our approach is to transfer data from Noida gluster client will reach to
> the USA gluster client in a minimum time. Please suggest the best approach
> to achieve it.
>
>
>
> [root at noi-dcops ~]# date ; rsync -avh --progress /tmp/gentoo_root.img
> /glusterfs/ ; date
>
> Thu Aug 30 12:26:26 IST 2018
>
> sending incremental file list
>
> gentoo_root.img
>
>           1.07G 100%  490.70kB/s    0:35:36 (xfr#1, to-chk=0/1)
>
>
>
> Is this I/O time to write to master volume?
>
>
>
> sent 1.07G bytes  received 35 bytes  499.65K bytes/sec
>
> total size is 1.07G  speedup is 1.00
>
> Thu Aug 30 13:02:15 IST 2018
>
> [root at noi-dcops ~]#
>
>
>
>
>
>
>
> [root at gluster-poc-noida gluster]#  gluster volume geo-replication status
>
>
>
> MASTER NODE          MASTER VOL    MASTER BRICK         SLAVE USER
> SLAVE                              SLAVE NODE        STATUS     CRAWL
> STATUS       LAST_SYNCED
>
> ------------------------------------------------------------
> ------------------------------------------------------------
> ---------------------------------------------------
>
> gluster-poc-noida    glusterep     /data/gluster/gv0    root
> ssh://gluster-poc-sj::glusterep    gluster-poc-sj    Active     Changelog
> Crawl    2018-08-30 13:42:18
>
> noi-poc-gluster      glusterep     /data/gluster/gv0    root
> ssh://gluster-poc-sj::glusterep    gluster-poc-sj    Passive
> N/A                N/A
>
> [root at gluster-poc-noida gluster]#
>
>
>
> Thanks in advance for your all time support.
>
>
>
> /Krishna
>
>
>
> *From:* Kotresh Hiremath Ravishankar <khiremat at redhat.com>
> *Sent:* Thursday, August 30, 2018 10:51 AM
>
>
> *To:* Krishna Verma <kverma at cadence.com>
> *Cc:* Sunny Kumar <sunkumar at redhat.com>; Gluster Users <
> gluster-users at gluster.org>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
> Did you fix the library link on node "noi-poc-gluster " as well?
>
> If not please fix it. Please share the geo-rep log this node if it's
>
> as different issue.
>
> -Kotresh HR
>
>
>
> On Thu, Aug 30, 2018 at 12:17 AM, Krishna Verma <kverma at cadence.com>
> wrote:
>
> Hi Kotresh,
>
>
>
> Thank you so much for you input. Geo-replication is now showing “Active”
> atleast for 1 master node. But its still at faulty state for the 2nd
> master server.
>
>
>
> Below is the detail.
>
>
>
> [root at gluster-poc-noida glusterfs]# gluster volume geo-replication
> glusterep gluster-poc-sj::glusterep status
>
>
>
> MASTER NODE          MASTER VOL    MASTER BRICK         SLAVE USER
> SLAVE                        SLAVE NODE        STATUS    CRAWL STATUS
> LAST_SYNCED
>
> ------------------------------------------------------------
> ------------------------------------------------------------
> --------------------------------------------
>
> gluster-poc-noida    glusterep     /data/gluster/gv0    root
> gluster-poc-sj::glusterep    gluster-poc-sj    Active    Changelog Crawl
> 2018-08-29 23:56:06
>
> noi-poc-gluster      glusterep     /data/gluster/gv0    root
> gluster-poc-sj::glusterep    N/A               Faulty    N/A
> N/A
>
>
>
>
>
> [root at gluster-poc-noida glusterfs]# gluster volume status
>
> Status of volume: glusterep
>
> Gluster process                             TCP Port  RDMA Port  Online
> Pid
>
> ------------------------------------------------------------
> ------------------
>
> Brick gluster-poc-noida:/data/gluster/gv0   49152     0          Y
> 22463
>
> Brick noi-poc-gluster:/data/gluster/gv0     49152     0          Y
> 19471
>
> Self-heal Daemon on localhost               N/A       N/A        Y
> 32087
>
> Self-heal Daemon on noi-poc-gluster         N/A       N/A        Y
> 6272
>
>
>
> Task Status of Volume glusterep
>
> ------------------------------------------------------------
> ------------------
>
> There are no active volume tasks
>
>
>
>
>
>
>
> [root at gluster-poc-noida glusterfs]# gluster volume info
>
>
>
> Volume Name: glusterep
>
> Type: Replicate
>
> Volume ID: 4a71bc94-14ce-4b2c-abc4-e6a9a9765161
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 1 x 2 = 2
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: gluster-poc-noida:/data/gluster/gv0
>
> Brick2: noi-poc-gluster:/data/gluster/gv0
>
> Options Reconfigured:
>
> transport.address-family: inet
>
> nfs.disable: on
>
> performance.client-io-threads: off
>
> geo-replication.indexing: on
>
> geo-replication.ignore-pid-check: on
>
> changelog.changelog: on
>
> [root at gluster-poc-noida glusterfs]#
>
>
>
> Could you please help me in that also please?
>
>
>
> It would be really a great help from your side.
>
>
>
> /Krishna
>
> *From:* Kotresh Hiremath Ravishankar <khiremat at redhat.com>
> *Sent:* Wednesday, August 29, 2018 10:47 AM
>
>
> *To:* Krishna Verma <kverma at cadence.com>
> *Cc:* Sunny Kumar <sunkumar at redhat.com>; Gluster Users <
> gluster-users at gluster.org>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
> Answer inline
>
>
>
> On Tue, Aug 28, 2018 at 4:28 PM, Krishna Verma <kverma at cadence.com> wrote:
>
> Hi Kotresh,
>
>
>
> I created the links before. Below is the detail.
>
>
>
> [root at gluster-poc-noida ~]# ls -l /usr/lib64 | grep libgfch
>
> lrwxrwxrwx   1 root root          30 Aug 28 14:59 libgfchangelog.so ->
> /usr/lib64/libgfchangelog.so.1
>
>
>
> The link created is pointing to wrong library. Please fix this
>
>
>
> #cd /usr/lib64
>
> #rm libgfchangelog.so
>
> #ln -s "libgfchangelog.so.0.0.1" libgfchangelog.so
>
>
>
> lrwxrwxrwx   1 root root          23 Aug 23 23:35 libgfchangelog.so.0 ->
> libgfchangelog.so.0.0.1
>
> -rwxr-xr-x   1 root root       63384 Jul 24 19:11 libgfchangelog.so.0.0.1
>
> [root at gluster-poc-noida ~]# locate libgfchangelog.so
>
> /usr/lib64/libgfchangelog.so.0
>
> /usr/lib64/libgfchangelog.so.0.0.1
>
> [root at gluster-poc-noida ~]#
>
>
>
> Is it looks good what we exactly need or di I need to create any more link
> or How to get “libgfchangelog.so” file if missing.
>
>
>
> /Krishna
>
>
>
> *From:* Kotresh Hiremath Ravishankar <khiremat at redhat.com>
> *Sent:* Tuesday, August 28, 2018 4:22 PM
> *To:* Krishna Verma <kverma at cadence.com>
> *Cc:* Sunny Kumar <sunkumar at redhat.com>; Gluster Users <
> gluster-users at gluster.org>
>
>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
> Hi Krishna,
>
> As per the output shared, I don't see the file "libgfchangelog.so" which
> is what is required.
>
> I only see "libgfchangelog.so.0". Please confirm "libgfchangelog.so" is
> present in "/usr/lib64/".
>
> If not create a symlink similar to "libgfchangelog.so.0"
>
>
>
> It should be something like below.
>
>
>
> #ls -l /usr/lib64 | grep libgfch
> -rwxr-xr-x. 1 root root    1078 Aug 28 05:56 libgfchangelog.la
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__libgfchangelog.la&d=DwMFaQ&c=aUq983L2pue2FqKFoP6PGHMJQyoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=77GIqpHy9HY8RQd6lKzSJ-Z1PCuIhZJ3I3IvIuDX-xo&s=kIFnrBaSFV_DdqZezd6PXcDnD8Iy_gVN69ETZYtykEE&e=>
> lrwxrwxrwx. 1 root root      23 Aug 28 05:56 libgfchangelog.so ->
> libgfchangelog.so.0.0.1
> lrwxrwxrwx. 1 root root      23 Aug 28 05:56 libgfchangelog.so.0 ->
> libgfchangelog.so.0.0.1
> -rwxr-xr-x. 1 root root  336888 Aug 28 05:56 libgfchangelog.so.0.0.1
>
>
>
> On Tue, Aug 28, 2018 at 4:04 PM, Krishna Verma <kverma at cadence.com> wrote:
>
> Hi Kotresh,
>
>
>
> Thanks for the response, I did that also but nothing changed.
>
>
>
> [root at gluster-poc-noida ~]# ldconfig /usr/lib64
>
> [root at gluster-poc-noida ~]# ldconfig -p | grep libgfchangelog
>
>         libgfchangelog.so.0 (libc6,x86-64) =>
> /usr/lib64/libgfchangelog.so.0
>
> [root at gluster-poc-noida ~]#
>
>
>
> [root at gluster-poc-noida ~]# gluster volume geo-replication glusterep
> gluster-poc-sj::glusterep stop
>
> Stopping geo-replication session between glusterep &
> gluster-poc-sj::glusterep has been successful
>
> [root at gluster-poc-noida ~]# gluster volume geo-replication glusterep
> gluster-poc-sj::glusterep start
>
> Starting geo-replication session between glusterep &
> gluster-poc-sj::glusterep has been successful
>
>
>
> [root at gluster-poc-noida ~]# gluster volume geo-replication glusterep
> gluster-poc-sj::glusterep status
>
>
>
> MASTER NODE          MASTER VOL    MASTER BRICK         SLAVE USER
> SLAVE                        SLAVE NODE    STATUS    CRAWL STATUS
> LAST_SYNCED
>
> ------------------------------------------------------------
> ------------------------------------------------------------
> -----------------------------
>
> gluster-poc-noida    glusterep     /data/gluster/gv0    root
> gluster-poc-sj::glusterep    N/A           Faulty    N/A             N/A
>
> noi-poc-gluster      glusterep     /data/gluster/gv0    root
>        gluster-poc-sj::glusterep    N/A           Faulty
> N/A             N/A
>
> [root at gluster-poc-noida ~]#
>
>
>
> /Krishna
>
>
>
> *From:* Kotresh Hiremath Ravishankar <khiremat at redhat.com>
> *Sent:* Tuesday, August 28, 2018 4:00 PM
> *To:* Sunny Kumar <sunkumar at redhat.com>
> *Cc:* Krishna Verma <kverma at cadence.com>; Gluster Users <
> gluster-users at gluster.org>
>
>
> *Subject:* Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
>
>
>
> EXTERNAL MAIL
>
> Hi Krishna,
>
> Since your libraries are in /usr/lib64, you should be doing
>
> #ldconfig /usr/lib64
>
> Confirm that below command lists the library
>
> #ldconfig -p | grep libgfchangelog
>
>
>
>
>
> On Tue, Aug 28, 2018 at 3:52 PM, Sunny Kumar <sunkumar at redhat.com> wrote:
>
> can you do ldconfig /usr/local/lib and share the output of ldconfig -p
> /usr/local/lib | grep libgf
>
> On Tue, Aug 28, 2018 at 3:45 PM Krishna Verma <kverma at cadence.com> wrote:
> >
> > Hi Sunny,
> >
> > I did the mentioned changes given in patch and restart the session for
> geo-replication. But again same errors in the logs.
> >
> > I have attaching the config files and logs here.
> >
> >
> > [root at gluster-poc-noida ~]# gluster volume geo-replication glusterep
> gluster-poc-sj::glusterep stop
> > Stopping geo-replication session between glusterep &
> gluster-poc-sj::glusterep has been successful
> > [root at gluster-poc-noida ~]# gluster volume geo-replication glusterep
> gluster-poc-sj::glusterep delete
> > Deleting geo-replication session between glusterep &
> gluster-poc-sj::glusterep has been successful
> > [root at gluster-poc-noida ~]# gluster volume geo-replication glusterep
> gluster-poc-sj::glusterep create push-pem force
> > Creating geo-replication session between glusterep &
> gluster-poc-sj::glusterep has been successful
> > [root at gluster-poc-noida ~]# gluster volume geo-replication glusterep
> gluster-poc-sj::glusterep start
> > geo-replication start failed for glusterep gluster-poc-sj::glusterep
> > geo-replication command failed
> > [root at gluster-poc-noida ~]# gluster volume geo-replication glusterep
> gluster-poc-sj::glusterep start
> > geo-replication start failed for glusterep gluster-poc-sj::glusterep
> > geo-replication command failed
> > [root at gluster-poc-noida ~]# vim /usr/libexec/glusterfs/python/
> syncdaemon/repce.py
> > [root at gluster-poc-noida ~]# systemctl restart glusterd
> > [root at gluster-poc-noida ~]# gluster volume geo-replication glusterep
> gluster-poc-sj::glusterep start
> > Starting geo-replication session between glusterep &
> gluster-poc-sj::glusterep has been successful
> > [root at gluster-poc-noida ~]# gluster volume geo-replication glusterep
> gluster-poc-sj::glusterep status
> >
> > MASTER NODE          MASTER VOL    MASTER BRICK         SLAVE USER
> SLAVE                        SLAVE NODE    STATUS    CRAWL STATUS
> LAST_SYNCED
> > ------------------------------------------------------------
> ------------------------------------------------------------
> -----------------------------
> > gluster-poc-noida    glusterep     /data/gluster/gv0    root
> gluster-poc-sj::glusterep    N/A           Faulty    N/A             N/A
> > noi-poc-gluster      glusterep     /data/gluster/gv0    root
> gluster-poc-sj::glusterep    N/A           Faulty    N/A             N/A
> > [root at gluster-poc-noida ~]#
> >
> >
> > /Krishna.
> >
> > -----Original Message-----
> > From: Sunny Kumar <sunkumar at redhat.com>
> > Sent: Tuesday, August 28, 2018 3:17 PM
> > To: Krishna Verma <kverma at cadence.com>
> > Cc: gluster-users at gluster.org
> > Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> work
> >
> > EXTERNAL MAIL
> >
> >
> > With same log message ?
> >
> > Can you please verify that
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__review.
> gluster.org_-23_c_glusterfs_-2B_20207_&d=DwIBaQ&c=
> aUq983L2pue2FqKFoP6PGHMJQyoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_
> 6ZItAWQ2jH8rVLG6tiXhoLFE&m=F0ExtFUfa_YCktOGvy82x3IAxvi2GrbPR72jZ8beuYk&s=
> fGtkmezHJj5YoLN3dUeVUCcYFnREHyOSk36mRjbTTEQ&e= patch is present if not
> can you please apply that.
> > and try with symlinking ln -s /usr/lib64/libgfchangelog.so.0
> /usr/lib64/libgfchangelog.so.
> >
> > Please share the log also.
> >
> > Regards,
> > Sunny
> > On Tue, Aug 28, 2018 at 3:02 PM Krishna Verma <kverma at cadence.com>
> wrote:
> > >
> > > Hi Sunny,
> > >
> > > Thanks for your response, I tried both, but still I am getting the
> same error.
> > >
> > >
> > > [root at noi-poc-gluster ~]# ldconfig /usr/lib [root at noi-poc-gluster ~]#
> > >
> > > [root at noi-poc-gluster ~]# ln -s /usr/lib64/libgfchangelog.so.1
> > > /usr/lib64/libgfchangelog.so [root at noi-poc-gluster ~]# ls -l
> > > /usr/lib64/libgfchangelog.so lrwxrwxrwx. 1 root root 30 Aug 28 14:59
> > > /usr/lib64/libgfchangelog.so -> /usr/lib64/libgfchangelog.so.1
> > >
> > > /Krishna
> > >
> > > -----Original Message-----
> > > From: Sunny Kumar <sunkumar at redhat.com>
> > > Sent: Tuesday, August 28, 2018 2:55 PM
> > > To: Krishna Verma <kverma at cadence.com>
> > > Cc: gluster-users at gluster.org
> > > Subject: Re: [Gluster-users] Upgrade to 4.1.2 geo-replication does not
> > > work
> > >
> > > EXTERNAL MAIL
> > >
> > >
> > > Hi Krish,
> > >
> > > You can run -
> > > #ldconfig /usr/lib
> > >
> > > If that still does not solves your problem you can do manual symlink
> > > like - ln -s /usr/lib64/libgfchangelog.so.1
> > > /usr/lib64/libgfchangelog.so
> > >
> > > Thanks,
> > > Sunny Kumar
> > > On Tue, Aug 28, 2018 at 1:47 PM Krishna Verma <kverma at cadence.com>
> wrote:
> > > >
> > > > Hi
> > > >
> > > >
> > > >
> > > > I am getting below error in gsyncd.log
> > > >
> > > >
> > > >
> > > > OSError: libgfchangelog.so: cannot open shared object file: No such
> > > > file or directory
> > > >
> > > > [2018-08-28 07:19:41.446785] E [repce(worker
> /data/gluster/gv0):197:__call__] RepceClient: call failed
>  call=26469:139794524604224:1535440781.44        method=init
>  error=OSError
> > > >
> > > > [2018-08-28 07:19:41.447041] E [syncdutils(worker
> /data/gluster/gv0):330:log_raise_exception] <top>: FAIL:
> > > >
> > > > Traceback (most recent call last):
> > > >
> > > >   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line
> > > > 311, in main
> > > >
> > > >     func(args)
> > > >
> > > >   File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line
> > > > 72, in subcmd_worker
> > > >
> > > >     local.service_loop(remote)
> > > >
> > > >   File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line
> > > > 1236, in service_loop
> > > >
> > > >     changelog_agent.init()
> > > >
> > > >   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line
> > > > 216, in __call__
> > > >
> > > >     return self.ins(self.meth, *a)
> > > >
> > > >   File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line
> > > > 198, in __call__
> > > >
> > > >     raise res
> > > >
> > > > OSError: libgfchangelog.so: cannot open shared object file: No such
> > > > file or directory
> > > >
> > > > [2018-08-28 07:19:41.457555] I [repce(agent
> /data/gluster/gv0):80:service_loop] RepceServer: terminating on reaching
> EOF.
> > > >
> > > > [2018-08-28 07:19:42.440184] I [monitor(monitor):272:monitor]
> Monitor:
> > > > worker died in startup phase  brick=/data/gluster/gv0
> > > >
> > > >
> > > >
> > > > Below is my file location:
> > > >
> > > >
> > > >
> > > > /usr/lib64/libgfchangelog.so.0
> > > >
> > > > /usr/lib64/libgfchangelog.so.0.0.1
> > > >
> > > >
> > > >
> > > > What I can do to fix it ?
> > > >
> > > >
> > > >
> > > > /Krish
> > > >
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gluster.o
> > > > rg
> > > > _mailman_listinfo_gluster-2Dusers&d=DwIBaQ&c=aUq983L2pue2FqKFoP6PGHM
> > > > JQ
> > > > yoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=_
> > > > u6
> > > > vGRjlVsype7Z8hXDgCONilqVe4sIWkXNqqz2n3IQ&s=i0EUwtUHurhJHyw9UPpepCdLB
> > > > 70
> > > > 1mkxoNZWYvU7XXug&e=
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
>
> https://lists.gluster.org/mailman/listinfo/gluster-users
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gluster.org_mailman_listinfo_gluster-2Dusers&d=DwMFaQ&c=aUq983L2pue2FqKFoP6PGHMJQyoJ7kl3s3GZ-_haXqY&r=0E5nRoxLsT2ZXgCpJM_6ZItAWQ2jH8rVLG6tiXhoLFE&m=BuUVeXUwqxdQDoP-IWDI_JPFCQGrRFdzV9g0enLn8kM&s=X0eVYgZ1emUQnEwRTg0AGbAWyoIwIyrE_gzmuolgiPE&e=>
>
>
>
>
> --
>
> Thanks and Regards,
>
> Kotresh H R
>
>
>
>
> --
>
> Thanks and Regards,
>
> Kotresh H R
>
>
>
>
> --
>
> Thanks and Regards,
>
> Kotresh H R
>
>
>
>
> --
>
> Thanks and Regards,
>
> Kotresh H R
>
>
>
>
> --
>
> Thanks and Regards,
>
> Kotresh H R
>
>
>
>
> --
>
> Thanks and Regards,
>
> Kotresh H R
>
>
>
>
> --
>
> Thanks and Regards,
>
> Kotresh H R
>
>
>
>
> --
>
> Thanks and Regards,
>
> Kotresh H R
>



-- 
Thanks and Regards,
Kotresh H R
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180903/d764d659/attachment.html>


More information about the Gluster-users mailing list