[Gluster-users] Geo-Replication just won't go past 1TB - half the replication process crashing?
Venky Shankar
yknev.shankar at gmail.com
Wed Sep 11 05:13:37 UTC 2013
So the slave is slowly catching up. But I agree, it's too slow. The
"slowness" you mention could be due to an inherent delay issue with
Geo-replication. After geo-replication identifies a set of files that needs
to be synced, it invokes rsync to transfer data. After a successful rsync,
it sets extended attributes on "every" file/directory on the slave to mark
it as synced. This does take up a significant amount of time and
contributes to the "stalls" that you mention (even in our tests this has
been a pain point).
There is a workaround for the above mentioned issue, which is to use rsync
to sync extended attributes, but I think that's not available in the
version you're using.
Now, regarding manually rsync'ing the initial data set: Geo-replication
replies on a extended attribute (xtime) comparison between the master and
the slave to identify files that needs to be synced. Rsync without '-X'
would not sync extended attributes, which implies that after
geo-replication is started, crawling would still happen for the entire tree
(data sync may not happen as rsync would identify no data differences).
Now, in case you get tempted to use rsync (with -X), could you do the
following:
# mkdir /tmp/1
# glusterfs --volfile-id=<volume> --volfile-server=<server> --client-pid=-1
/tmp/1/
# getfattr -m . -d /tmp/1
# umount /tmp/1
The third command would tell what all extended attributes would rsync try
to sync in case '-X' is used.
Thanks,
-venky
On Tue, Sep 10, 2013 at 9:50 PM, Tony Maro <tonym at evrichart.com> wrote:
> Venky,
>
> Just touching back with you to see where else I might want to look on
> this. The primary brick running the replication is still using a good deal
> of CPU though not quite what it was when the geo-rep was first started:
>
> [image: Inline image 2]
>
> You can see just above Week 34 when I started the geo-rep process. The
> major drop just before the words "Week 36" is when it stopped sending the
> initial data to the target.
>
> The replication target machine continues to slowly - very slowly - gain
> new data. Unfortunately I think it's only data that's been added recently.
> It doesn't appear to still be replicating the initial batch. That seems
> to have stopped right about 1TB total. I'm currently at 1.2 TB replicated
> and it's been over a week now since I eclipsed 1 TB.
>
> Would I do better to stop the process, wipe the geo-rep information from
> Gluster, do an rsync of the files to the destination and then start the
> geo-rep process again?
>
>
> On Fri, Sep 6, 2013 at 11:40 AM, Tony Maro <tonym at evrichart.com> wrote:
>
>> Correction: Different between the two local bricks is more likely a
>> result of the ZFS snapshot process I've run on the larger of them.
>>
>>
>> On Fri, Sep 6, 2013 at 11:39 AM, Tony Maro <tonym at evrichart.com> wrote:
>>
>>> More stats in case you need them:
>>>
>>> Grand total of the /data/docstore1 tree that's being sync'd is about 8.5
>>> million files.
>>> About 1 KB of new data has appeared on the destination server since we
>>> started this discussion yesterday...
>>>
>>>
>>> tony at backup-ds2:~$ df
>>> Filesystem 1K-blocks Used Available Use% Mounted on
>>> /dev/sda1 768989240 1841364 728085452 1% /
>>> udev 16450080 4 16450076 1% /dev
>>> tmpfs 6583556 348 6583208 1% /run
>>> none 5120 0 5120 0% /run/lock
>>> none 16458888 0 16458888 0% /run/shm
>>> data 6601579264 0 6601579264 0% /data
>>> data/docstore1 7651735936 1050156672 6601579264 14% /data/docstore1
>>> tony at backup-ds2:~$ date
>>> Thu Sep 5 09:36:24 EDT 2013
>>>
>>> tony at backup-ds2:~$ df
>>> Filesystem 1K-blocks Used Available Use% Mounted on
>>> /dev/sda1 768989240 1842524 728084292 1% /
>>> udev 16450080 4 16450076 1% /dev
>>> tmpfs 6583556 348 6583208 1% /run
>>> none 5120 0 5120 0% /run/lock
>>> none 16458888 0 16458888 0% /run/shm
>>> data 6601577984 0 6601577984 0% /data
>>> data/docstore1 7651735808 1050157824 6601577984 14% /data/docstore1
>>> tony at backup-ds2:~$ date
>>> Fri Sep 6 11:34:07 EDT 2013
>>> tony at backup-ds2:~$
>>>
>>> The source data I'm attempting to geo-replicate is 3.3 TB. Mirroring
>>> between the local bricks seems to be working fine. They are within 20KB of
>>> drive usage between each other for the /data/docstore1 partition. I
>>> attribute that to the new geo-rep info on gfs6 making it larger by a bit?
>>> Also the data is constantly changing, so maybe it's a bit of replication
>>> lag.
>>>
>>>
>>>
>>> On Fri, Sep 6, 2013 at 11:29 AM, Tony Maro <tonym at evrichart.com> wrote:
>>>
>>>> ./evds3 contains 1,262,970 files. No more than 255 files or folders
>>>> per subdirectory.
>>>>
>>>> Network latency is almost nonexistent - the servers are currently
>>>> jacked into the same switch on a separate network segment for the initial
>>>> sync:
>>>>
>>>> tony at gfs6:~$ ping backup-ds2.gluster
>>>> PING backup-ds2.gluster (10.200.1.12) 56(84) bytes of data.
>>>> 64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=1 ttl=64
>>>> time=0.245 ms
>>>> 64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=2 ttl=64
>>>> time=0.197 ms
>>>> 64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=3 ttl=64
>>>> time=0.174 ms
>>>> 64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=4 ttl=64
>>>> time=0.221 ms
>>>> 64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=5 ttl=64
>>>> time=0.163 ms
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Sep 6, 2013 at 5:13 AM, Venky Shankar <yknev.shankar at gmail.com>wrote:
>>>>
>>>>> From the trace logs it looks like its still crawling (the lgetxattr()
>>>>> on the master and lgetxattr() on the slave). How many files are under './evds3'
>>>>> ?
>>>>>
>>>>> Further more, what's the latency b/w the two sites?
>>>>>
>>>>>
>>>>> On Thu, Sep 5, 2013 at 11:31 PM, Tony Maro <tonym at evrichart.com>wrote:
>>>>>
>>>>>> No rsync process is running. I tested for it several times over 10
>>>>>> seconds.
>>>>>>
>>>>>> Here's the trace on feedback. I put it on TinyPaste instead of in
>>>>>> everyone's inbox out of courtesy: http://tny.cz/505b09c8
>>>>>>
>>>>>> Thanks,
>>>>>> Tony
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 5, 2013 at 12:24 PM, Venky Shankar <
>>>>>> yknev.shankar at gmail.com> wrote:
>>>>>>
>>>>>>> So, nothing fishy in slave logs too.
>>>>>>>
>>>>>>> Debugging this would need more information. Lets start with:
>>>>>>>
>>>>>>> 1. Is there any rsync process that is running? (ps auxww | grep
>>>>>>> rsync)
>>>>>>>
>>>>>>> 2. I would need strace logs for the worker process (the process
>>>>>>> which scans the filesystem and invokes rsync to transfer data). Something
>>>>>>> like this would do:
>>>>>>>
>>>>>>> - get the pid of the worker process
>>>>>>> # ps auxww | grep feedback (a gsyncd.py python process)
>>>>>>>
>>>>>>> - trace the pid above
>>>>>>> # strace -s 500 -o /tmp/worker.log -f -p <pid>
>>>>>>>
>>>>>>> Let the trace run for about 20 seconds. Lets see what's in
>>>>>>> /tmp/worker.log
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -venky
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Sep 5, 2013 at 7:56 PM, Tony Maro <tonym at evrichart.com>wrote:
>>>>>>>
>>>>>>>> Looks like the slave stopped logging things just before the sending
>>>>>>>> brick did:
>>>>>>>>
>>>>>>>> [2013-09-01 14:56:45.13972] I [gsyncd(slave):354:main_i] <top>:
>>>>>>>> syncing: file:///data/docstore1
>>>>>>>> [2013-09-01 14:56:45.15433] I [resource(slave):453:service_loop]
>>>>>>>> FILE: slave listening
>>>>>>>> [2013-09-01 15:57:34.592938] I [repce(slave):78:service_loop]
>>>>>>>> RepceServer: terminating on reaching EOF.
>>>>>>>> [2013-09-01 15:57:34.593383] I [syncdutils(slave):142:finalize]
>>>>>>>> <top>: exiting.
>>>>>>>> [2013-09-01 15:57:45.374301] I [gsyncd(slave):354:main_i] <top>:
>>>>>>>> syncing: file:///data/docstore1
>>>>>>>> [2013-09-01 15:57:45.375871] I [resource(slave):453:service_loop]
>>>>>>>> FILE: slave listening
>>>>>>>> [2013-09-01 16:02:09.115976] I [repce(slave):78:service_loop]
>>>>>>>> RepceServer: terminating on reaching EOF.
>>>>>>>> [2013-09-01 16:02:09.116446] I [syncdutils(slave):142:finalize]
>>>>>>>> <top>: exiting.
>>>>>>>> [2013-09-01 16:02:19.869340] I [gsyncd(slave):354:main_i] <top>:
>>>>>>>> syncing: file:///data/docstore1
>>>>>>>> [2013-09-01 16:02:19.870598] I [resource(slave):453:service_loop]
>>>>>>>> FILE: slave listening
>>>>>>>> [2013-09-01 16:32:58.701617] I [repce(slave):78:service_loop]
>>>>>>>> RepceServer: terminating on reaching EOF.
>>>>>>>> [2013-09-01 16:32:58.702126] I [syncdutils(slave):142:finalize]
>>>>>>>> <top>: exiting.
>>>>>>>> [2013-09-01 16:33:09.456556] I [gsyncd(slave):354:main_i] <top>:
>>>>>>>> syncing: file:///data/docstore1
>>>>>>>> [2013-09-01 16:33:09.458009] I [resource(slave):453:service_loop]
>>>>>>>> FILE: slave listening
>>>>>>>> [2013-09-01 21:47:13.442514] I [repce(slave):78:service_loop]
>>>>>>>> RepceServer: terminating on reaching EOF.
>>>>>>>> [2013-09-01 21:47:13.442899] I [syncdutils(slave):142:finalize]
>>>>>>>> <top>: exiting.
>>>>>>>> [2013-09-01 21:47:24.240978] I [gsyncd(slave):354:main_i] <top>:
>>>>>>>> syncing: file:///data/docstore1
>>>>>>>> [2013-09-01 21:47:24.242424] I [resource(slave):453:service_loop]
>>>>>>>> FILE: slave listening
>>>>>>>> [2013-09-02 02:12:14.26339] I [repce(slave):78:service_loop]
>>>>>>>> RepceServer: terminating on reaching EOF.
>>>>>>>> [2013-09-02 02:12:14.26809] I [syncdutils(slave):142:finalize]
>>>>>>>> <top>: exiting.
>>>>>>>> [2013-09-02 02:12:24.818355] I [gsyncd(slave):354:main_i] <top>:
>>>>>>>> syncing: file:///data/docstore1
>>>>>>>> [2013-09-02 02:12:24.820008] I [resource(slave):453:service_loop]
>>>>>>>> FILE: slave listening
>>>>>>>> [2013-09-02 02:16:14.525187] I [repce(slave):78:service_loop]
>>>>>>>> RepceServer: terminating on reaching EOF.
>>>>>>>> [2013-09-02 02:16:14.525675] I [syncdutils(slave):142:finalize]
>>>>>>>> <top>: exiting.
>>>>>>>> [2013-09-02 02:16:25.263712] I [gsyncd(slave):354:main_i] <top>:
>>>>>>>> syncing: file:///data/docstore1
>>>>>>>> [2013-09-02 02:16:25.265168] I [resource(slave):453:service_loop]
>>>>>>>> FILE: slave listening
>>>>>>>> [2013-09-02 02:37:39.315608] I [repce(slave):78:service_loop]
>>>>>>>> RepceServer: terminating on reaching EOF.
>>>>>>>> [2013-09-02 02:37:39.316071] I [syncdutils(slave):142:finalize]
>>>>>>>> <top>: exiting.
>>>>>>>> [2013-09-02 02:37:50.78136] I [gsyncd(slave):354:main_i] <top>:
>>>>>>>> syncing: file:///data/docstore1
>>>>>>>> [2013-09-02 02:37:50.79577] I [resource(slave):453:service_loop]
>>>>>>>> FILE: slave listening
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 5, 2013 at 10:09 AM, Venky Shankar <
>>>>>>>> yknev.shankar at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Could you also provide the slave logs? (log location on the
>>>>>>>>> slave: /var/log/glusterfs/geo-replication-slaves)
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> -venky
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Sep 5, 2013 at 7:29 PM, Tony Maro <tonym at evrichart.com>wrote:
>>>>>>>>>
>>>>>>>>>> I'm trying to create a new Geo-Rep of about 3 TB of data
>>>>>>>>>> currently stored in a 2 brick mirror config. Obviously the geo-rep
>>>>>>>>>> destination is a third server.
>>>>>>>>>>
>>>>>>>>>> This is my 150th attempt. Okay, maybe not that far, but it's
>>>>>>>>>> pretty darn bad.
>>>>>>>>>>
>>>>>>>>>> Replication works fine until I hit around 1TB of data sync'd,
>>>>>>>>>> then it just stalls. For the past two days it hasn't gone past 1050156672
>>>>>>>>>> bytes sync'd to the destination server.
>>>>>>>>>>
>>>>>>>>>> I did some digging in the logs and it looks like the brick that's
>>>>>>>>>> running the geo-rep process thinks it's syncing:
>>>>>>>>>>
>>>>>>>>>> [2013-09-05 09:45:37.354831] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/00000863.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.358669] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/0000083b.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.362251] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/0000087b.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.366027] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/00000834.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.369752] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/00000845.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.373528] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/00000864.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.377037] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/0000087f.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.391432] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/00000897.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.395059] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/00000829.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.398725] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/00000816.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.402559] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/000008cc.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.406450] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/000008d2.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.410310] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/000008df.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.414344] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/00/00/08/000008bd.enc ...
>>>>>>>>>> [2013-09-05 09:45:37.438173] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/volume.info ...
>>>>>>>>>> [2013-09-05 09:45:37.441675] D [master:386:crawl] GMaster:
>>>>>>>>>> syncing ./evds3/Sky_Main_66/volume.enc ...
>>>>>>>>>>
>>>>>>>>>> But, *those files never appear on the destination server,*however the containing folders are there, just empty.
>>>>>>>>>>
>>>>>>>>>> Also, the other log file (...gluster.log) in the geo-replication
>>>>>>>>>> log folder that matches the destination stopped updating when the syncing
>>>>>>>>>> stopped apparently. It's last timestamp is from the 2nd, which is the last
>>>>>>>>>> time data transferred.
>>>>>>>>>>
>>>>>>>>>> The last bit from that log file is as such:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> +------------------------------------------------------------------------------+
>>>>>>>>>> [2013-09-02 06:37:50.109730] I
>>>>>>>>>> [rpc-clnt.c:1654:rpc_clnt_reconfig] 0-docstore1-client-1: changing port to
>>>>>>>>>> 24009 (from 0)
>>>>>>>>>> [2013-09-02 06:37:50.109857] I
>>>>>>>>>> [rpc-clnt.c:1654:rpc_clnt_reconfig] 0-docstore1-client-0: changing port to
>>>>>>>>>> 24009 (from 0)
>>>>>>>>>> [2013-09-02 06:37:54.097468] I
>>>>>>>>>> [client-handshake.c:1614:select_server_supported_programs]
>>>>>>>>>> 0-docstore1-client-1: Using Program GlusterFS 3.3.2, Num (1298437), Version
>>>>>>>>>> (330)
>>>>>>>>>> [2013-09-02 06:37:54.097973] I
>>>>>>>>>> [client-handshake.c:1411:client_setvolume_cbk] 0-docstore1-client-1:
>>>>>>>>>> Connected to 10.200.1.6:24009, attached to remote volume
>>>>>>>>>> '/data/docstore1'.
>>>>>>>>>> [2013-09-02 06:37:54.098005] I
>>>>>>>>>> [client-handshake.c:1423:client_setvolume_cbk] 0-docstore1-client-1: Server
>>>>>>>>>> and Client lk-version numbers are not same, reopening the fds
>>>>>>>>>> [2013-09-02 06:37:54.098094] I [afr-common.c:3685:afr_notify]
>>>>>>>>>> 0-docstore1-replicate-0: Subvolume 'docstore1-client-1' came back up; going
>>>>>>>>>> online.
>>>>>>>>>> [2013-09-02 06:37:54.098274] I
>>>>>>>>>> [client-handshake.c:453:client_set_lk_version_cbk] 0-docstore1-client-1:
>>>>>>>>>> Server lk version = 1
>>>>>>>>>> [2013-09-02 06:37:54.098619] I
>>>>>>>>>> [client-handshake.c:1614:select_server_supported_programs]
>>>>>>>>>> 0-docstore1-client-0: Using Program GlusterFS 3.3.2, Num (1298437), Version
>>>>>>>>>> (330)
>>>>>>>>>> [2013-09-02 06:37:54.099191] I
>>>>>>>>>> [client-handshake.c:1411:client_setvolume_cbk] 0-docstore1-client-0:
>>>>>>>>>> Connected to 10.200.1.5:24009, attached to remote volume
>>>>>>>>>> '/data/docstore1'.
>>>>>>>>>> [2013-09-02 06:37:54.099222] I
>>>>>>>>>> [client-handshake.c:1423:client_setvolume_cbk] 0-docstore1-client-0: Server
>>>>>>>>>> and Client lk-version numbers are not same, reopening the fds
>>>>>>>>>> [2013-09-02 06:37:54.105891] I
>>>>>>>>>> [fuse-bridge.c:4191:fuse_graph_setup] 0-fuse: switched to graph 0
>>>>>>>>>> [2013-09-02 06:37:54.106039] I
>>>>>>>>>> [client-handshake.c:453:client_set_lk_version_cbk] 0-docstore1-client-0:
>>>>>>>>>> Server lk version = 1
>>>>>>>>>> [2013-09-02 06:37:54.106179] I [fuse-bridge.c:3376:fuse_init]
>>>>>>>>>> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel
>>>>>>>>>> 7.17
>>>>>>>>>> [2013-09-02 06:37:54.108766] I
>>>>>>>>>> [afr-common.c:2022:afr_set_root_inode_on_first_lookup]
>>>>>>>>>> 0-docstore1-replicate-0: added root inode
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is driving me nuts - I've been working on getting
>>>>>>>>>> Geo-Replication working for over 2 months now without any success.
>>>>>>>>>>
>>>>>>>>>> Status on the geo-rep shows OK:
>>>>>>>>>>
>>>>>>>>>> root at gfs6:~# gluster volume geo-replication docstore1
>>>>>>>>>> ssh://root@backup-ds2.gluster:/data/docstore1 status
>>>>>>>>>> MASTER SLAVE
>>>>>>>>>> STATUS
>>>>>>>>>>
>>>>>>>>>> --------------------------------------------------------------------------------
>>>>>>>>>> docstore1 ssh://root@backup-ds2.gluster:/data/docstore1
>>>>>>>>>> OK
>>>>>>>>>> root at gfs6:~#
>>>>>>>>>>
>>>>>>>>>> Here's the config:
>>>>>>>>>>
>>>>>>>>>> root at gfs6:~# gluster volume geo-replication docstore1
>>>>>>>>>> ssh://root@backup-ds2.gluster:/data/docstore1 config
>>>>>>>>>> log_level: DEBUG
>>>>>>>>>> gluster_log_file:
>>>>>>>>>> /var/log/glusterfs/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.gluster.log
>>>>>>>>>> ssh_command: ssh -oPasswordAuthentication=no
>>>>>>>>>> -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem
>>>>>>>>>> session_owner: 24f8c92d-723e-4513-9593-40ef4b7e766a
>>>>>>>>>> remote_gsyncd: /usr/lib/glusterfs/glusterfs/gsyncd
>>>>>>>>>> state_file:
>>>>>>>>>> /var/lib/glusterd/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.status
>>>>>>>>>> gluster_command_dir: /usr/sbin/
>>>>>>>>>> pid_file:
>>>>>>>>>> /var/lib/glusterd/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.pid
>>>>>>>>>> log_file:
>>>>>>>>>> /var/log/glusterfs/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.log
>>>>>>>>>> gluster_params: xlator-option=*-dht.assert-no-child-down=true
>>>>>>>>>> root at gfs6:~#
>>>>>>>>>>
>>>>>>>>>> I'm running Ubuntu packages 3.3.2-ubuntu1-precise2 from the ppa.
>>>>>>>>>> Any ideas for why it's stalling?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> *Tony Maro*
>>>>>>>> Chief Information Officer
>>>>>>>> EvriChart • www.evrichart.com
>>>>>>>> Advanced Records Management
>>>>>>>> Office | 888.801.2020 • 304.536.1290
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>>
>>>>>> *Tony Maro*
>>>>>> Chief Information Officer
>>>>>> EvriChart • www.evrichart.com
>>>>>> Advanced Records Management
>>>>>> Office | 888.801.2020 • 304.536.1290
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>>
>>>> *Tony Maro*
>>>> Chief Information Officer
>>>> EvriChart • www.evrichart.com
>>>> Advanced Records Management
>>>> Office | 888.801.2020 • 304.536.1290
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks,
>>>
>>> *Tony Maro*
>>> Chief Information Officer
>>> EvriChart • www.evrichart.com
>>> Advanced Records Management
>>> Office | 888.801.2020 • 304.536.1290
>>>
>>>
>>
>>
>> --
>> Thanks,
>>
>> *Tony Maro*
>> Chief Information Officer
>> EvriChart • www.evrichart.com
>> Advanced Records Management
>> Office | 888.801.2020 • 304.536.1290
>>
>>
>
>
> --
> Thanks,
>
> *Tony Maro*
> Chief Information Officer
> EvriChart • www.evrichart.com
> Advanced Records Management
> Office | 888.801.2020 • 304.536.1290
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130911/9adb7573/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graphmonth.gif
Type: image/gif
Size: 12318 bytes
Desc: not available
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130911/9adb7573/attachment.gif>
More information about the Gluster-users
mailing list