[Gluster-users] 3.1.2 feedback
luomails at gmail.com
luomails at gmail.com
Tue Apr 5 22:36:32 UTC 2011
Service is down when programs are running
System: Ubuntu 10.04 (Linux version 2.6.32-21-server (buildd at yellow) (gcc
version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #32-Ubuntu SMP Fri Apr 16 09:17:34 UTC
2010)
Mount type: Gluster Native Client
Our service was down when we were running several programs, one machine was
running that would generate thousands files(maybe more), connected with the
main sever(Samba) and saved data in gluster-vol. The machine would run at least
one week, couple days ago, it was down. By the way, we also opened other
sessions to do the data backup(sftp) and some computing.
When I checked the gluster-vol, I got:
$df -ah
df: `/mnt/gluster-vol': Transport endpoint is not connected
Then I checked the mount point:
$ ls -la
ls: cannot access gluster-vol: Transport endpoint is not connected
d????????? ? ? ? ? ? gluster-vol
I also checked the peers from the main server, it couldn't see the peers, when
I ssh to data storages that can see each other.
$ sudo gluster volume info all
No volumes present
$ sudo gluster peer status
No peers present
Then I checked the status
$ sudo /etc/init.d/glusterd status
* glusterd service is not running.
After I restarted the services, unmount the gluster-vol, and mount it. It
works. My questions is: Is there any restriction for the tasks loaded for
glusterfs 3.1.2? For example, how many files can be accessed at the same time
from gluster-vol? and how many programs can we running at the same time?
Please find the attached log file for your reference at that time.
Note: Someone has reported from this group "I have been testing 3.1.2 over the last few days.
My overall impression is that it resolved several bugs from 3.1.1, but the
latest version is still prone to crashing under moderate to heavy loads. "
(http://www.opensubscriber.com/message/gluster-users@gluster.org/15035860.html).
Is it same issue?
Thanks for your help. We want to figure it out and prevent it in the future.
--------Related content of log file--------
[2011-03-31 20:57:05.462233] I [afr-common.c:716:afr_lookup_done]
gluster-vol-replicate-4: background meta-data data entry self-heal triggered.
path: /home/junzhu/Projects/Mito_Xing_03_23_11/data
[2011-03-31 20:57:05.465182] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-7: background meta-data data entry self-heal completed
on /home/junzhu/Projects/Mito_Xing_03_23_11/data
[2011-03-31 20:57:05.465748] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-8: background meta-data data entry self-heal completed
on /home/junzhu/Projects/Mito_Xing_03_23_11/data
[2011-03-31 20:57:05.465788] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-6: background meta-data data entry self-heal completed
on /home/junzhu/Projects/Mito_Xing_03_23_11/data
[2011-03-31 20:57:05.466198] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-0: background meta-data data entry self-heal completed
on /home/junzhu/Projects/Mito_Xing_03_23_11/data
[2011-03-31 20:57:05.466243] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-2: background meta-data data entry self-heal completed
on /home/junzhu/Projects/Mito_Xing_03_23_11/data
[2011-03-31 20:57:05.496131] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-4: background meta-data data entry self-heal completed
on /home/junzhu/Projects/Mito_Xing_03_23_11/data
[2011-03-31 20:57:05.529025] I [afr-common.c:662:afr_lookup_done]
gluster-vol-replicate-8: entries are missing in lookup of
/home/junzhu/Projects/Mito_Xing_03_23_11/data/read1.txt.
[2011-03-31 20:57:05.529097] I [afr-common.c:716:afr_lookup_done]
gluster-vol-replicate-8: background meta-data data entry self-heal triggered.
path: /home/junzhu/Projects/Mito_Xing_03_23_11/data/read1.txt
[2011-03-31 20:57:05.535098] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-8: background meta-data data entry self-heal completed
on /home/junzhu/Projects/Mito_Xing_03_23_11/data/read1.txt
pending frames:
frame : type(1) op(READ)
frame : type(1) op(READ)
frame : type(1) op(READ)
frame : type(1) op(READ)
frame : type(1) op(READ)
frame : type(1) op(READ)
frame : type(1) op(READ)
frame : type(1) op(READ)
frame : type(1) op(CREATE)
frame : type(1) op(CREATE)
frame : type(1) op(CREATE)
frame : type(1) op(CREATE)
frame : type(1) op(CREATE)
frame : type(1) op(READ)
frame : type(1) op(READ)
patchset: v3.1.1-64-gf2a067c
signal received: 6
time of crash: 2011-03-31 22:10:01
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.2
/lib/libc.so.6(+0x33af0)[0x7ff413899af0]
/lib/libc.so.6(gsignal+0x35)[0x7ff413899a75]
/lib/libc.so.6(abort+0x180)[0x7ff41389d5c0]
/lib/libc.so.6(__assert_fail+0xf1)[0x7ff413892941]
/lib/libpthread.so.0(pthread_mutex_lock+0x7b)[0x7ff413bf243b]
/usr/lib64/glusterfs/3.1.2/xlator/performance/io-cache.so(ioc_create_cbk+0xb0)[0x7ff4101d9540]
/usr/lib64/glusterfs/3.1.2/xlator/performance/read-ahead.so(ra_create_cbk+0x1ba)[0x7ff4103e4daa]
/usr/lib64/glusterfs/3.1.2/xlator/performance/write-behind.so(wb_create_cbk+0x10b)[0x7ff4105eee0b]
/usr/lib64/glusterfs/3.1.2/xlator/cluster/distribute.so(dht_create_cbk+0x2b8)[0x7ff410816b18]
/usr/lib64/glusterfs/3.1.2/xlator/cluster/replicate.so(afr_create_unwind+0x12b)[0x7ff410a2fc9b]
/usr/lib64/glusterfs/3.1.2/xlator/cluster/replicate.so(afr_create_wind_cbk+0x128)[0x7ff410a32d78]
/usr/lib64/glusterfs/3.1.2/xlator/protocol/client.so(client3_1_create_cbk+0x92f)[0x7ff410c939cf]
/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7ff41422dc15]
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xc9)[0x7ff41422de69]
/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7ff41422902d]
/usr/lib64/glusterfs/3.1.2/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7ff411abd344]
/usr/lib64/glusterfs/3.1.2/rpc-transport/socket.so(socket_event_handler+0xb3)[0x7ff411abd413]
/usr/lib64/libglusterfs.so.0(+0x38592)[0x7ff41446d592]
/usr/sbin/glusterfs(main+0x247)[0x4055a7]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7ff413884c4d]
/usr/sbin/glusterfs[0x4032b9]
---------
[2011-04-01 08:43:10.105965] W [io-stats.c:1644:init] gluster-vol: dangling
volume. check volfile
[2011-04-01 08:43:10.106106] W [dict.c:1205:data_to_str] dict: @data=(nil)
[2011-04-01 08:43:10.106142] W [dict.c:1205:data_to_str] dict: @data=(nil)
Given volfile:
+------------------------------------------------------------------------------+
1: volume gluster-vol-client-0
2: type protocol/client
3: option remote-host dscbc-storage1
4: option remote-subvolume /mnt/gluster1
5: option transport-type tcp
6: end-volume
7:
8: volume gluster-vol-client-1
9: type protocol/client
10: option remote-host dscbc-storage2
11: option remote-subvolume /mnt/gluster1
12: option transport-type tcp
13: end-volume
14:
15: volume gluster-vol-client-2
16: type protocol/client
17: option remote-host dscbc-storage3
18: option remote-subvolume /mnt/gluster1
19: option transport-type tcp
20: end-volume
21:
22: volume gluster-vol-client-3
23: type protocol/client
24: option remote-host dscbc-storage4
25: option remote-subvolume /mnt/gluster1
26: option transport-type tcp
27: end-volume
28:
29: volume gluster-vol-client-4
30: type protocol/client
31: option remote-host dscbc-storage1
32: option remote-subvolume /mnt/gluster2
33: option transport-type tcp
34: end-volume
35:
36: volume gluster-vol-client-5
37: type protocol/client
38: option remote-host dscbc-storage2
39: option remote-subvolume /mnt/gluster2
40: option transport-type tcp
41: end-volume
42:
43: volume gluster-vol-client-6
44: type protocol/client
45: option remote-host dscbc-storage3
46: option remote-subvolume /mnt/gluster2
47: option transport-type tcp
48: end-volume
49:
50: volume gluster-vol-client-7
51: type protocol/client
52: option remote-host dscbc-storage4
53: option remote-subvolume /mnt/gluster2
54: option transport-type tcp
55: end-volume
56:
57: volume gluster-vol-client-8
58: type protocol/client
59: option remote-host dscbc-storage1
60: option remote-subvolume /mnt/gluster3
61: option transport-type tcp
62: end-volume
63:
64: volume gluster-vol-client-9
65: type protocol/client
66: option remote-host dscbc-storage2
67: option remote-subvolume /mnt/gluster3
68: option transport-type tcp
69: end-volume
70:
71: volume gluster-vol-client-10
72: type protocol/client
73: option remote-host dscbc-storage3
74: option remote-subvolume /mnt/gluster3
75: option transport-type tcp
76: end-volume
77:
78: volume gluster-vol-client-11
79: type protocol/client
80: option remote-host dscbc-storage4
81: option remote-subvolume /mnt/gluster3
82: option transport-type tcp
83: end-volume
84:
85: volume gluster-vol-client-12
86: type protocol/client
87: option remote-host dscbc-storage1
88: option remote-subvolume /mnt/gluster4
89: option transport-type tcp
90: end-volume
91:
92: volume gluster-vol-client-13
93: type protocol/client
94: option remote-host dscbc-storage2
95: option remote-subvolume /mnt/gluster4
96: option transport-type tcp
97: end-volume
98:
99: volume gluster-vol-client-14
100: type protocol/client
101: option remote-host dscbc-storage1
102: option remote-subvolume /mnt/gluster5
103: option transport-type tcp
104: end-volume
105:
106: volume gluster-vol-client-15
107: type protocol/client
108: option remote-host dscbc-storage2
109: option remote-subvolume /mnt/gluster5
110: option transport-type tcp
111: end-volume
112:
113: volume gluster-vol-client-16
114: type protocol/client
115: option remote-host dscbc-storage1
116: option remote-subvolume /mnt/gluster6
117: option transport-type tcp
118: end-volume
119:
120: volume gluster-vol-client-17
121: type protocol/client
122: option remote-host dscbc-storage2
123: option remote-subvolume /mnt/gluster6
124: option transport-type tcp
125: end-volume
126:
127: volume gluster-vol-replicate-0
128: type cluster/replicate
129: subvolumes gluster-vol-client-0 gluster-vol-client-1
130: end-volume
131:
132: volume gluster-vol-replicate-1
133: type cluster/replicate
134: subvolumes gluster-vol-client-2 gluster-vol-client-3
135: end-volume
136:
137: volume gluster-vol-replicate-2
138: type cluster/replicate
139: subvolumes gluster-vol-client-4 gluster-vol-client-5
140: end-volume
141:
142: volume gluster-vol-replicate-3
143: type cluster/replicate
144: subvolumes gluster-vol-client-6 gluster-vol-client-7
145: end-volume
146:
147: volume gluster-vol-replicate-4
148: type cluster/replicate
149: subvolumes gluster-vol-client-8 gluster-vol-client-9
150: end-volume
151:
152: volume gluster-vol-replicate-5
153: type cluster/replicate
154: subvolumes gluster-vol-client-10 gluster-vol-client-11
155: end-volume
156:
157: volume gluster-vol-replicate-6
158: type cluster/replicate
159: subvolumes gluster-vol-client-12 gluster-vol-client-13
160: end-volume
161:
162: volume gluster-vol-replicate-7
163: type cluster/replicate
164: subvolumes gluster-vol-client-14 gluster-vol-client-15
165: end-volume
166:
167: volume gluster-vol-replicate-8
168: type cluster/replicate
169: subvolumes gluster-vol-client-16 gluster-vol-client-17
170: end-volume
171:
172: volume gluster-vol-dht
173: type cluster/distribute
174: subvolumes gluster-vol-replicate-0 gluster-vol-replicate-1
gluster-vol-replicate-2 gluster-vol-replicate-3 gluster-vol-replicate-4
gluster-vol-replicate-5 gluster-vol-replicate-6 gluster-vol-replicate-7
gluster-vol-replicate-8
175: end-volume
176:
177: volume gluster-vol-write-behind
178: type performance/write-behind
179: subvolumes gluster-vol-dht
180: end-volume
181:
182: volume gluster-vol-read-ahead
183: type performance/read-ahead
184: subvolumes gluster-vol-write-behind
185: end-volume
186:
187: volume gluster-vol-io-cache
188: type performance/io-cache
189: subvolumes gluster-vol-read-ahead
190: end-volume
191:
192: volume gluster-vol-quick-read
193: type performance/quick-read
194: subvolumes gluster-vol-io-cache
195: end-volume
196:
197: volume gluster-vol-stat-prefetch
198: type performance/stat-prefetch
199: subvolumes gluster-vol-quick-read
200: end-volume
201:
202: volume gluster-vol
203: type debug/io-stats
204: subvolumes gluster-vol-stat-prefetch
205: end-volume
+------------------------------------------------------------------------------+
[2011-04-01 08:43:14.48158] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-13: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.48607] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-17: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.49124] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.49581] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-13: Connected to 165.112.107.217:24012, attached to remote
volume '/mnt/gluster4'.
[2011-04-01 08:43:14.49694] I [afr-common.c:2572:afr_notify]
gluster-vol-replicate-6: Subvolume 'gluster-vol-client-13' came back up; going
online.
[2011-04-01 08:43:14.49937] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-17: Connected to 165.112.107.217:24014, attached to remote
volume '/mnt/gluster6'.
[2011-04-01 08:43:14.49981] I [afr-common.c:2572:afr_notify]
gluster-vol-replicate-8: Subvolume 'gluster-vol-client-17' came back up; going
online.
[2011-04-01 08:43:14.50182] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-0: Connected to 165.112.107.216:24009, attached to remote
volume '/mnt/gluster1'.
[2011-04-01 08:43:14.50221] I [afr-common.c:2572:afr_notify]
gluster-vol-replicate-0: Subvolume 'gluster-vol-client-0' came back up; going
online.
[2011-04-01 08:43:14.50834] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.51365] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-8: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.51719] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-3: Connected to 165.112.107.214:24009, attached to remote
volume '/mnt/gluster1'.
[2011-04-01 08:43:14.51791] I [afr-common.c:2572:afr_notify]
gluster-vol-replicate-1: Subvolume 'gluster-vol-client-3' came back up; going
online.
[2011-04-01 08:43:14.51970] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-10: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.52103] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-8: Connected to 165.112.107.216:24011, attached to remote
volume '/mnt/gluster3'.
[2011-04-01 08:43:14.52140] I [afr-common.c:2572:afr_notify]
gluster-vol-replicate-4: Subvolume 'gluster-vol-client-8' came back up; going
online.
[2011-04-01 08:43:14.52264] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-14: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.52840] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-14: Connected to 165.112.107.216:24013, attached to remote
volume '/mnt/gluster5'.
[2011-04-01 08:43:14.52948] I [afr-common.c:2572:afr_notify]
gluster-vol-replicate-7: Subvolume 'gluster-vol-client-14' came back up; going
online.
[2011-04-01 08:43:14.53177] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-10: Connected to 165.112.107.213:24011, attached to remote
volume '/mnt/gluster3'.
[2011-04-01 08:43:14.53218] I [afr-common.c:2572:afr_notify]
gluster-vol-replicate-5: Subvolume 'gluster-vol-client-10' came back up; going
online.
[2011-04-01 08:43:14.53449] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-4: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.54084] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-4: Connected to 165.112.107.216:24010, attached to remote
volume '/mnt/gluster2'.
[2011-04-01 08:43:14.54147] I [afr-common.c:2572:afr_notify]
gluster-vol-replicate-2: Subvolume 'gluster-vol-client-4' came back up; going
online.
[2011-04-01 08:43:14.54307] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-12: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.54487] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-16: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.54973] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-12: Connected to 165.112.107.216:24012, attached to remote
volume '/mnt/gluster4'.
[2011-04-01 08:43:14.55087] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-11: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.55223] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-16: Connected to 165.112.107.216:24014, attached to remote
volume '/mnt/gluster6'.
[2011-04-01 08:43:14.55797] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.56001] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-11: Connected to 165.112.107.214:24011, attached to remote
volume '/mnt/gluster3'.
[2011-04-01 08:43:14.56299] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-6: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.56455] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-2: Connected to 165.112.107.213:24009, attached to remote
volume '/mnt/gluster1'.
[2011-04-01 08:43:14.56732] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-7: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.56909] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-6: Connected to 165.112.107.213:24010, attached to remote
volume '/mnt/gluster2'.
[2011-04-01 08:43:14.56943] I [afr-common.c:2572:afr_notify]
gluster-vol-replicate-3: Subvolume 'gluster-vol-client-6' came back up; going
online.
[2011-04-01 08:43:14.69629] I [fuse-bridge.c:2821:fuse_init] glusterfs-fuse:
FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
[2011-04-01 08:43:14.69745] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-7: Connected to 165.112.107.214:24010, attached to remote
volume '/mnt/gluster2'.
[2011-04-01 08:43:14.69961] I
[client-handshake.c:1005:select_server_supported_programs]
gluster-vol-client-15: Using Program GlusterFS-3.1.0, Num (1298437), Version
(310)
[2011-04-01 08:43:14.70467] I [client-handshake.c:841:client_setvolume_cbk]
gluster-vol-client-15: Connected to 165.112.107.217:24013, attached to remote
volume '/mnt/gluster5'.
[2011-04-01 08:43:14.70627] I [afr-common.c:819:afr_fresh_lookup_cbk]
gluster-vol-replicate-0: added root inode
[2011-04-01 08:43:14.72159] I [afr-common.c:819:afr_fresh_lookup_cbk]
gluster-vol-replicate-3: added root inode
[2011-04-01 08:43:14.72267] I [afr-common.c:819:afr_fresh_lookup_cbk]
gluster-vol-replicate-2: added root inode
[2011-04-01 08:43:14.72350] I [afr-common.c:819:afr_fresh_lookup_cbk]
gluster-vol-replicate-4: added root inode
[2011-04-01 08:43:14.72440] I [afr-common.c:819:afr_fresh_lookup_cbk]
gluster-vol-replicate-1: added root inode
[2011-04-01 08:43:14.72551] I [afr-common.c:819:afr_fresh_lookup_cbk]
gluster-vol-replicate-5: added root inode
[2011-04-01 08:43:14.72654] I [afr-common.c:819:afr_fresh_lookup_cbk]
gluster-vol-replicate-7: added root inode
[2011-04-01 08:43:14.72712] I [afr-common.c:819:afr_fresh_lookup_cbk]
gluster-vol-replicate-6: added root inode
[2011-04-01 08:43:14.72981] I [afr-common.c:819:afr_fresh_lookup_cbk]
gluster-vol-replicate-8: added root inode
[2011-04-01 08:43:14.148489] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-0: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005, reason:
checksums of directory differ, forced merge option set
[2011-04-01 08:43:14.149294] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-2: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005, reason:
checksums of directory differ, forced merge option set
[2011-04-01 08:43:14.149411] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-4: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005, reason:
checksums of directory differ, forced merge option set
[2011-04-01 08:43:14.149529] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-0: invalid argument: inode
[2011-04-01 08:43:14.149563] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-0: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005
[2011-04-01 08:43:14.150346] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-2: invalid argument: inode
[2011-04-01 08:43:14.150409] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-2: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005
[2011-04-01 08:43:14.150474] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-4: invalid argument: inode
[2011-04-01 08:43:14.150503] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-4: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005
[2011-04-01 08:43:14.725449] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-0: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1,
reason: checksums of directory differ, forced merge option set
[2011-04-01 08:43:14.725637] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-2: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1,
reason: checksums of directory differ, forced merge option set
[2011-04-01 08:43:14.726038] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-4: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1,
reason: checksums of directory differ, forced merge option set
[2011-04-01 08:43:14.726518] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-0: invalid argument: inode
[2011-04-01 08:43:14.726582] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-0: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1
[2011-04-01 08:43:14.726641] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-2: invalid argument: inode
[2011-04-01 08:43:14.726670] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-2: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1
[2011-04-01 08:43:14.727285] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-4: invalid argument: inode
[2011-04-01 08:43:14.727351] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-4: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1
[2011-04-01 08:43:15.50797] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-2: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005, reason:
checksums of directory differ, forced merge option set
[2011-04-01 08:43:15.51193] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-4: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005, reason:
checksums of directory differ, forced merge option set
[2011-04-01 08:43:15.51713] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-0: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005, reason:
checksums of directory differ, forced merge option set
[2011-04-01 08:43:15.52627] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-2: invalid argument: inode
[2011-04-01 08:43:15.52690] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-2: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005
[2011-04-01 08:43:15.52753] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-4: invalid argument: inode
[2011-04-01 08:43:15.52781] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-4: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005
[2011-04-01 08:43:15.53061] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-0: invalid argument: inode
[2011-04-01 08:43:15.53123] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-0: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005
[2011-04-01 08:43:15.490907] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-0: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1,
reason: checksums of directory differ, forced merge option set
[2011-04-01 08:43:15.491743] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-2: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1,
reason: checksums of directory differ, forced merge option set
[2011-04-01 08:43:15.492202] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
gluster-vol-replicate-4: entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1,
reason: checksums of directory differ, forced merge option set
[2011-04-01 08:43:15.492399] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-0: invalid argument: inode
[2011-04-01 08:43:15.492434] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-0: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1
[2011-04-01 08:43:15.492789] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-2: invalid argument: inode
[2011-04-01 08:43:15.492850] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-2: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1
[2011-04-01 08:43:15.493051] E [afr-common.c:110:afr_set_split_brain]
gluster-vol-replicate-4: invalid argument: inode
[2011-04-01 08:43:15.493105] I
[afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
gluster-vol-replicate-4: background entry self-heal completed on
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1
[2011-04-01 08:43:15.553051] I [afr-common.c:716:afr_lookup_done]
gluster-vol-replicate-0: background entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005
[2011-04-01 08:43:15.553465] I [afr-common.c:716:afr_lookup_done]
gluster-vol-replicate-4: background entry self-heal triggered. path:
/gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005
[2011-04-01 08:43:15.553640] I [afr-common.c:716:afr_lookup_done]
gluster-vol-replicate-2: background entry self-heal triggered. path: /
--
This message was sent on behalf of luomails at gmail.com at openSubscriber.com
http://www.opensubscriber.com/message/gluster-users@gluster.org/15035860.html
More information about the Gluster-users
mailing list