[Gluster-users] 3.1.2 feedback
Mohit Anchlia
mohitanchlia at gmail.com
Mon Apr 11 16:24:30 UTC 2011
This is very disturbing. It will be good to see a response to this thread!
On Tue, Apr 5, 2011 at 3:36 PM, <luomails at gmail.com> wrote:
> Service is down when programs are running
>
> System: Ubuntu 10.04 (Linux version 2.6.32-21-server (buildd at yellow) (gcc
> version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #32-Ubuntu SMP Fri Apr 16 09:17:34 UTC
> 2010)
> Mount type: Gluster Native Client
>
> Our service was down when we were running several programs, one machine was
> running that would generate thousands files(maybe more), connected with the
> main sever(Samba) and saved data in gluster-vol. The machine would run at least
> one week, couple days ago, it was down. By the way, we also opened other
> sessions to do the data backup(sftp) and some computing.
>
> When I checked the gluster-vol, I got:
> $df -ah
> df: `/mnt/gluster-vol': Transport endpoint is not connected
>
> Then I checked the mount point:
> $ ls -la
> ls: cannot access gluster-vol: Transport endpoint is not connected
> d????????? ? ? ? ? ? gluster-vol
>
> I also checked the peers from the main server, it couldn't see the peers, when
> I ssh to data storages that can see each other.
> $ sudo gluster volume info all
> No volumes present
> $ sudo gluster peer status
> No peers present
>
> Then I checked the status
> $ sudo /etc/init.d/glusterd status
> * glusterd service is not running.
>
> After I restarted the services, unmount the gluster-vol, and mount it. It
> works. My questions is: Is there any restriction for the tasks loaded for
> glusterfs 3.1.2? For example, how many files can be accessed at the same time
> from gluster-vol? and how many programs can we running at the same time?
>
> Please find the attached log file for your reference at that time.
>
> Note: Someone has reported from this group "I have been testing 3.1.2 over the last few days.
> My overall impression is that it resolved several bugs from 3.1.1, but the
> latest version is still prone to crashing under moderate to heavy loads. "
> (http://www.opensubscriber.com/message/gluster-users@gluster.org/15035860.html).
> Is it same issue?
>
> Thanks for your help. We want to figure it out and prevent it in the future.
>
> --------Related content of log file--------
> [2011-03-31 20:57:05.462233] I [afr-common.c:716:afr_lookup_done]
> gluster-vol-replicate-4: background meta-data data entry self-heal triggered.
> path: /home/junzhu/Projects/Mito_Xing_03_23_11/data
> [2011-03-31 20:57:05.465182] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-7: background meta-data data entry self-heal completed
> on /home/junzhu/Projects/Mito_Xing_03_23_11/data
> [2011-03-31 20:57:05.465748] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-8: background meta-data data entry self-heal completed
> on /home/junzhu/Projects/Mito_Xing_03_23_11/data
> [2011-03-31 20:57:05.465788] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-6: background meta-data data entry self-heal completed
> on /home/junzhu/Projects/Mito_Xing_03_23_11/data
> [2011-03-31 20:57:05.466198] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-0: background meta-data data entry self-heal completed
> on /home/junzhu/Projects/Mito_Xing_03_23_11/data
> [2011-03-31 20:57:05.466243] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-2: background meta-data data entry self-heal completed
> on /home/junzhu/Projects/Mito_Xing_03_23_11/data
> [2011-03-31 20:57:05.496131] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-4: background meta-data data entry self-heal completed
> on /home/junzhu/Projects/Mito_Xing_03_23_11/data
> [2011-03-31 20:57:05.529025] I [afr-common.c:662:afr_lookup_done]
> gluster-vol-replicate-8: entries are missing in lookup of
> /home/junzhu/Projects/Mito_Xing_03_23_11/data/read1.txt.
> [2011-03-31 20:57:05.529097] I [afr-common.c:716:afr_lookup_done]
> gluster-vol-replicate-8: background meta-data data entry self-heal triggered.
> path: /home/junzhu/Projects/Mito_Xing_03_23_11/data/read1.txt
> [2011-03-31 20:57:05.535098] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-8: background meta-data data entry self-heal completed
> on /home/junzhu/Projects/Mito_Xing_03_23_11/data/read1.txt
> pending frames:
>
> frame : type(1) op(READ)
> frame : type(1) op(READ)
> frame : type(1) op(READ)
> frame : type(1) op(READ)
> frame : type(1) op(READ)
> frame : type(1) op(READ)
> frame : type(1) op(READ)
> frame : type(1) op(READ)
> frame : type(1) op(CREATE)
> frame : type(1) op(CREATE)
> frame : type(1) op(CREATE)
> frame : type(1) op(CREATE)
> frame : type(1) op(CREATE)
> frame : type(1) op(READ)
> frame : type(1) op(READ)
>
> patchset: v3.1.1-64-gf2a067c
> signal received: 6
> time of crash: 2011-03-31 22:10:01
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> fdatasync 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 3.1.2
> /lib/libc.so.6(+0x33af0)[0x7ff413899af0]
> /lib/libc.so.6(gsignal+0x35)[0x7ff413899a75]
> /lib/libc.so.6(abort+0x180)[0x7ff41389d5c0]
> /lib/libc.so.6(__assert_fail+0xf1)[0x7ff413892941]
> /lib/libpthread.so.0(pthread_mutex_lock+0x7b)[0x7ff413bf243b]
> /usr/lib64/glusterfs/3.1.2/xlator/performance/io-cache.so(ioc_create_cbk+0xb0)[0x7ff4101d9540]
> /usr/lib64/glusterfs/3.1.2/xlator/performance/read-ahead.so(ra_create_cbk+0x1ba)[0x7ff4103e4daa]
> /usr/lib64/glusterfs/3.1.2/xlator/performance/write-behind.so(wb_create_cbk+0x10b)[0x7ff4105eee0b]
> /usr/lib64/glusterfs/3.1.2/xlator/cluster/distribute.so(dht_create_cbk+0x2b8)[0x7ff410816b18]
> /usr/lib64/glusterfs/3.1.2/xlator/cluster/replicate.so(afr_create_unwind+0x12b)[0x7ff410a2fc9b]
> /usr/lib64/glusterfs/3.1.2/xlator/cluster/replicate.so(afr_create_wind_cbk+0x128)[0x7ff410a32d78]
> /usr/lib64/glusterfs/3.1.2/xlator/protocol/client.so(client3_1_create_cbk+0x92f)[0x7ff410c939cf]
> /usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7ff41422dc15]
> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xc9)[0x7ff41422de69]
> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7ff41422902d]
> /usr/lib64/glusterfs/3.1.2/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7ff411abd344]
> /usr/lib64/glusterfs/3.1.2/rpc-transport/socket.so(socket_event_handler+0xb3)[0x7ff411abd413]
> /usr/lib64/libglusterfs.so.0(+0x38592)[0x7ff41446d592]
> /usr/sbin/glusterfs(main+0x247)[0x4055a7]
> /lib/libc.so.6(__libc_start_main+0xfd)[0x7ff413884c4d]
> /usr/sbin/glusterfs[0x4032b9]
> ---------
> [2011-04-01 08:43:10.105965] W [io-stats.c:1644:init] gluster-vol: dangling
> volume. check volfile
> [2011-04-01 08:43:10.106106] W [dict.c:1205:data_to_str] dict: @data=(nil)
> [2011-04-01 08:43:10.106142] W [dict.c:1205:data_to_str] dict: @data=(nil)
> Given volfile:
> +------------------------------------------------------------------------------+
> 1: volume gluster-vol-client-0
> 2: type protocol/client
> 3: option remote-host dscbc-storage1
> 4: option remote-subvolume /mnt/gluster1
> 5: option transport-type tcp
> 6: end-volume
> 7:
> 8: volume gluster-vol-client-1
> 9: type protocol/client
> 10: option remote-host dscbc-storage2
> 11: option remote-subvolume /mnt/gluster1
> 12: option transport-type tcp
> 13: end-volume
> 14:
> 15: volume gluster-vol-client-2
> 16: type protocol/client
> 17: option remote-host dscbc-storage3
> 18: option remote-subvolume /mnt/gluster1
> 19: option transport-type tcp
> 20: end-volume
> 21:
> 22: volume gluster-vol-client-3
> 23: type protocol/client
> 24: option remote-host dscbc-storage4
> 25: option remote-subvolume /mnt/gluster1
> 26: option transport-type tcp
> 27: end-volume
> 28:
> 29: volume gluster-vol-client-4
> 30: type protocol/client
> 31: option remote-host dscbc-storage1
> 32: option remote-subvolume /mnt/gluster2
> 33: option transport-type tcp
> 34: end-volume
> 35:
> 36: volume gluster-vol-client-5
> 37: type protocol/client
> 38: option remote-host dscbc-storage2
> 39: option remote-subvolume /mnt/gluster2
> 40: option transport-type tcp
> 41: end-volume
> 42:
> 43: volume gluster-vol-client-6
> 44: type protocol/client
> 45: option remote-host dscbc-storage3
> 46: option remote-subvolume /mnt/gluster2
> 47: option transport-type tcp
> 48: end-volume
> 49:
> 50: volume gluster-vol-client-7
> 51: type protocol/client
> 52: option remote-host dscbc-storage4
> 53: option remote-subvolume /mnt/gluster2
> 54: option transport-type tcp
> 55: end-volume
> 56:
> 57: volume gluster-vol-client-8
> 58: type protocol/client
> 59: option remote-host dscbc-storage1
> 60: option remote-subvolume /mnt/gluster3
> 61: option transport-type tcp
> 62: end-volume
> 63:
> 64: volume gluster-vol-client-9
> 65: type protocol/client
> 66: option remote-host dscbc-storage2
> 67: option remote-subvolume /mnt/gluster3
> 68: option transport-type tcp
> 69: end-volume
> 70:
> 71: volume gluster-vol-client-10
> 72: type protocol/client
> 73: option remote-host dscbc-storage3
> 74: option remote-subvolume /mnt/gluster3
> 75: option transport-type tcp
> 76: end-volume
> 77:
> 78: volume gluster-vol-client-11
> 79: type protocol/client
> 80: option remote-host dscbc-storage4
> 81: option remote-subvolume /mnt/gluster3
> 82: option transport-type tcp
> 83: end-volume
> 84:
> 85: volume gluster-vol-client-12
> 86: type protocol/client
> 87: option remote-host dscbc-storage1
> 88: option remote-subvolume /mnt/gluster4
> 89: option transport-type tcp
> 90: end-volume
> 91:
> 92: volume gluster-vol-client-13
> 93: type protocol/client
> 94: option remote-host dscbc-storage2
> 95: option remote-subvolume /mnt/gluster4
> 96: option transport-type tcp
> 97: end-volume
> 98:
> 99: volume gluster-vol-client-14
> 100: type protocol/client
> 101: option remote-host dscbc-storage1
> 102: option remote-subvolume /mnt/gluster5
> 103: option transport-type tcp
> 104: end-volume
> 105:
> 106: volume gluster-vol-client-15
> 107: type protocol/client
> 108: option remote-host dscbc-storage2
> 109: option remote-subvolume /mnt/gluster5
> 110: option transport-type tcp
> 111: end-volume
> 112:
> 113: volume gluster-vol-client-16
> 114: type protocol/client
> 115: option remote-host dscbc-storage1
> 116: option remote-subvolume /mnt/gluster6
> 117: option transport-type tcp
> 118: end-volume
> 119:
> 120: volume gluster-vol-client-17
> 121: type protocol/client
> 122: option remote-host dscbc-storage2
> 123: option remote-subvolume /mnt/gluster6
> 124: option transport-type tcp
> 125: end-volume
> 126:
> 127: volume gluster-vol-replicate-0
> 128: type cluster/replicate
> 129: subvolumes gluster-vol-client-0 gluster-vol-client-1
> 130: end-volume
> 131:
> 132: volume gluster-vol-replicate-1
> 133: type cluster/replicate
> 134: subvolumes gluster-vol-client-2 gluster-vol-client-3
> 135: end-volume
> 136:
> 137: volume gluster-vol-replicate-2
> 138: type cluster/replicate
> 139: subvolumes gluster-vol-client-4 gluster-vol-client-5
> 140: end-volume
> 141:
> 142: volume gluster-vol-replicate-3
> 143: type cluster/replicate
> 144: subvolumes gluster-vol-client-6 gluster-vol-client-7
> 145: end-volume
> 146:
> 147: volume gluster-vol-replicate-4
> 148: type cluster/replicate
> 149: subvolumes gluster-vol-client-8 gluster-vol-client-9
> 150: end-volume
> 151:
> 152: volume gluster-vol-replicate-5
> 153: type cluster/replicate
> 154: subvolumes gluster-vol-client-10 gluster-vol-client-11
> 155: end-volume
> 156:
> 157: volume gluster-vol-replicate-6
> 158: type cluster/replicate
> 159: subvolumes gluster-vol-client-12 gluster-vol-client-13
> 160: end-volume
> 161:
> 162: volume gluster-vol-replicate-7
> 163: type cluster/replicate
> 164: subvolumes gluster-vol-client-14 gluster-vol-client-15
> 165: end-volume
> 166:
> 167: volume gluster-vol-replicate-8
> 168: type cluster/replicate
> 169: subvolumes gluster-vol-client-16 gluster-vol-client-17
> 170: end-volume
> 171:
> 172: volume gluster-vol-dht
> 173: type cluster/distribute
> 174: subvolumes gluster-vol-replicate-0 gluster-vol-replicate-1
> gluster-vol-replicate-2 gluster-vol-replicate-3 gluster-vol-replicate-4
> gluster-vol-replicate-5 gluster-vol-replicate-6 gluster-vol-replicate-7
> gluster-vol-replicate-8
> 175: end-volume
> 176:
> 177: volume gluster-vol-write-behind
> 178: type performance/write-behind
> 179: subvolumes gluster-vol-dht
> 180: end-volume
> 181:
> 182: volume gluster-vol-read-ahead
> 183: type performance/read-ahead
> 184: subvolumes gluster-vol-write-behind
> 185: end-volume
> 186:
> 187: volume gluster-vol-io-cache
> 188: type performance/io-cache
> 189: subvolumes gluster-vol-read-ahead
> 190: end-volume
> 191:
> 192: volume gluster-vol-quick-read
> 193: type performance/quick-read
> 194: subvolumes gluster-vol-io-cache
> 195: end-volume
> 196:
> 197: volume gluster-vol-stat-prefetch
> 198: type performance/stat-prefetch
> 199: subvolumes gluster-vol-quick-read
> 200: end-volume
> 201:
> 202: volume gluster-vol
> 203: type debug/io-stats
> 204: subvolumes gluster-vol-stat-prefetch
> 205: end-volume
>
> +------------------------------------------------------------------------------+
> [2011-04-01 08:43:14.48158] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-13: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.48607] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-17: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.49124] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.49581] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-13: Connected to 165.112.107.217:24012, attached to remote
> volume '/mnt/gluster4'.
> [2011-04-01 08:43:14.49694] I [afr-common.c:2572:afr_notify]
> gluster-vol-replicate-6: Subvolume 'gluster-vol-client-13' came back up; going
> online.
> [2011-04-01 08:43:14.49937] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-17: Connected to 165.112.107.217:24014, attached to remote
> volume '/mnt/gluster6'.
> [2011-04-01 08:43:14.49981] I [afr-common.c:2572:afr_notify]
> gluster-vol-replicate-8: Subvolume 'gluster-vol-client-17' came back up; going
> online.
> [2011-04-01 08:43:14.50182] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-0: Connected to 165.112.107.216:24009, attached to remote
> volume '/mnt/gluster1'.
> [2011-04-01 08:43:14.50221] I [afr-common.c:2572:afr_notify]
> gluster-vol-replicate-0: Subvolume 'gluster-vol-client-0' came back up; going
> online.
> [2011-04-01 08:43:14.50834] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.51365] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-8: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.51719] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-3: Connected to 165.112.107.214:24009, attached to remote
> volume '/mnt/gluster1'.
> [2011-04-01 08:43:14.51791] I [afr-common.c:2572:afr_notify]
> gluster-vol-replicate-1: Subvolume 'gluster-vol-client-3' came back up; going
> online.
> [2011-04-01 08:43:14.51970] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-10: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.52103] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-8: Connected to 165.112.107.216:24011, attached to remote
> volume '/mnt/gluster3'.
> [2011-04-01 08:43:14.52140] I [afr-common.c:2572:afr_notify]
> gluster-vol-replicate-4: Subvolume 'gluster-vol-client-8' came back up; going
> online.
> [2011-04-01 08:43:14.52264] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-14: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.52840] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-14: Connected to 165.112.107.216:24013, attached to remote
> volume '/mnt/gluster5'.
> [2011-04-01 08:43:14.52948] I [afr-common.c:2572:afr_notify]
> gluster-vol-replicate-7: Subvolume 'gluster-vol-client-14' came back up; going
> online.
> [2011-04-01 08:43:14.53177] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-10: Connected to 165.112.107.213:24011, attached to remote
> volume '/mnt/gluster3'.
> [2011-04-01 08:43:14.53218] I [afr-common.c:2572:afr_notify]
> gluster-vol-replicate-5: Subvolume 'gluster-vol-client-10' came back up; going
> online.
> [2011-04-01 08:43:14.53449] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-4: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.54084] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-4: Connected to 165.112.107.216:24010, attached to remote
> volume '/mnt/gluster2'.
> [2011-04-01 08:43:14.54147] I [afr-common.c:2572:afr_notify]
> gluster-vol-replicate-2: Subvolume 'gluster-vol-client-4' came back up; going
> online.
> [2011-04-01 08:43:14.54307] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-12: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.54487] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-16: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.54973] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-12: Connected to 165.112.107.216:24012, attached to remote
> volume '/mnt/gluster4'.
> [2011-04-01 08:43:14.55087] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-11: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.55223] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-16: Connected to 165.112.107.216:24014, attached to remote
> volume '/mnt/gluster6'.
> [2011-04-01 08:43:14.55797] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.56001] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-11: Connected to 165.112.107.214:24011, attached to remote
> volume '/mnt/gluster3'.
> [2011-04-01 08:43:14.56299] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-6: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.56455] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-2: Connected to 165.112.107.213:24009, attached to remote
> volume '/mnt/gluster1'.
> [2011-04-01 08:43:14.56732] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-7: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.56909] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-6: Connected to 165.112.107.213:24010, attached to remote
> volume '/mnt/gluster2'.
> [2011-04-01 08:43:14.56943] I [afr-common.c:2572:afr_notify]
> gluster-vol-replicate-3: Subvolume 'gluster-vol-client-6' came back up; going
> online.
> [2011-04-01 08:43:14.69629] I [fuse-bridge.c:2821:fuse_init] glusterfs-fuse:
> FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
> [2011-04-01 08:43:14.69745] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-7: Connected to 165.112.107.214:24010, attached to remote
> volume '/mnt/gluster2'.
> [2011-04-01 08:43:14.69961] I
> [client-handshake.c:1005:select_server_supported_programs]
> gluster-vol-client-15: Using Program GlusterFS-3.1.0, Num (1298437), Version
> (310)
> [2011-04-01 08:43:14.70467] I [client-handshake.c:841:client_setvolume_cbk]
> gluster-vol-client-15: Connected to 165.112.107.217:24013, attached to remote
> volume '/mnt/gluster5'.
> [2011-04-01 08:43:14.70627] I [afr-common.c:819:afr_fresh_lookup_cbk]
> gluster-vol-replicate-0: added root inode
> [2011-04-01 08:43:14.72159] I [afr-common.c:819:afr_fresh_lookup_cbk]
> gluster-vol-replicate-3: added root inode
> [2011-04-01 08:43:14.72267] I [afr-common.c:819:afr_fresh_lookup_cbk]
> gluster-vol-replicate-2: added root inode
> [2011-04-01 08:43:14.72350] I [afr-common.c:819:afr_fresh_lookup_cbk]
> gluster-vol-replicate-4: added root inode
> [2011-04-01 08:43:14.72440] I [afr-common.c:819:afr_fresh_lookup_cbk]
> gluster-vol-replicate-1: added root inode
> [2011-04-01 08:43:14.72551] I [afr-common.c:819:afr_fresh_lookup_cbk]
> gluster-vol-replicate-5: added root inode
> [2011-04-01 08:43:14.72654] I [afr-common.c:819:afr_fresh_lookup_cbk]
> gluster-vol-replicate-7: added root inode
> [2011-04-01 08:43:14.72712] I [afr-common.c:819:afr_fresh_lookup_cbk]
> gluster-vol-replicate-6: added root inode
> [2011-04-01 08:43:14.72981] I [afr-common.c:819:afr_fresh_lookup_cbk]
> gluster-vol-replicate-8: added root inode
> [2011-04-01 08:43:14.148489] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-0: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005, reason:
> checksums of directory differ, forced merge option set
> [2011-04-01 08:43:14.149294] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-2: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005, reason:
> checksums of directory differ, forced merge option set
> [2011-04-01 08:43:14.149411] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-4: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005, reason:
> checksums of directory differ, forced merge option set
> [2011-04-01 08:43:14.149529] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-0: invalid argument: inode
> [2011-04-01 08:43:14.149563] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-0: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005
> [2011-04-01 08:43:14.150346] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-2: invalid argument: inode
> [2011-04-01 08:43:14.150409] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-2: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005
> [2011-04-01 08:43:14.150474] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-4: invalid argument: inode
> [2011-04-01 08:43:14.150503] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-4: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005
> [2011-04-01 08:43:14.725449] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-0: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1,
> reason: checksums of directory differ, forced merge option set
> [2011-04-01 08:43:14.725637] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-2: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1,
> reason: checksums of directory differ, forced merge option set
> [2011-04-01 08:43:14.726038] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-4: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1,
> reason: checksums of directory differ, forced merge option set
> [2011-04-01 08:43:14.726518] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-0: invalid argument: inode
> [2011-04-01 08:43:14.726582] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-0: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1
> [2011-04-01 08:43:14.726641] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-2: invalid argument: inode
> [2011-04-01 08:43:14.726670] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-2: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1
> [2011-04-01 08:43:14.727285] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-4: invalid argument: inode
> [2011-04-01 08:43:14.727351] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-4: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005/C104.1
> [2011-04-01 08:43:15.50797] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-2: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005, reason:
> checksums of directory differ, forced merge option set
> [2011-04-01 08:43:15.51193] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-4: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005, reason:
> checksums of directory differ, forced merge option set
> [2011-04-01 08:43:15.51713] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-0: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005, reason:
> checksums of directory differ, forced merge option set
> [2011-04-01 08:43:15.52627] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-2: invalid argument: inode
> [2011-04-01 08:43:15.52690] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-2: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005
> [2011-04-01 08:43:15.52753] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-4: invalid argument: inode
> [2011-04-01 08:43:15.52781] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-4: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005
> [2011-04-01 08:43:15.53061] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-0: invalid argument: inode
> [2011-04-01 08:43:15.53123] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-0: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005
> [2011-04-01 08:43:15.490907] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-0: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1,
> reason: checksums of directory differ, forced merge option set
> [2011-04-01 08:43:15.491743] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-2: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1,
> reason: checksums of directory differ, forced merge option set
> [2011-04-01 08:43:15.492202] I [afr-dir-read.c:171:afr_examine_dir_readdir_cbk]
> gluster-vol-replicate-4: entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1,
> reason: checksums of directory differ, forced merge option set
> [2011-04-01 08:43:15.492399] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-0: invalid argument: inode
> [2011-04-01 08:43:15.492434] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-0: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1
> [2011-04-01 08:43:15.492789] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-2: invalid argument: inode
> [2011-04-01 08:43:15.492850] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-2: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1
> [2011-04-01 08:43:15.493051] E [afr-common.c:110:afr_set_split_brain]
> gluster-vol-replicate-4: invalid argument: inode
> [2011-04-01 08:43:15.493105] I
> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
> gluster-vol-replicate-4: background entry self-heal completed on
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Data/Intensities/L005/C104.1
> [2011-04-01 08:43:15.553051] I [afr-common.c:716:afr_lookup_done]
> gluster-vol-replicate-0: background entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005
> [2011-04-01 08:43:15.553465] I [afr-common.c:716:afr_lookup_done]
> gluster-vol-replicate-4: background entry self-heal triggered. path:
> /gluster/data/hiseq/110328_SN183_0222_BC008HABXX/Thumbnail_Images/L005
> [2011-04-01 08:43:15.553640] I [afr-common.c:716:afr_lookup_done]
> gluster-vol-replicate-2: background entry self-heal triggered. path: /
>
> --
> This message was sent on behalf of luomails at gmail.com at openSubscriber.com
> http://www.opensubscriber.com/message/gluster-users@gluster.org/15035860.html
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list