[Gluster-users] question about Transport endpoint is not connected

songxin songxin_1980 at 126.com
Mon Nov 21 09:06:53 UTC 2016 

Hi all,
I have a question to consult you.


Gluterfs version is 3.7.6.
I create a replicate volume using two node.


When I access the mount point, it show that "Transport endpoint is not connected" in client log.
From log it seems that the glusterfs and glusterfsd disconnect.But I don't why disconnection happen on local sockect.




Below is the log from client and service.


mnt-c.log(client log) 


[2016-10-31 04:04:08.023782] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs --acl --volfile-server=10.32.1.144 --volfile-id=c_glusterfs /mnt/c)
[2016-10-31 04:04:08.042622] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-10-31 04:04:08.052641] I [graph.c:269:gf_add_cmdline_options] 0-c_glusterfs-md-cache: adding option 'cache-posix-acl' for volume 'c_glusterfs-md-cache' with value 'true'
[2016-10-31 04:04:08.058854] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-10-31 04:04:08.060066] I [MSGID: 114020] [client.c:2118:notify] 0-c_glusterfs-client-9: parent translators are ready, attempting connect on transport
[2016-10-31 04:04:08.061169] I [MSGID: 114020] [client.c:2118:notify] 0-c_glusterfs-client-10: parent translators are ready, attempting connect on transport
[2016-10-31 04:04:08.061694] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-9: changing port to 49391 (from 0)
Final graph:
+------------------------------------------------------------------------------+
  1: volume c_glusterfs-client-9
  2:     type protocol/client
  3:     option ping-timeout 4
  4:     option remote-host 10.32.1.144
  5:     option remote-subvolume /opt/lvmdir/c2/brick
  6:     option transport-type socket
  7:     option username 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
  8:     option password ade893cc-19dd-4d67-ae8e-cb929ef4e770
  9:     option send-gids true
 10: end-volume
 11:  
 12: volume c_glusterfs-client-10
 13:     type protocol/client
 14:     option ping-timeout 4
 15:     option remote-host 10.32.0.48
 16:     option remote-subvolume /opt/lvmdir/c2/brick
 17:     option transport-type socket
 18:     option username 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
 19:     option password ade893cc-19dd-4d67-ae8e-cb929ef4e770
 20:     option send-gids true
 21: end-volume
 22:  
 23: volume c_glusterfs-replicate-0
 24:     type cluster/replicate
 25:     subvolumes c_glusterfs-client-9 c_glusterfs-client-10
 26: end-volume
 27:  
 28: volume c_glusterfs-dht
 29:     type cluster/distribute
 30:     subvolumes c_glusterfs-replicate-0
 31: end-volume
 32:  
 33: volume c_glusterfs-write-behind
 34:     type performance/write-behind
 35:     subvolumes c_glusterfs-dht
 36: end-volume
 37:  
 38: volume c_glusterfs-read-ahead
 39:     type performance/read-ahead
 40:     subvolumes c_glusterfs-write-behind
 41: end-volume
 42:  
 43: volume c_glusterfs-readdir-ahead
 44:     type performance/readdir-ahead
 45:     subvolumes c_glusterfs-read-ahead
 46: end-volume
 47:  
 48: volume c_glusterfs-io-cache
 49:     type performance/io-cache
 50:     subvolumes c_glusterfs-readdir-ahead
 51: end-volume
 52:  
 53: volume c_glusterfs-quick-read
 54:     type performance/quick-read
 55:     subvolumes c_glusterfs-io-cache
 56: end-volume
 57:  
 58: volume c_glusterfs-open-behind
 59:     type performance/open-behind
 60:     subvolumes c_glusterfs-quick-read
 61: end-volume
 62:  
 63: volume c_glusterfs-md-cache
 64:     type performance/md-cache
 65:     option cache-posix-acl true
 66:     subvolumes c_glusterfs-open-behind
 67: end-volume
 68:  
 69: volume c_glusterfs
 70:     type debug/io-stats
 71:     option latency-measurement off
 72:     option count-fop-hits off
 73:     subvolumes c_glusterfs-md-cache
 74: end-volume
 75:  
 76: volume posix-acl-autoload
 77:     type system/posix-acl
 78:     subvolumes c_glusterfs
 79: end-volume
 80:  
 81: volume meta-autoload
 82:     type meta
 83:     subvolumes posix-acl-autoload
 84: end-volume
 85:  
+------------------------------------------------------------------------------+
[2016-10-31 04:04:08.062941] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-c_glusterfs-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-10-31 04:04:08.063367] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-10: changing port to 49309 (from 0)
[2016-10-31 04:04:08.063543] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-c_glusterfs-client-9: Connected to c_glusterfs-client-9, attached to remote volume '/opt/lvmdir/c2/brick'.
[2016-10-31 04:04:08.063576] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-c_glusterfs-client-9: Server and Client lk-version numbers are not same, reopening the fds
[2016-10-31 04:04:08.063716] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-c_glusterfs-replicate-0: Subvolume 'c_glusterfs-client-9' came back up; going online.
[2016-10-31 04:04:08.063785] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-c_glusterfs-client-9: Server lk version = 1
[2016-10-31 04:04:08.064659] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-c_glusterfs-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-10-31 04:04:08.065344] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-c_glusterfs-client-10: Connected to c_glusterfs-client-10, attached to remote volume '/opt/lvmdir/c2/brick'.
[2016-10-31 04:04:08.065374] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-c_glusterfs-client-10: Server and Client lk-version numbers are not same, reopening the fds
[2016-10-31 04:04:08.074629] I [fuse-bridge.c:5137:fuse_graph_setup] 0-fuse: switched to graph 0
[2016-10-31 04:04:08.074884] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-c_glusterfs-client-10: Server lk version = 1
[2016-10-31 04:04:08.075006] I [fuse-bridge.c:4030:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22
[2016-10-31 04:04:08.077023] I [MSGID: 108031] [afr-common.c:1782:afr_local_discovery_cbk] 0-c_glusterfs-replicate-0: selecting local read_child c_glusterfs-client-9


[2016-10-31 04:04:15.908933] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] 
                                                      -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] 
                                                      -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] 
                                                      ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]




[2016-10-31 04:04:15.909054] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.913787] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.913885] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.923030] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.923135] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.944227] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.944343] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.949661] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.949766] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.954882] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:15.954982] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:15.960768] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
...
[2016-10-31 04:04:16.107262] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 199e1862-7e74-4b13-8c42-c4cea7e9f2fc
[2016-10-31 04:04:16.111886] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 199e1862-7e74-4b13-8c42-c4cea7e9f2fc. source=0 sinks=1 
[2016-10-31 04:04:16.114518] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.114620] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.122321] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 014e2eca-bb61-4939-9a4a-9a1af4351178
[2016-10-31 04:04:16.126960] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 014e2eca-bb61-4939-9a4a-9a1af4351178. source=0 sinks=1 
[2016-10-31 04:04:16.128101] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.128202] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.134165] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 60b5380b-0cea-423d-a1af-d3064968cc54
[2016-10-31 04:04:16.139036] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 60b5380b-0cea-423d-a1af-d3064968cc54. source=0 sinks=1 
[2016-10-31 04:04:16.141317] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.141432] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.147456] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on accef130-4cf5-4171-be08-2089e7ba1ca5
[2016-10-31 04:04:16.152572] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on accef130-4cf5-4171-be08-2089e7ba1ca5. source=0 sinks=1 
[2016-10-31 04:04:16.153633] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.153744] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
[2016-10-31 04:04:16.159954] I [MSGID: 108026] [afr-self-heal-metadata.c:56:__afr_selfheal_metadata_do] 0-c_glusterfs-replicate-0: performing metadata selfheal on 9c666f31-f3d7-40c6-b17b-0f41768ddea4
[2016-10-31 04:04:16.164773] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-c_glusterfs-replicate-0: Completed metadata selfheal on 9c666f31-f3d7-40c6-b17b-0f41768ddea4. source=0 sinks=1 
[2016-10-31 04:04:16.167219] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15bb8) [0x3fff86a5b8e8] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_access [Invalid argument]
[2016-10-31 04:04:16.167323] I [dict.c:473:dict_get] (-->/usr/lib64/glusterfs/3.7.6/xlator/debug/io-stats.so(io_stats_lookup_cbk-0x1d7dc) [0x3fff86a85574] -->/usr/lib64/glusterfs/3.7.6/xlator/system/posix-acl.so(posix_acl_lookup_cbk-0x15b5c) [0x3fff86a5b944] -->/usr/lib64/libglusterfs.so.0(dict_get-0xc10f4) [0x3fff8ab6dc2c] ) 0-dict: !this || key=system.posix_acl_default [Invalid argument]
...




[2016-10-31 04:06:03.627345] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] 
                                                                     ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) 
                                                                    op(FINODELK(30)) called at 2016-10-31 04:06:03.626033 (xid=0x7f5e)










[2016-10-31 04:06:03.627395] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]






[2016-10-31 04:06:03.628381] I [socket.c:3308:socket_submit_request] 0-c_glusterfs-client-9: not connected (priv->connected = 0)






[2016-10-31 04:06:03.628432] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f5f Program: GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport (c_glusterfs-client-9)








[2016-10-31 04:06:03.628466] E [MSGID: 114031] [client-rpc-fops.c:1673:client3_3_finodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]






[2016-10-31 04:06:03.628475] I [MSGID: 108019] [afr-lk-common.c:1086:afr_lock_blocking] 0-c_glusterfs-replicate-0: unable to lock on even one child
[2016-10-31 04:06:03.628539] I [MSGID: 108019] [afr-transaction.c:1224:afr_post_blocking_inodelk_cbk] 0-c_glusterfs-replicate-0: Blocking inodelks failed.




[2016-10-31 04:06:03.628630] W [fuse-bridge.c:1282:fuse_err_cbk] 0-glusterfs-fuse: 20790: FLUSH() ERR => -1 (Transport endpoint is not connected)








[2016-10-31 04:06:03.629149] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] 
                                                                      ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) 
                                                                     op(LOOKUP(27)) called at 2016-10-31 04:06:03.624346 (xid=0x7f5a)


[2016-10-31 04:06:03.629183] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-c_glusterfs-client-9: changing port to 49391 (from 0)


[2016-10-31 04:06:03.629210] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not connected]




[2016-10-31 04:06:03.629266] I [socket.c:3308:socket_submit_request] 0-c_glusterfs-client-9: not connected (priv->connected = 255)




[2016-10-31 04:06:03.629277] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (gfid = b0e5a94e-a432-4dce-b86f-a551555780a2). Holes=1 overlaps=0




[2016-10-31 04:06:03.629293] W [rpc-clnt.c:1586:rpc_clnt_submit] 0-c_glusterfs-client-9: failed to submit rpc-request (XID: 0x7f62 Program: GlusterFS 3.3, ProgVers: 330, Proc: 41) to rpc-transport (c_glusterfs-client-9)






[2016-10-31 04:06:03.629333] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: b0e5a94e-a432-4dce-b86f-a551555780a2: failed to resolve (Transport endpoint is not connected)




[2016-10-31 04:06:03.629363] W [fuse-bridge.c:3385:fuse_getxattr_resume] 0-glusterfs-fuse: 20784: GETXATTR b0e5a94e-a432-4dce-b86f-a551555780a2/70366685109140 (system.posix_acl_access) resolution failed






[2016-10-31 04:06:03.629729] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (
                                                                     --> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (
                                                                     --> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ))))) 
                                                                     0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) 
                                                                     op(LOOKUP(27)) called at 2016-10-31 04:06:03.624568 (xid=0x7f5b)










[2016-10-31 04:06:03.629779] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /pmd/2 (fa866805-ab22-4afe-8acb-528d15ce4117) [Transport endpoint is not connected]




[2016-10-31 04:06:03.629784] E [MSGID: 114031] [client-rpc-fops.c:2883:client3_3_opendir_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews (b0e5a94e-a432-4dce-b86f-a551555780a2) [Transport endpoint is not connected]






[2016-10-31 04:06:03.629836] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /pmd/2 (gfid = fa866805-ab22-4afe-8acb-528d15ce4117). Holes=1 overlaps=0






[2016-10-31 04:06:03.629857] W [fuse-bridge.c:980:fuse_fd_cbk] 0-glusterfs-fuse: 20794: OPENDIR() /loadmodules_norepl/CXC1725605_P93A001/cello/emasviews => -1 (Transport endpoint is not connected)






[2016-10-31 04:06:03.629896] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: fa866805-ab22-4afe-8acb-528d15ce4117: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.629990] E [fuse-bridge.c:787:fuse_getattr_resume] 0-glusterfs-fuse: 20785: GETATTR 70366685017568 (fa866805-ab22-4afe-8acb-528d15ce4117) resolution failed
[2016-10-31 04:06:03.630164] W [defaults.c:2212:default_releasedir] (-->/usr/lib64/glusterfs/3.7.6/xlator/protocol/client.so(client_local_wipe-0x42d64) [0x3fff86c9bc5c] -->/usr/lib64/libglusterfs.so.0(fd_unref-0x7ba98) [0x3fff8abb7ad0] -->/usr/lib64/libglusterfs.so.0(default_releasedir-0x9d6d8) [0x3fff8ab93c58] ) 0-fuse: xlator does not implement releasedir_cbk
[2016-10-31 04:06:03.630367] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2016-10-31 04:06:03.625675 (xid=0x7f5c)
[2016-10-31 04:06:03.630368] W [MSGID: 114031] [client-rpc-fops.c:1569:client3_3_fstat_cbk] 0-c_glusterfs-client-9: remote operation failed [Transport endpoint is not connected]
[2016-10-31 04:06:03.630497] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f) [Transport endpoint is not connected]
[2016-10-31 04:06:03.630871] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-c_glusterfs-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-10-31 04:06:03.631488] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-c_glusterfs-client-9: Connected to c_glusterfs-client-9, attached to remote volume '/opt/lvmdir/c2/brick'.
[2016-10-31 04:06:03.631523] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-c_glusterfs-client-9: Server and Client lk-version numbers are not same, reopening the fds
[2016-10-31 04:06:03.631555] I [MSGID: 114042] [client-handshake.c:1056:client_post_handshake] 0-c_glusterfs-client-9: 6 fds open - Delaying child_up until they are re-opened
[2016-10-31 04:06:03.631475] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/ethmac.cfg (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]








[2016-10-31 04:06:03.632256] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:00000000-0000-0000-0000-000000000001> succeeded (fd = 1)
[2016-10-31 04:06:03.632728] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:257add5a-9b87-4014-8a3a-09dd0a699eec> succeeded (fd = 2)
[2016-10-31 04:06:03.632790] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:6fd47fe9-a566-47c7-8df7-e7a5e379101f> succeeded (fd = 3)
[2016-10-31 04:06:03.632847] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:fa866805-ab22-4afe-8acb-528d15ce4117> succeeded (fd = 4)
[2016-10-31 04:06:03.632905] I [MSGID: 114060] [client-handshake.c:820:client3_3_reopendir_cbk] 0-c_glusterfs-client-9: reopendir on <gfid:139f5034-5b42-4fb8-9372-6dd7cac1e674> succeeded (fd = 5)










[2016-10-31 04:06:03.632935] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-c_glusterfs-client-9: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2016-10-31 04:06:03.633028] I [MSGID: 108005] [afr-common.c:3841:afr_notify] 0-c_glusterfs-replicate-0: Subvolume 'c_glusterfs-client-9' came back up; going online.
[2016-10-31 04:06:03.633386] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-c_glusterfs-client-9: Server lk version = 1






[2016-10-31 04:06:03.634579] W [fuse-bridge.c:758:fuse_attr_cbk] 0-glusterfs-fuse: 20795: FSTAT() /configuration/oamrdncy.cfg => -1 (Transport endpoint is not connected)






[2016-10-31 04:06:03.634889] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xb5c80)[0x3fff8ab79f58] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind-0x1b7a0)[0x3fff8ab1dc90] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy-0x1b638)[0x3fff8ab1de10] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup-0x19af8)[0x3fff8ab1fb18] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0x18e68)[0x3fff8ab20808] ))))) 0-c_glusterfs-client-9: forced unwinding frame type(GlusterFS 3.3) op(XATTROP(33)) called at 2016-10-31 04:06:03.625716 (xid=0x7f5d)






[2016-10-31 04:06:03.634950] W [MSGID: 114031] [client-rpc-fops.c:1845:client3_3_xattrop_cbk] 0-c_glusterfs-client-9: remote operation failed. Path: /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f)






[2016-10-31 04:06:03.635134] W [MSGID: 108008] [afr-read-txn.c:250:afr_read_txn] 0-c_glusterfs-replicate-0: Unreadable subvolume -1 found with event generation 3 for gfid 200b0bb2-d903-4924-b025-262d805c0f2f. (Possible split-brain)




[2016-10-31 04:06:03.635364] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-c_glusterfs-client-9: disconnected from c_glusterfs-client-9. Client process will keep trying to connect to glusterd until brick's port is available






[2016-10-31 04:06:03.635420] E [MSGID: 108006] [afr-common.c:3880:afr_notify] 0-c_glusterfs-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.










[2016-10-31 04:06:03.635762] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.635812] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /pmd/15 (gfid = 52a8bce8-00ac-48a7-8abf-9a3fdf9a498b). Holes=1 overlaps=0
[2016-10-31 04:06:03.635858] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: 52a8bce8-00ac-48a7-8abf-9a3fdf9a498b: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.635885] E [fuse-bridge.c:787:fuse_getattr_resume] 0-glusterfs-fuse: 20802: GETATTR 70366685021780 (52a8bce8-00ac-48a7-8abf-9a3fdf9a498b) resolution failed
[2016-10-31 04:06:03.638853] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.639020] E [MSGID: 108008] [afr-read-txn.c:89:afr_read_txn_refresh_done] 0-c_glusterfs-replicate-0: Failing FSTAT on gfid 200b0bb2-d903-4924-b025-262d805c0f2f: split-brain observed. [Input/output error]
[2016-10-31 04:06:03.640564] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.641160] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20808: LOOKUP() /license => -1 (Transport endpoint is not connected)
The message "I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up" repeated 6 times between [2016-10-31 04:06:03.640564] and [2016-10-31 04:06:03.641444]
[2016-10-31 04:06:03.641484] E [MSGID: 114031] [client-rpc-fops.c:1621:client3_3_inodelk_cbk] 0-c_glusterfs-client-9: remote operation failed [Invalid argument]
[2016-10-31 04:06:03.641562] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.641573] E [MSGID: 108010] [afr-lk-common.c:665:afr_unlock_inodelk_cbk] 0-c_glusterfs-replicate-0: path=/configuration/oamrdncy.cfg gfid=200b0bb2-d903-4924-b025-262d805c0f2f: unlock failed on subvolume c_glusterfs-client-9 with lock owner 6cd0c189ff3f0000 [Invalid argument]
[2016-10-31 04:06:03.641655] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up










[2016-10-31 04:06:03.641716] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20809: LOOKUP() /lost+found => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.642387] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20813: LOOKUP() /lost+found => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.647166] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20815: LOOKUP() /license => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.647647] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20816: LOOKUP() /tmp => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.648023] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20817: LOOKUP() /lost+found => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.653160] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20819: LOOKUP() /tmp => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.653635] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20820: LOOKUP() /logfiles => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.653992] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20821: LOOKUP() /security => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.654333] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20822: LOOKUP() /logfiles => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.660036] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20824: LOOKUP() /security => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.660675] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20825: LOOKUP() /logfiles => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.661018] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20826: LOOKUP() /up => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.665422] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20828: LOOKUP() /up => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.665886] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20829: LOOKUP() /usr => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.666242] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20830: LOOKUP() /xb => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.666790] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20832: LOOKUP() /xb => -1 (Transport endpoint is not connected)








The message "I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up" repeated 61 times between [2016-10-31 04:06:03.641655] and [2016-10-31 04:06:03.670371]
[2016-10-31 04:06:03.670392] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-c_glusterfs-dht: Found anomalies in /pmd/2/000301 (gfid = 7af60568-eace-40f1-a94f-5ba92a3afadf). Holes=1 overlaps=0
[2016-10-31 04:06:03.670499] W [fuse-resolve.c:149:fuse_resolve_gfid_cbk] 0-fuse: 7af60568-eace-40f1-a94f-5ba92a3afadf: failed to resolve (Transport endpoint is not connected)
[2016-10-31 04:06:03.670529] W [fuse-bridge.c:2517:fuse_opendir_resume] 0-glusterfs-fuse: 20834: OPENDIR (7af60568-eace-40f1-a94f-5ba92a3afadf) resolution failed
[2016-10-31 04:06:03.675300] I [MSGID: 108006] [afr-common.c:4008:afr_local_init] 0-c_glusterfs-replicate-0: no subvolumes up
[2016-10-31 04:06:03.675553] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20835: LOOKUP() /usr => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.675990] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20836: LOOKUP() /xb => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.676348] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20837: LOOKUP() /node_id => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.680652] W [fuse-bridge.c:462:fuse_entry_cbk] 0-glusterfs-fuse: 20839: LOOKUP() /node_id => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.681150] W [fuse-bridge.c:980:fuse_fd_cbk] 0-glusterfs-fuse: 20842: OPENDIR() /configuration => -1 (Transport endpoint is not connected)
[2016-10-31 04:06:03.681510] W [defaults.c:2212:default_releasedir] (-->/usr/lib64/glusterfs/3.7.6/xlator/cluster/distribute.so(dht_local_wipe-0x70e58) [0x3fff86b88f58] -->/usr/lib64/libglusterfs.so.0(fd_unref-0x7ba98) [0x3fff8abb7ad0] -->/usr/lib64/libglusterfs.so.0(default_releasedir-0x9d6d8) [0x3fff8ab93c58] ) 0-fuse: xlator does not implement releasedir_cbk


opt-lvmdir-c2-brick.log(server log)


[2016-10-31 04:03:41.554325] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.6 (args: /usr/sbin/glusterfsd -s 10.32.1.144 --volfile-id c_glusterfs.10.32.1.144.opt-lvmdir-c2-brick -p /system/glusterd/vols/c_glusterfs/run/10.32.1.144-opt-lvmdir-c2-brick.pid -S /var/run/gluster/697c0e4a16ebc734cd06fd9150723005.socket --brick-name /opt/lvmdir/c2/brick -l /var/log/glusterfs/bricks/opt-lvmdir-c2-brick.log --xlator-option *-posix.glusterd-uuid=5818a372-d517-41af-bd2f-0fce648773df --brick-port 49391 --xlator-option c_glusterfs-server.listen-port=49391)
[2016-10-31 04:03:41.571442] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-10-31 04:03:41.588390] I [graph.c:269:gf_add_cmdline_options] 0-c_glusterfs-server: adding option 'listen-port' for volume 'c_glusterfs-server' with value '49391'
[2016-10-31 04:03:41.588440] I [graph.c:269:gf_add_cmdline_options] 0-c_glusterfs-posix: adding option 'glusterd-uuid' for volume 'c_glusterfs-posix' with value '5818a372-d517-41af-bd2f-0fce648773df'
[2016-10-31 04:03:41.588815] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-10-31 04:03:41.588870] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/opt/lvmdir/c2/brick: skip format check for non-addr auth option auth.login./opt/lvmdir/c2/brick.allow
[2016-10-31 04:03:41.588907] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/opt/lvmdir/c2/brick: skip format check for non-addr auth option auth.login.5c5afe17-02ce-4b9b-aef4-e66efa1c6132.password
[2016-10-31 04:03:41.590577] I [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2016-10-31 04:03:41.590773] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-c_glusterfs-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2016-10-31 04:03:41.593929] I [trash.c:2366:init] 0-c_glusterfs-trash: no option specified for 'eliminate', using NULL
[2016-10-31 04:03:41.595494] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
[2016-10-31 04:03:41.595528] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-server: option 'rpc-auth.auth-unix' is not recognized
[2016-10-31 04:03:41.595556] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-server: option 'rpc-auth.auth-null' is not recognized
[2016-10-31 04:03:41.595619] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-quota: option 'timeout' is not recognized
[2016-10-31 04:03:41.595647] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-marker: option 'quota-version' is not recognized
[2016-10-31 04:03:41.595677] W [graph.c:357:_log_if_unknown_option] 0-c_glusterfs-trash: option 'brick-path' is not recognized
[2016-10-31 04:03:41.605103] W [MSGID: 113036] [posix.c:2018:posix_rename] 0-c_glusterfs-posix: found directory at /opt/lvmdir/c2/brick/.trashcan/ while expecting ENOENT [File exists]
Final graph:
+------------------------------------------------------------------------------+
  1: volume c_glusterfs-posix
  2:     type storage/posix
  3:     option glusterd-uuid 5818a372-d517-41af-bd2f-0fce648773df
  4:     option directory /opt/lvmdir/c2/brick
  5:     option volume-id 0a50e4e5-ecfc-432c-88d9-2083455e4baf
  6: end-volume
  7:  
  8: volume c_glusterfs-trash
  9:     type features/trash
 10:     option trash-dir .trashcan
 11:     option brick-path /opt/lvmdir/c2/brick
 12:     option trash-internal-op off
 13:     subvolumes c_glusterfs-posix
 14: end-volume
 15:  
 16: volume c_glusterfs-changelog
 17:     type features/changelog
 18:     option changelog-brick /opt/lvmdir/c2/brick
 19:     option changelog-dir /opt/lvmdir/c2/brick/.glusterfs/changelogs
 20:     option changelog-barrier-timeout 120
 21:     subvolumes c_glusterfs-trash
 22: end-volume
 23:  
 24: volume c_glusterfs-bitrot-stub
 25:     type features/bitrot-stub
 26:     option export /opt/lvmdir/c2/brick
 27:     subvolumes c_glusterfs-changelog
 28: end-volume
 29:  
 30: volume c_glusterfs-access-control
 31:     type features/access-control
 32:     subvolumes c_glusterfs-bitrot-stub
 33: end-volume
 34:  
 35: volume c_glusterfs-locks
 36:     type features/locks
 37:     subvolumes c_glusterfs-access-control
 38: end-volume
 39:  
 40: volume c_glusterfs-upcall
 41:     type features/upcall
 42:     option cache-invalidation off
 43:     subvolumes c_glusterfs-locks
 44: end-volume
 45:  
 46: volume c_glusterfs-io-threads
 47:     type performance/io-threads
 48:     subvolumes c_glusterfs-upcall
 49: end-volume
 50:  
 51: volume c_glusterfs-marker
 52:     type features/marker
 53:     option volume-uuid 0a50e4e5-ecfc-432c-88d9-2083455e4baf
 54:     option timestamp-file /system/glusterd/vols/c_glusterfs/marker.tstamp
 55:     option quota-version 0
 56:     option xtime off
 57:     option gsync-force-xtime off
 58:     option quota off
 59:     option inode-quota off
 60:     subvolumes c_glusterfs-io-threads
 61: end-volume
 62:  
 63: volume c_glusterfs-barrier
 64:     type features/barrier
 65:     option barrier disable
 66:     option barrier-timeout 120
 67:     subvolumes c_glusterfs-marker
 68: end-volume
 69:  
 70: volume c_glusterfs-index
 71:     type features/index
 72:     option index-base /opt/lvmdir/c2/brick/.glusterfs/indices
 73:     subvolumes c_glusterfs-barrier
 74: end-volume
 75:  
 76: volume c_glusterfs-quota
 77:     type features/quota
 78:     option volume-uuid c_glusterfs
 79:     option server-quota off
 80:     option timeout 0
 81:     option deem-statfs off
 82:     subvolumes c_glusterfs-index
 83: end-volume
 84:  
 85: volume c_glusterfs-worm
 86:     type features/worm
 87:     option worm off
 88:     subvolumes c_glusterfs-quota
 89: end-volume
 90:  
 91: volume c_glusterfs-read-only
 92:     type features/read-only
 93:     option read-only off
 94:     subvolumes c_glusterfs-worm
 95: end-volume
 96:  
 97: volume /opt/lvmdir/c2/brick
 98:     type debug/io-stats
 99:     option latency-measurement off
100:     option count-fop-hits off
101:     subvolumes c_glusterfs-read-only
102: end-volume
103:  
104: volume c_glusterfs-server
105:     type protocol/server
106:     option transport.socket.listen-port 49391
107:     option rpc-auth.auth-glusterfs on
108:     option rpc-auth.auth-unix on
109:     option rpc-auth.auth-null on
110:     option rpc-auth-allow-insecure on
111:     option transport-type tcp
112:     option auth.login./opt/lvmdir/c2/brick.allow 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
113:     option auth.login.5c5afe17-02ce-4b9b-aef4-e66efa1c6132.password ade893cc-19dd-4d67-ae8e-cb929ef4e770
114:     option auth.addr./opt/lvmdir/c2/brick.allow *
115:     option transport.tcp-user-timeout 4
116:     subvolumes /opt/lvmdir/c2/brick
117: end-volume
118:  
+------------------------------------------------------------------------------+
[2016-10-31 04:03:43.191850] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:03:43.191927] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-3365-2016/10/31-04:01:45:605337-c_glusterfs-client-9-0-1 (version: 3.7.6)
[2016-10-31 04:03:45.556522] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:03:45.556581] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-1599-2016/10/31-04:03:41:530232-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:04:07.831958] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:04:07.832023] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-1603-2016/10/31-04:01:41:508960-c_glusterfs-client-9-0-1 (version: 3.7.6)
[2016-10-31 04:04:07.974625] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:04:07.974688] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-8222-2016/10/31-04:04:07:930474-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:04:08.063317] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:04:08.063359] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:04:10.032591] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-8222-2016/10/31-04:04:07:930474-c_glusterfs-client-9-0-0
[2016-10-31 04:04:10.032672] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-8222-2016/10/31-04:04:07:930474-c_glusterfs-client-9-0-0
[2016-10-31 04:05:04.530186] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-3365-2016/10/31-04:01:45:605337-c_glusterfs-client-9-0-1
[2016-10-31 04:05:04.530267] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-c_glusterfs-server: releasing lock on 60a17763-95fd-4162-8402-9651dfbe9ec4 held by {client=0x3fff7c002990, pid=0 lk-owner=a87b0a7cff3f0000}
[2016-10-31 04:05:04.530306] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /logfiles/systemlog/syslog
[2016-10-31 04:05:04.530375] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-3365-2016/10/31-04:01:45:605337-c_glusterfs-client-9-0-1
[2016-10-31 04:05:05.760618] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 000300-1603-2016/10/31-04:01:41:508960-c_glusterfs-client-9-0-1
[2016-10-31 04:05:05.760693] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 000300-1603-2016/10/31-04:01:41:508960-c_glusterfs-client-9-0-1
[2016-10-31 04:05:57.341892] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:05:57.341928] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-1604-2016/10/31-04:05:53:637848-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:01.291380] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:01.291441] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:01.403916] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:01.403963] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 000300-3292-2016/10/31-04:05:57:685417-c_glusterfs-client-9-0-0 (version: 3.7.6)
[2016-10-31 04:06:03.631296] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 5c5afe17-02ce-4b9b-aef4-e66efa1c6132
[2016-10-31 04:06:03.631332] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-c_glusterfs-server: accepted client from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-1 (version: 3.7.6)
[2016-10-31 04:06:03.634762] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0
[2016-10-31 04:06:03.634945] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-c_glusterfs-server: releasing lock on 200b0bb2-d903-4924-b025-262d805c0f2f held by {client=0x3fff7c026520, pid=6232 lk-owner=6cd0c189ff3f0000}
[2016-10-31 04:06:03.634987] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /pmd/2
[2016-10-31 04:06:03.635035] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /configuration/oamrdncy.cfg
[2016-10-31 04:06:03.635349] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /
[2016-10-31 04:06:03.635414] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /logfiles/systemlog/syslog
[2016-10-31 04:06:03.635468] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl/CXC1725605_P93A001
[2016-10-31 04:06:03.635554] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl
[2016-10-31 04:06:03.635889] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-c_glusterfs-server: fd cleanup on /loadmodules_norepl/CXC1725605_P93A001/cello
[2016-10-31 04:06:03.640329] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 002500-4291-2016/10/31-04:04:08:14796-c_glusterfs-client-9-0-0
[2016-10-31 04:06:03.641143] E [inodelk.c:309:__inode_unlock_lock] 0-c_glusterfs-locks:  Matching lock not found for unlock 0-9223372036854775807, by 6cd0c189ff3f0000 on 0x3fff840fd9a0
[2016-10-31 04:06:03.641200] E [MSGID: 115053] [server-rpc-fops.c:275:server_inodelk_cbk] 0-c_glusterfs-server: 32623: INODELK /configuration/oamrdncy.cfg (200b0bb2-d903-4924-b025-262d805c0f2f) ==> (Invalid argument) [Invalid argument]
[2016-10-31 04:06:06.270314] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-c_glusterfs-server: disconnecting connection from 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0
[2016-10-31 04:06:06.270449] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-c_glusterfs-server: Shutting down connection 002500-6738-2016/10/31-04:06:01:245058-c_glusterfs-client-9-0-0






Thanks,
Xin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161121/e0dad62e/attachment.html>

More information about the Gluster-users mailing list