[Gluster-users] I give up on Gluster with error...
NOC - IP·RED
noc at ipred.com.ar
Wed Aug 19 21:57:14 UTC 2015
Dear Group.
Many days ago I had to restart one of the servers and after that, it doesn’t
heal anymore.
I tried lots of thinks, no answer on Internet and the server stills down by
now.
Could please lend me a hand.
Logs reports [client-handshake.c:1760:client_query_portmap_cbk]
0-data-cotelser-client-3: failed to get the port number for remote
subvolume. Please run 'gluster volume status' on server to see if brick
process is running.
But brick processes run okay without any problems.
root at ipred01:/var/log/glusterfs# gluster volume status
Status of volume: data-cotelser
Gluster process Port Online Pid
----------------------------------------------------------------------------
--
Brick cotelser01:/data/sdb1/data-cotelser 49152 Y 2188
Brick ipred01:/data/sdb1/data-cotelser 49154 Y 2138
Brick cotelser01:/data/sdc1/data-cotelser 49153 Y 2201
Brick ipred01:/data/sdc1/data-cotelser 49155 Y 2152
NFS Server on localhost 2049 Y 2162
Self-heal Daemon on localhost N/A Y 2167
NFS Server on cotelser01 2049 Y 9893
Self-heal Daemon on cotelser01 N/A Y 9900
Task Status of Volume data-cotelser
----------------------------------------------------------------------------
--
There are no active volume tasks
Status of volume: data-ipred
Gluster process Port Online Pid
----------------------------------------------------------------------------
--
Brick cotelser01:/data/sdb1/data-ipred 49154 Y 2202
Brick ipred01:/data/sdb1/data-ipred 49156 Y 2140
Brick cotelser01:/data/sdc1/data-ipred 49155 Y 2212
Brick ipred01:/data/sdc1/data-ipred 49157 Y 2142
NFS Server on localhost 2049 Y 2162
Self-heal Daemon on localhost N/A Y 2167
NFS Server on cotelser01 2049 Y 9893
Self-heal Daemon on cotelser01 N/A Y 9900
Task Status of Volume data-ipred
----------------------------------------------------------------------------
--
There are no active volume tasks
root at ipred01:/var/log/glusterfs#
LOG glustershd.log (bellow)
[2015-08-19 21:19:38.200445] I [glusterfsd.c:1959:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2
(/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p
/var/lib/glusterd/glustershd/run/glustershd.pid -l
/var/log/glusterfs/glustershd.log -S
/var/run/17c62c5aeddf9c73dfed4aa2d7407aaa.socket --xlator-option
*replicate*.node-uuid=c53ad0c0-3940-443a-9687-597f6a14b783)
[2015-08-19 21:19:38.201761] I [socket.c:3578:socket_init]
0-socket.glusterfsd: SSL support is NOT enabled
[2015-08-19 21:19:38.201778] I [socket.c:3593:socket_init]
0-socket.glusterfsd: using system polling thread
[2015-08-19 21:19:38.201873] I [socket.c:3578:socket_init] 0-glusterfs: SSL
support is NOT enabled
[2015-08-19 21:19:38.201882] I [socket.c:3593:socket_init] 0-glusterfs:
using system polling thread
[2015-08-19 21:19:38.679468] I [graph.c:254:gf_add_cmdline_options]
0-data-cotelser-replicate-1: adding option 'node-uuid' for volume
'data-cotelser-replicate-1' with value
'c53ad0c0-3940-443a-9687-597f6a14b783'
[2015-08-19 21:19:38.679484] I [graph.c:254:gf_add_cmdline_options]
0-data-cotelser-replicate-0: adding option 'node-uuid' for volume
'data-cotelser-replicate-0' with value
'c53ad0c0-3940-443a-9687-597f6a14b783'
[2015-08-19 21:19:38.679490] I [graph.c:254:gf_add_cmdline_options]
0-data-ipred-replicate-1: adding option 'node-uuid' for volume
'data-ipred-replicate-1' with value 'c53ad0c0-3940-443a-9687-597f6a14b783'
[2015-08-19 21:19:38.679500] I [graph.c:254:gf_add_cmdline_options]
0-data-ipred-replicate-0: adding option 'node-uuid' for volume
'data-ipred-replicate-0' with value 'c53ad0c0-3940-443a-9687-597f6a14b783'
[2015-08-19 21:19:38.682874] I [socket.c:3578:socket_init]
0-data-cotelser-client-3: SSL support is NOT enabled
[2015-08-19 21:19:38.682904] I [socket.c:3593:socket_init]
0-data-cotelser-client-3: using system polling thread
[2015-08-19 21:19:38.683110] I [socket.c:3578:socket_init]
0-data-cotelser-client-2: SSL support is NOT enabled
[2015-08-19 21:19:38.683117] I [socket.c:3593:socket_init]
0-data-cotelser-client-2: using system polling thread
[2015-08-19 21:19:38.683314] I [socket.c:3578:socket_init]
0-data-cotelser-client-1: SSL support is NOT enabled
[2015-08-19 21:19:38.683321] I [socket.c:3593:socket_init]
0-data-cotelser-client-1: using system polling thread
[2015-08-19 21:19:38.683516] I [socket.c:3578:socket_init]
0-data-cotelser-client-0: SSL support is NOT enabled
[2015-08-19 21:19:38.683523] I [socket.c:3593:socket_init]
0-data-cotelser-client-0: using system polling thread
[2015-08-19 21:19:38.686272] I [socket.c:3578:socket_init]
0-data-ipred-client-3: SSL support is NOT enabled
[2015-08-19 21:19:38.686295] I [socket.c:3593:socket_init]
0-data-ipred-client-3: using system polling thread
[2015-08-19 21:19:38.686446] I [socket.c:3578:socket_init]
0-data-ipred-client-2: SSL support is NOT enabled
[2015-08-19 21:19:38.686451] I [socket.c:3593:socket_init]
0-data-ipred-client-2: using system polling thread
[2015-08-19 21:19:38.686599] I [socket.c:3578:socket_init]
0-data-ipred-client-1: SSL support is NOT enabled
[2015-08-19 21:19:38.686604] I [socket.c:3593:socket_init]
0-data-ipred-client-1: using system polling thread
[2015-08-19 21:19:38.686750] I [socket.c:3578:socket_init]
0-data-ipred-client-0: SSL support is NOT enabled
[2015-08-19 21:19:38.686755] I [socket.c:3593:socket_init]
0-data-ipred-client-0: using system polling thread
[2015-08-19 21:19:38.686789] I [client.c:2294:notify]
0-data-cotelser-client-0: parent translators are ready, attempting connect
on transport
[2015-08-19 21:19:38.687080] I [client.c:2294:notify]
0-data-cotelser-client-1: parent translators are ready, attempting connect
on transport
[2015-08-19 21:19:38.687269] I [client.c:2294:notify]
0-data-cotelser-client-2: parent translators are ready, attempting connect
on transport
[2015-08-19 21:19:38.687427] I [client.c:2294:notify]
0-data-cotelser-client-3: parent translators are ready, attempting connect
on transport
[2015-08-19 21:19:38.687591] I [client.c:2294:notify] 0-data-ipred-client-0:
parent translators are ready, attempting connect on transport
[2015-08-19 21:19:38.687755] I [client.c:2294:notify] 0-data-ipred-client-1:
parent translators are ready, attempting connect on transport
[2015-08-19 21:19:38.687916] I [client.c:2294:notify] 0-data-ipred-client-2:
parent translators are ready, attempting connect on transport
[2015-08-19 21:19:38.688066] I [client.c:2294:notify] 0-data-ipred-client-3:
parent translators are ready, attempting connect on transport
Final graph:
+---------------------------------------------------------------------------
---+
1: volume data-cotelser-client-0
2: type protocol/client
3: option remote-host cotelser01
4: option remote-subvolume /data/sdb1/data-cotelser
5: option transport-type socket
6: option username 691af945-5b60-4c66-9246-1834143ace9b
7: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c
8: end-volume
9:
10: volume data-cotelser-client-1
11: type protocol/client
12: option remote-host ipred01
13: option remote-subvolume /data/sdb1/data-cotelser
14: option transport-type socket
15: option username 691af945-5b60-4c66-9246-1834143ace9b
16: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c
17: end-volume
18:
19: volume data-cotelser-replicate-0
20: type cluster/replicate
21: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783
22: option background-self-heal-count 0
23: option metadata-self-heal on
24: option data-self-heal on
25: option entry-self-heal on
26: option self-heal-daemon on
27: option iam-self-heal-daemon yes
28: subvolumes data-cotelser-client-0 data-cotelser-client-1
29: end-volume
30:
31: volume data-cotelser-client-2
32: type protocol/client
33: option remote-host cotelser01
34: option remote-subvolume /data/sdc1/data-cotelser
35: option transport-type socket
36: option username 691af945-5b60-4c66-9246-1834143ace9b
37: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c
38: end-volume
39:
40: volume data-cotelser-client-3
41: type protocol/client
42: option remote-host ipred01
43: option remote-subvolume /data/sdc1/data-cotelser
44: option transport-type socket
45: option username 691af945-5b60-4c66-9246-1834143ace9b
46: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c
47: end-volume
48:
49: volume data-cotelser-replicate-1
50: type cluster/replicate
51: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783
52: option background-self-heal-count 0
53: option metadata-self-heal on
54: option data-self-heal on
55: option entry-self-heal on
56: option self-heal-daemon on
57: option iam-self-heal-daemon yes
58: subvolumes data-cotelser-client-2 data-cotelser-client-3
59: end-volume
60:
61: volume data-ipred-client-0
62: type protocol/client
63: option remote-host cotelser01
64: option remote-subvolume /data/sdb1/data-ipred
65: option transport-type socket
66: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1
67: option password c1ab7f12-fe61-43e4-af17-67906d09cea5
68: end-volume
69:
70: volume data-ipred-client-1
71: type protocol/client
72: option remote-host ipred01
73: option remote-subvolume /data/sdb1/data-ipred
74: option transport-type socket
75: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1
76: option password c1ab7f12-fe61-43e4-af17-67906d09cea5
77: end-volume
78:
79: volume data-ipred-replicate-0
80: type cluster/replicate
81: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783
82: option background-self-heal-count 0
83: option metadata-self-heal on
84: option data-self-heal on
85: option entry-self-heal on
86: option self-heal-daemon on
87: option iam-self-heal-daemon yes
88: subvolumes data-ipred-client-0 data-ipred-client-1
89: end-volume
90:
91: volume data-ipred-client-2
92: type protocol/client
93: option remote-host cotelser01
94: option remote-subvolume /data/sdc1/data-ipred
95: option transport-type socket
96: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1
97: option password c1ab7f12-fe61-43e4-af17-67906d09cea5
98: end-volume
99:
100: volume data-ipred-client-3
101: type protocol/client
102: option remote-host ipred01
103: option remote-subvolume /data/sdc1/data-ipred
104: option transport-type socket
105: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1
106: option password c1ab7f12-fe61-43e4-af17-67906d09cea5
107: end-volume
108:
109: volume data-ipred-replicate-1
110: type cluster/replicate
111: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783
112: option background-self-heal-count 0
113: option metadata-self-heal on
114: option data-self-heal on
115: option entry-self-heal on
116: option self-heal-daemon on
117: option iam-self-heal-daemon yes
118: subvolumes data-ipred-client-2 data-ipred-client-3
119: end-volume
120:
121: volume glustershd
122: type debug/io-stats
123: subvolumes data-cotelser-replicate-0 data-cotelser-replicate-1
data-ipred-replicate-0 data-ipred-replicate-1
124: end-volume
125:
+---------------------------------------------------------------------------
---+
[2015-08-19 21:19:38.688612] E
[client-handshake.c:1760:client_query_portmap_cbk] 0-data-cotelser-client-1:
failed to get the port number for remote subvolume. Please run 'gluster
volume status' on server to see if brick process is running.
[2015-08-19 21:19:38.688650] I [client.c:2229:client_rpc_notify]
0-data-cotelser-client-1: disconnected from 172.25.30.2:24007. Client
process will keep trying to connect to glusterd until brick's port is
available
[2015-08-19 21:19:38.688662] E
[client-handshake.c:1760:client_query_portmap_cbk] 0-data-ipred-client-3:
failed to get the port number for remote subvolume. Please run 'gluster
volume status' on server to see if brick process is running.
[2015-08-19 21:19:38.688693] E
[client-handshake.c:1760:client_query_portmap_cbk] 0-data-cotelser-client-3:
failed to get the port number for remote subvolume. Please run 'gluster
volume status' on server to see if brick process is running.
[2015-08-19 21:19:38.688715] I [client.c:2229:client_rpc_notify]
0-data-ipred-client-3: disconnected from 172.25.30.2:24007. Client process
will keep trying to connect to glusterd until brick's port is available
[2015-08-19 21:19:38.688724] I [client.c:2229:client_rpc_notify]
0-data-cotelser-client-3: disconnected from 172.25.30.2:24007. Client
process will keep trying to connect to glusterd until brick's port is
available
[2015-08-19 21:19:38.688732] E
[client-handshake.c:1760:client_query_portmap_cbk] 0-data-ipred-client-1:
failed to get the port number for remote subvolume. Please run 'gluster
volume status' on server to see if brick process is running.
[2015-08-19 21:19:38.688771] I [client.c:2229:client_rpc_notify]
0-data-ipred-client-1: disconnected from 172.25.30.2:24007. Client process
will keep trying to connect to glusterd until brick's port is available
[2015-08-19 21:19:38.688860] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-cotelser-client-0: changing port to 49152 (from 0)
[2015-08-19 21:19:38.689076] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-cotelser-client-2: changing port to 49153 (from 0)
[2015-08-19 21:19:38.689098] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-ipred-client-0: changing port to 49154 (from 0)
[2015-08-19 21:19:38.689110] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-ipred-client-2: changing port to 49155 (from 0)
[2015-08-19 21:19:38.689762] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-cotelser-client-2: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2015-08-19 21:19:38.689824] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-ipred-client-0: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2015-08-19 21:19:38.689852] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-cotelser-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2015-08-19 21:19:38.689887] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-ipred-client-2: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2015-08-19 21:19:38.690090] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-2:
Connected to 172.25.30.1:49153, attached to remote volume
'/data/sdc1/data-cotelser'.
[2015-08-19 21:19:38.690099] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-2:
Server and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:19:38.690121] I [afr-common.c:4131:afr_notify]
0-data-cotelser-replicate-1: Subvolume 'data-cotelser-client-2' came back
up; going online.
[2015-08-19 21:19:38.690193] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-0:
Connected to 172.25.30.1:49152, attached to remote volume
'/data/sdb1/data-cotelser'.
[2015-08-19 21:19:38.690204] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:19:38.690227] I [afr-common.c:4131:afr_notify]
0-data-cotelser-replicate-0: Subvolume 'data-cotelser-client-0' came back
up; going online.
[2015-08-19 21:19:38.690283] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-0:
Connected to 172.25.30.1:49154, attached to remote volume
'/data/sdb1/data-ipred'.
[2015-08-19 21:19:38.690296] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-0: Server
and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:19:38.690346] I [afr-common.c:4131:afr_notify]
0-data-ipred-replicate-0: Subvolume 'data-ipred-client-0' came back up;
going online.
[2015-08-19 21:19:38.690382] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-0:
Server lk version = 1
[2015-08-19 21:19:38.690399] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-2:
Connected to 172.25.30.1:49155, attached to remote volume
'/data/sdc1/data-ipred'.
[2015-08-19 21:19:38.690405] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-2: Server
and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:19:38.690433] I [afr-common.c:4131:afr_notify]
0-data-ipred-replicate-1: Subvolume 'data-ipred-client-2' came back up;
going online.
[2015-08-19 21:19:38.690486] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-2:
Server lk version = 1
[2015-08-19 21:19:38.690534] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-0:
Server lk version = 1
[2015-08-19 21:19:38.690582] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-2:
Server lk version = 1
[2015-08-19 21:19:42.204115] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-cotelser-client-1: changing port to 49154 (from 0)
[2015-08-19 21:19:42.204180] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-cotelser-client-3: changing port to 49155 (from 0)
[2015-08-19 21:19:42.204637] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-ipred-client-1: changing port to 49156 (from 0)
[2015-08-19 21:19:42.204869] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-ipred-client-3: changing port to 49157 (from 0)
[2015-08-19 21:19:42.205205] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-cotelser-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2015-08-19 21:19:42.205258] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-cotelser-client-3: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2015-08-19 21:19:42.205335] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-ipred-client-1: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2015-08-19 21:19:42.205384] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-1:
Connected to 172.25.30.2:49154, attached to remote volume
'/data/sdb1/data-cotelser'.
[2015-08-19 21:19:42.205396] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:19:42.205517] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-3:
Connected to 172.25.30.2:49155, attached to remote volume
'/data/sdc1/data-cotelser'.
[2015-08-19 21:19:42.205539] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-3:
Server and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:19:42.205652] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-3:
Server lk version = 1
[2015-08-19 21:19:42.205674] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-1:
Connected to 172.25.30.2:49156, attached to remote volume
'/data/sdb1/data-ipred'.
[2015-08-19 21:19:42.205686] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-1: Server
and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:19:42.205848] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-1:
Server lk version = 1
[2015-08-19 21:19:42.206392] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-1:
Server lk version = 1
[2015-08-19 21:19:42.206415] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-ipred-client-3: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2015-08-19 21:19:42.206605] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-3:
Connected to 172.25.30.2:49157, attached to remote volume
'/data/sdc1/data-ipred'.
[2015-08-19 21:19:42.206622] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-3: Server
and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:19:42.206816] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-3:
Server lk version = 1
[2015-08-19 21:19:42.207424] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-1:
Another crawl is in progress for data-cotelser-client-3
[2015-08-19 21:19:42.208042] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-0:
Another crawl is in progress for data-ipred-client-1
[2015-08-19 21:19:42.208066] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-1:
Another crawl is in progress for data-ipred-client-3
[2015-08-19 21:19:42.208760] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-0:
Another crawl is in progress for data-cotelser-client-1
[2015-08-19 21:26:20.555892] W [socket.c:523:__socket_rwv] 0-glusterfs:
readv on 127.0.0.1:24007 failed (No data available)
[2015-08-19 21:26:21.647156] W [glusterfsd.c:1095:cleanup_and_exit]
(-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f55d518004d]
(-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4) [0x7f55d58460a4]
(-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x4073bd]))) 0-: received
signum (15), shutting down
[2015-08-19 21:27:25.946841] I [glusterfsd.c:1959:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2
(/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p
/var/lib/glusterd/glustershd/run/glustershd.pid -l
/var/log/glusterfs/glustershd.log -S
/var/run/17c62c5aeddf9c73dfed4aa2d7407aaa.socket --xlator-option
*replicate*.node-uuid=c53ad0c0-3940-443a-9687-597f6a14b783)
[2015-08-19 21:27:26.162911] I [socket.c:3578:socket_init]
0-socket.glusterfsd: SSL support is NOT enabled
[2015-08-19 21:27:26.162929] I [socket.c:3593:socket_init]
0-socket.glusterfsd: using system polling thread
[2015-08-19 21:27:26.163006] I [socket.c:3578:socket_init] 0-glusterfs: SSL
support is NOT enabled
[2015-08-19 21:27:26.163014] I [socket.c:3593:socket_init] 0-glusterfs:
using system polling thread
[2015-08-19 21:27:31.860980] I [graph.c:254:gf_add_cmdline_options]
0-data-cotelser-replicate-1: adding option 'node-uuid' for volume
'data-cotelser-replicate-1' with value
'c53ad0c0-3940-443a-9687-597f6a14b783'
[2015-08-19 21:27:31.860999] I [graph.c:254:gf_add_cmdline_options]
0-data-cotelser-replicate-0: adding option 'node-uuid' for volume
'data-cotelser-replicate-0' with value
'c53ad0c0-3940-443a-9687-597f6a14b783'
[2015-08-19 21:27:31.861007] I [graph.c:254:gf_add_cmdline_options]
0-data-ipred-replicate-1: adding option 'node-uuid' for volume
'data-ipred-replicate-1' with value 'c53ad0c0-3940-443a-9687-597f6a14b783'
[2015-08-19 21:27:31.861025] I [graph.c:254:gf_add_cmdline_options]
0-data-ipred-replicate-0: adding option 'node-uuid' for volume
'data-ipred-replicate-0' with value 'c53ad0c0-3940-443a-9687-597f6a14b783'
[2015-08-19 21:27:31.864670] I [socket.c:3578:socket_init]
0-data-cotelser-client-3: SSL support is NOT enabled
[2015-08-19 21:27:31.864698] I [socket.c:3593:socket_init]
0-data-cotelser-client-3: using system polling thread
[2015-08-19 21:27:31.864975] I [socket.c:3578:socket_init]
0-data-cotelser-client-2: SSL support is NOT enabled
[2015-08-19 21:27:31.864990] I [socket.c:3593:socket_init]
0-data-cotelser-client-2: using system polling thread
[2015-08-19 21:27:31.865260] I [socket.c:3578:socket_init]
0-data-cotelser-client-1: SSL support is NOT enabled
[2015-08-19 21:27:31.865273] I [socket.c:3593:socket_init]
0-data-cotelser-client-1: using system polling thread
[2015-08-19 21:27:31.865496] I [socket.c:3578:socket_init]
0-data-cotelser-client-0: SSL support is NOT enabled
[2015-08-19 21:27:31.865511] I [socket.c:3593:socket_init]
0-data-cotelser-client-0: using system polling thread
[2015-08-19 21:27:31.868522] I [socket.c:3578:socket_init]
0-data-ipred-client-3: SSL support is NOT enabled
[2015-08-19 21:27:31.868540] I [socket.c:3593:socket_init]
0-data-ipred-client-3: using system polling thread
[2015-08-19 21:27:31.868721] I [socket.c:3578:socket_init]
0-data-ipred-client-2: SSL support is NOT enabled
[2015-08-19 21:27:31.868731] I [socket.c:3593:socket_init]
0-data-ipred-client-2: using system polling thread
[2015-08-19 21:27:31.868909] I [socket.c:3578:socket_init]
0-data-ipred-client-1: SSL support is NOT enabled
[2015-08-19 21:27:31.868919] I [socket.c:3593:socket_init]
0-data-ipred-client-1: using system polling thread
[2015-08-19 21:27:31.869099] I [socket.c:3578:socket_init]
0-data-ipred-client-0: SSL support is NOT enabled
[2015-08-19 21:27:31.869108] I [socket.c:3593:socket_init]
0-data-ipred-client-0: using system polling thread
[2015-08-19 21:27:31.869144] I [client.c:2294:notify]
0-data-cotelser-client-0: parent translators are ready, attempting connect
on transport
[2015-08-19 21:27:31.869468] I [client.c:2294:notify]
0-data-cotelser-client-1: parent translators are ready, attempting connect
on transport
[2015-08-19 21:27:31.869725] I [client.c:2294:notify]
0-data-cotelser-client-2: parent translators are ready, attempting connect
on transport
[2015-08-19 21:27:31.869958] I [client.c:2294:notify]
0-data-cotelser-client-3: parent translators are ready, attempting connect
on transport
[2015-08-19 21:27:31.870195] I [client.c:2294:notify] 0-data-ipred-client-0:
parent translators are ready, attempting connect on transport
[2015-08-19 21:27:31.870418] I [client.c:2294:notify] 0-data-ipred-client-1:
parent translators are ready, attempting connect on transport
[2015-08-19 21:27:31.870670] I [client.c:2294:notify] 0-data-ipred-client-2:
parent translators are ready, attempting connect on transport
[2015-08-19 21:27:31.870905] I [client.c:2294:notify] 0-data-ipred-client-3:
parent translators are ready, attempting connect on transport
Final graph:
+---------------------------------------------------------------------------
---+
1: volume data-cotelser-client-0
2: type protocol/client
3: option remote-host cotelser01
4: option remote-subvolume /data/sdb1/data-cotelser
5: option transport-type socket
6: option username 691af945-5b60-4c66-9246-1834143ace9b
7: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c
8: end-volume
9:
10: volume data-cotelser-client-1
11: type protocol/client
12: option remote-host ipred01
13: option remote-subvolume /data/sdb1/data-cotelser
14: option transport-type socket
15: option username 691af945-5b60-4c66-9246-1834143ace9b
16: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c
17: end-volume
18:
19: volume data-cotelser-replicate-0
20: type cluster/replicate
21: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783
22: option background-self-heal-count 0
23: option metadata-self-heal on
24: option data-self-heal on
25: option entry-self-heal on
26: option self-heal-daemon on
27: option iam-self-heal-daemon yes
28: subvolumes data-cotelser-client-0 data-cotelser-client-1
29: end-volume
30:
31: volume data-cotelser-client-2
32: type protocol/client
33: option remote-host cotelser01
34: option remote-subvolume /data/sdc1/data-cotelser
35: option transport-type socket
36: option username 691af945-5b60-4c66-9246-1834143ace9b
37: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c
38: end-volume
39:
40: volume data-cotelser-client-3
41: type protocol/client
42: option remote-host ipred01
43: option remote-subvolume /data/sdc1/data-cotelser
44: option transport-type socket
45: option username 691af945-5b60-4c66-9246-1834143ace9b
46: option password 37ce79ae-503c-4a8e-993b-d9b08ee5a98c
47: end-volume
48:
49: volume data-cotelser-replicate-1
50: type cluster/replicate
51: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783
52: option background-self-heal-count 0
53: option metadata-self-heal on
54: option data-self-heal on
55: option entry-self-heal on
56: option self-heal-daemon on
57: option iam-self-heal-daemon yes
58: subvolumes data-cotelser-client-2 data-cotelser-client-3
59: end-volume
60:
61: volume data-ipred-client-0
62: type protocol/client
63: option remote-host cotelser01
64: option remote-subvolume /data/sdb1/data-ipred
65: option transport-type socket
66: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1
67: option password c1ab7f12-fe61-43e4-af17-67906d09cea5
68: end-volume
69:
70: volume data-ipred-client-1
71: type protocol/client
72: option remote-host ipred01
73: option remote-subvolume /data/sdb1/data-ipred
74: option transport-type socket
75: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1
76: option password c1ab7f12-fe61-43e4-af17-67906d09cea5
77: end-volume
78:
79: volume data-ipred-replicate-0
80: type cluster/replicate
81: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783
82: option background-self-heal-count 0
83: option metadata-self-heal on
84: option data-self-heal on
85: option entry-self-heal on
86: option self-heal-daemon on
87: option iam-self-heal-daemon yes
88: subvolumes data-ipred-client-0 data-ipred-client-1
89: end-volume
90:
91: volume data-ipred-client-2
92: type protocol/client
93: option remote-host cotelser01
94: option remote-subvolume /data/sdc1/data-ipred
95: option transport-type socket
96: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1
97: option password c1ab7f12-fe61-43e4-af17-67906d09cea5
98: end-volume
99:
100: volume data-ipred-client-3
101: type protocol/client
102: option remote-host ipred01
103: option remote-subvolume /data/sdc1/data-ipred
104: option transport-type socket
105: option username f39bb7e4-1b74-45ed-83f6-0b25987b0ec1
106: option password c1ab7f12-fe61-43e4-af17-67906d09cea5
107: end-volume
108:
109: volume data-ipred-replicate-1
110: type cluster/replicate
111: option node-uuid c53ad0c0-3940-443a-9687-597f6a14b783
112: option background-self-heal-count 0
113: option metadata-self-heal on
114: option data-self-heal on
115: option entry-self-heal on
116: option self-heal-daemon on
117: option iam-self-heal-daemon yes
118: subvolumes data-ipred-client-2 data-ipred-client-3
119: end-volume
120:
121: volume glustershd
122: type debug/io-stats
123: subvolumes data-cotelser-replicate-0 data-cotelser-replicate-1
data-ipred-replicate-0 data-ipred-replicate-1
124: end-volume
125:
+---------------------------------------------------------------------------
---+
[2015-08-19 21:27:31.871484] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-cotelser-client-1: changing port to 49154 (from 0)
[2015-08-19 21:27:31.871790] E
[client-handshake.c:1760:client_query_portmap_cbk] 0-data-cotelser-client-3:
failed to get the port number for remote subvolume. Please run 'gluster
volume status' on server to see if brick process is running.
[2015-08-19 21:27:31.871826] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-ipred-client-1: changing port to 49156 (from 0)
[2015-08-19 21:27:31.871871] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-cotelser-client-0: changing port to 49152 (from 0)
[2015-08-19 21:27:31.871894] I [client.c:2229:client_rpc_notify]
0-data-cotelser-client-3: disconnected from 172.25.30.2:24007. Client
process will keep trying to connect to glusterd until brick's port is
available
[2015-08-19 21:27:31.872242] E
[client-handshake.c:1760:client_query_portmap_cbk] 0-data-ipred-client-3:
failed to get the port number for remote subvolume. Please run 'gluster
volume status' on server to see if brick process is running.
[2015-08-19 21:27:31.872269] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-cotelser-client-2: changing port to 49153 (from 0)
[2015-08-19 21:27:31.872293] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-ipred-client-0: changing port to 49154 (from 0)
[2015-08-19 21:27:31.872343] I [client.c:2229:client_rpc_notify]
0-data-ipred-client-3: disconnected from 172.25.30.2:24007. Client process
will keep trying to connect to glusterd until brick's port is available
[2015-08-19 21:27:31.872676] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-ipred-client-2: changing port to 49155 (from 0)
[2015-08-19 21:27:31.872883] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-cotelser-client-1: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2015-08-19 21:27:31.872948] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-ipred-client-1: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2015-08-19 21:27:31.873009] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-cotelser-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2015-08-19 21:27:31.873067] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-1:
Connected to 172.25.30.2:49156, attached to remote volume
'/data/sdb1/data-ipred'.
[2015-08-19 21:27:31.873075] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-1: Server
and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:27:31.873101] I [afr-common.c:4131:afr_notify]
0-data-ipred-replicate-0: Subvolume 'data-ipred-client-1' came back up;
going online.
[2015-08-19 21:27:31.873144] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-1:
Connected to 172.25.30.2:49154, attached to remote volume
'/data/sdb1/data-cotelser'.
[2015-08-19 21:27:31.873154] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-1:
Server and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:27:31.873184] I [afr-common.c:4131:afr_notify]
0-data-cotelser-replicate-0: Subvolume 'data-cotelser-client-1' came back
up; going online.
[2015-08-19 21:27:31.873223] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-1:
Server lk version = 1
[2015-08-19 21:27:31.873240] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-1:
Server lk version = 1
[2015-08-19 21:27:31.873252] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-cotelser-client-2: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2015-08-19 21:27:31.873287] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-ipred-client-2: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2015-08-19 21:27:31.873322] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-ipred-client-0: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2015-08-19 21:27:31.873347] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-0:
Connected to 172.25.30.1:49152, attached to remote volume
'/data/sdb1/data-cotelser'.
[2015-08-19 21:27:31.873359] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:27:31.873469] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-2:
Connected to 172.25.30.1:49153, attached to remote volume
'/data/sdc1/data-cotelser'.
[2015-08-19 21:27:31.873480] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-2:
Server and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:27:31.873499] I [afr-common.c:4131:afr_notify]
0-data-cotelser-replicate-1: Subvolume 'data-cotelser-client-2' came back
up; going online.
[2015-08-19 21:27:31.873534] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-2:
Connected to 172.25.30.1:49155, attached to remote volume
'/data/sdc1/data-ipred'.
[2015-08-19 21:27:31.873544] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-2: Server
and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:27:31.873565] I [afr-common.c:4131:afr_notify]
0-data-ipred-replicate-1: Subvolume 'data-ipred-client-2' came back up;
going online.
[2015-08-19 21:27:31.873594] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-0:
Connected to 172.25.30.1:49154, attached to remote volume
'/data/sdb1/data-ipred'.
[2015-08-19 21:27:31.873604] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-0: Server
and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:27:31.873653] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-0:
Server lk version = 1
[2015-08-19 21:27:31.873670] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-2:
Server lk version = 1
[2015-08-19 21:27:31.873709] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-2:
Server lk version = 1
[2015-08-19 21:27:31.873755] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-0:
Server lk version = 1
[2015-08-19 21:27:31.873911] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-0:
Another crawl is in progress for data-ipred-client-1
[2015-08-19 21:27:31.874180] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-0:
Another crawl is in progress for data-ipred-client-1
[2015-08-19 21:27:31.907888] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-0:
Another crawl is in progress for data-cotelser-client-1
[2015-08-19 21:27:31.908846] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-0:
Another crawl is in progress for data-cotelser-client-1
[2015-08-19 21:27:35.164347] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-cotelser-client-3: changing port to 49155 (from 0)
[2015-08-19 21:27:35.164812] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-cotelser-client-3: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2015-08-19 21:27:35.165001] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-cotelser-client-3:
Connected to 172.25.30.2:49155, attached to remote volume
'/data/sdc1/data-cotelser'.
[2015-08-19 21:27:35.165014] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-cotelser-client-3:
Server and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:27:35.165112] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-data-ipred-client-3: changing port to 49157 (from 0)
[2015-08-19 21:27:35.165150] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-cotelser-client-3:
Server lk version = 1
[2015-08-19 21:27:35.165488] I
[client-handshake.c:1677:select_server_supported_programs]
0-data-ipred-client-3: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2015-08-19 21:27:35.165698] I
[client-handshake.c:1462:client_setvolume_cbk] 0-data-ipred-client-3:
Connected to 172.25.30.2:49157, attached to remote volume
'/data/sdc1/data-ipred'.
[2015-08-19 21:27:35.165712] I
[client-handshake.c:1474:client_setvolume_cbk] 0-data-ipred-client-3: Server
and Client lk-version numbers are not same, reopening the fds
[2015-08-19 21:27:35.165811] I
[client-handshake.c:450:client_set_lk_version_cbk] 0-data-ipred-client-3:
Server lk version = 1
[2015-08-19 21:27:35.166070] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-1:
Another crawl is in progress for data-cotelser-client-3
[2015-08-19 21:27:35.166336] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-1:
Another crawl is in progress for data-ipred-client-3
[2015-08-19 21:37:32.234149] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-0:
Another crawl is in progress for data-cotelser-client-1
[2015-08-19 21:37:35.237022] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-1:
Another crawl is in progress for data-ipred-client-3
[2015-08-19 21:47:32.322400] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-cotelser-replicate-0:
Another crawl is in progress for data-cotelser-client-1
[2015-08-19 21:47:35.323534] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-data-ipred-replicate-1:
Another crawl is in progress for data-ipred-client-3
root at ipred01:/var/log/glusterfs#
Saludos cordiales,
Descripción: Descripción: Descripción: Descripción: Descripción:
Descripción: Descripción: Descripción: Descripción: Descripción:
Descripción: cid:image001.jpg at 01CCA872.FCC7C370
NOC
IP·RED – BUNAR SRL
Maipú S/N – CASA CENTRAL
2126 – General Lagos - SF
Fijo: <tel:%2B54-341-4760459> +54-341-4760459 / +54-3402-490890
E-Mail: <mailto:noc at ipred.com.ar> noc at ipred.com.ar
Skype: nocipred
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150819/dea57922/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 1669 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150819/dea57922/attachment.jpg>
More information about the Gluster-users
mailing list