[Gluster-devel] [Gluster-users] can glusterfs replicate run in 100M ethernet?

Thu May 12 10:42:30 UTC 2016

VM pauses are happening because quorum is not met. i.e. at least two of the
brick in the setup are not connected to the client. So it is becoming
Read-only filesystem.

Pranith

On Wed, May 11, 2016 at 2:53 PM, maokx at sina.com <maokx at sina.com> wrote:

> Hi all:
>
> can glusterfs replicate run in 100M ethernet?
> gluster:
> [root at server187 qemu]# rpm -aq|grep gluster
> glusterfs-3.6.2-1.el7.x86_64
> glusterfs-extra-xlators-3.6.2-1.el7.x86_64
> svserver-gluster-4.16.14-0.el7.centos.noarch
> glusterfs-api-3.6.2-1.el7.x86_64
> glusterfs-fuse-3.6.2-1.el7.x86_64
> glusterfs-cli-3.6.2-1.el7.x86_64
> glusterfs-server-3.6.2-1.el7.x86_64
> glusterfs-rdma-3.6.2-1.el7.x86_64
> glusterfs-libs-3.6.2-1.el7.x86_64
> glusterfs-devel-3.6.2-1.el7.x86_64
> [root at server187 qemu]#
> *1、why vm on running server has paused?**Is 100M ethernet lead to?*
> I use replicate 3 glusterfs,one server down,two server still up.
> In this case,one vm on server running is become pause .
> detail:
> [2016-05-05 08:51:22.942682] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
> 0-pool01-client-2: changing port to 49152 (from 0)
> [2016-05-05 08:51:22.947960] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-05 08:51:22.948196] I
> [client-handshake.c:1413:select_server_supported_programs]
> 0-pool01-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
> [2016-05-05 08:51:22.948346] I
> [client-handshake.c:1200:client_setvolume_cbk] 0-pool01-client-2: Connected
> to pool01-client-2, attached to remote volume '/data/data01'.
> [2016-05-05 08:51:22.948358] I
> [client-handshake.c:1210:client_setvolume_cbk] 0-pool01-client-2: Server
> and Client lk-version numbers are not same, reopening the fds
> [2016-05-05 08:51:22.948365] I
> [client-handshake.c:1052:client_post_handshake] 0-pool01-client-2: 3 fds
> open - Delaying child_up until they are re-opened
> [2016-05-05 08:51:22.948612] I
> [client-handshake.c:674:client_child_up_reopen_done] 0-pool01-client-2:
> last fd open'd/lock-self-heal'd - notifying CHILD-UP
> [2016-05-05 08:51:22.948646] I [MSGID: 108002]
> [afr-common.c:3631:afr_notify] 0-pool01-replicate-0: Client-quorum is met
> [2016-05-05 08:51:22.948677] I
> [client-handshake.c:188:client_set_lk_version_cbk] 0-pool01-client-2:
> Server lk version = 1
> [2016-05-05 08:51:23.945232] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-05 08:51:26.948173] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-05 08:51:29.397014] I [afr-common.c:1484:afr_local_discovery_cbk]
> 0-pool01-replicate-0: selecting local read_child pool01-client-2
> [2016-05-05 08:51:29.951076] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-05 08:51:32.954015] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-05 08:51:35.956908] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-05 08:51:36.238878] W [fuse-bridge.c:1261:fuse_err_cbk]
> 0-glusterfs-fuse: 407688: FSYNC() ERR => -1 (Read-only file system)
> [2016-05-05 08:51:36.280494] W [fuse-bridge.c:1261:fuse_err_cbk]
> 0-glusterfs-fuse: 407689: FSYNC() ERR => -1 (Read-only file system)
> [2016-05-05 08:51:38.959928] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-05 08:51:41.962890] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-05 08:51:44.965974] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-05 08:51:47.968896] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-05 08:51:50.971802] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-05 08:51:53.974781] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
>
>
>
>
> *2、why Response so slowly?Is 100M ethernet lead to?*
> I use replicate 3 glusterfs,one server down,two server still up.
> In this case,one brick on server running is become readonly because of Client-quorum
> is not met.
> But after about three hours,that brick become better automatic.
> why Response so slowly?Is 100M ethernet lead to?
> Detail:
> [2016-05-04 06:49:47.924382] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 06:50:47.262468] W [socket.c:611:__socket_rwv]
> 0-pool01-client-0: readv on 192.168.101.185:49152 failed (No data
> available)
> [2016-05-04 06:50:47.262530] I [client.c:2215:client_rpc_notify]
> 0-pool01-client-0: disconnected from pool01-client-0. Client process will
> keep trying to connect to glusterd until brick's port is available
> [2016-05-04 06:50:47.262546] W [MSGID: 108001]
> [afr-common.c:3635:afr_notify] 0-pool01-replicate-0: Client-quorum is not
> met
> [2016-05-04 06:50:54.310210] I [afr-common.c:1484:afr_local_discovery_cbk]
> 0-pool01-replicate-0: selecting local read_child pool01-client-2
> [2016-05-04 06:50:57.932864] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 06:50:57.933201] E [socket.c:2267:socket_connect_finish]
> 0-pool01-client-0: connection to 192.168.101.185:24007 failed (Connection
> refused)
> [2016-05-04 06:51:00.935714] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 06:51:03.349865] W [fuse-bridge.c:2270:fuse_writev_cbk]
> 0-glusterfs-fuse: 37106: WRITE => -1 (Read-only file system)
> [2016-05-04 06:51:03.851940] W [fuse-bridge.c:2270:fuse_writev_cbk]
> 0-glusterfs-fuse: 37117: WRITE => -1 (Read-only file system)
> [2016-05-04 06:51:03.938817] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 06:51:04.354396] W [fuse-bridge.c:2270:fuse_writev_cbk]
> 0-glusterfs-fuse: 37142: WRITE => -1 (Read-only file system)
> ...........
> [2016-05-04 09:43:20.018833] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:21.775009] W [fuse-bridge.c:1079:fuse_setattr_cbk]
> 0-glusterfs-fuse: 51883: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only
> file system)
> [2016-05-04 09:43:23.021665] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:26.024536] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:29.027486] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:31.791565] W [fuse-bridge.c:1079:fuse_setattr_cbk]
> 0-glusterfs-fuse: 51903: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only
> file system)
> [2016-05-04 09:43:32.030394] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:35.033372] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:38.036299] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:41.039224] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:41.807885] W [fuse-bridge.c:1079:fuse_setattr_cbk]
> 0-glusterfs-fuse: 51915: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only
> file system)
> [2016-05-04 09:43:44.042165] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:47.045042] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:50.047917] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:51.824073] W [fuse-bridge.c:1079:fuse_setattr_cbk]
> 0-glusterfs-fuse: 51927: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only
> file system)
> [2016-05-04 09:43:53.050793] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:56.053775] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:43:59.056653] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:01.839749] W [fuse-bridge.c:1079:fuse_setattr_cbk]
> 0-glusterfs-fuse: 51939: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only
> file system)
> [2016-05-04 09:44:02.059552] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:07.062665] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:07.063593] I [rpc-clnt.c:1761:rpc_clnt_reconfig]
> 0-pool01-client-1: changing port to 49152 (from 0)
> [2016-05-04 09:44:07.066088] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:07.066641] I
> [client-handshake.c:1413:select_server_supported_programs]
> 0-pool01-client-1: Using Program GlusterFS 3.3, Num (1298437), Version
> (330)
> [2016-05-04 09:44:07.067234] I
> [client-handshake.c:1200:client_setvolume_cbk] 0-pool01-client-1: Connected
> to pool01-client-1, attached to remote volume '/data/data01'.
> [2016-05-04 09:44:07.067247] I
> [client-handshake.c:1210:client_setvolume_cbk] 0-pool01-client-1: Server
> and Client lk-version numbers are not same, reopening the fds
> [2016-05-04 09:44:07.067253] I
> [client-handshake.c:1052:client_post_handshake] 0-pool01-client-1: 2 fds
> open - Delaying child_up until they are re-opened
> [2016-05-04 09:44:07.067778] I
> [client-handshake.c:674:client_child_up_reopen_done] 0-pool01-client-1:
> last fd open'd/lock-self-heal'd - notifying CHILD-UP
> [2016-05-04 09:44:07.067804] I [MSGID: 108002]
> [afr-common.c:3631:afr_notify] 0-pool01-replicate-0: Client-quorum is met
> [2016-05-04 09:44:07.068272] I
> [client-handshake.c:188:client_set_lk_version_cbk] 0-pool01-client-1:
> Server lk version = 1
> [2016-05-04 09:44:08.065350] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:11.068215] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:11.842992] I [afr-common.c:1484:afr_local_discovery_cbk]
> 0-pool01-replicate-0: selecting local read_child pool01-client-2
> [2016-05-04 09:44:11.865405] W
> [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-pool01-client-2:
> remote operation failed: No data available
> [2016-05-04 09:44:11.865633] W
> [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-pool01-client-1:
> remote operation failed: No data available
> [2016-05-04 09:44:11.866102] W [fuse-bridge.c:1261:fuse_err_cbk]
> 0-glusterfs-fuse: 51953: REMOVEXATTR() /__DIRECT_IO_TEST__ => -1 (No data
> available)
> [2016-05-04 09:44:14.071042] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:17.073986] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:20.076992] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:23.079942] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:26.082841] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:29.085667] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:32.088532] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:35.091452] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:38.094352] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:41.097332] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:44.100262] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:47.103083] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:50.105942] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:53.108953] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:56.111865] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:44:59.114713] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:45:02.117608] W [common-utils.c:2621:gf_ports_reserved]
> 0-glusterfs-socket: 54321 is not a valid port identifier
> [2016-05-04 09:45:04.595414] W
> [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-pool01-client-2:
> remote operation failed: No data available
> [2016-05-04 09:45:04.595634] W
> [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-pool01-client-1:
> remote operation failed: No data available
> [2016-05-04 09:45:04.595966] W [fuse-bridge.c:1261:fuse_err_cbk]
> 0-glusterfs-fuse: 52091: REMOVEXATTR() /__DIRECT_IO_TEST__ => -1 (No data
> available)
>
> ------------------------------
> maokx at sina.com
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160512/806f7107/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Catch(05-11-17-15-57).jpg
Type: image/jpeg
Size: 109342 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160512/806f7107/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Catch8642.jpg
Type: image/jpeg
Size: 105532 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160512/806f7107/attachment-0003.jpg>