[Gluster-users] VM disks corruption on 3.7.11
Kevin Lemonnier
lemonnierk at ulrar.net
Thu May 12 15:19:50 UTC 2016
As discussed, the missing ipvr50 log file.
On Thu, May 12, 2016 at 04:24:14PM +0200, Kevin Lemonnier wrote:
> As requested on IRC, here are the logs on the 3 nodes.
>
> On Thu, May 12, 2016 at 04:03:02PM +0200, Kevin Lemonnier wrote:
> > Hi,
> >
> > I had a problem some time ago with 3.7.6 and freezing during heals,
> > and multiple persons advised to use 3.7.11 instead. Indeed, with that
> > version the freez problem is fixed, it works like a dream ! You can
> > almost not tell that a node is down or healing, everything keeps working
> > except for a little freez when the node just went down and I assume
> > hasn't timed out yet, but that's fine.
> >
> > Now I have a 3.7.11 volume on 3 nodes for testing, and the VM are proxmox
> > VMs with qCow2 disks stored on the gluster volume.
> > Here is the config :
> >
> > Volume Name: gluster
> > Type: Replicate
> > Volume ID: e4f01509-beaf-447d-821f-957cc5c20c0a
> > Status: Started
> > Number of Bricks: 1 x 3 = 3
> > Transport-type: tcp
> > Bricks:
> > Brick1: ipvr2.client:/mnt/storage/gluster
> > Brick2: ipvr3.client:/mnt/storage/gluster
> > Brick3: ipvr50.client:/mnt/storage/gluster
> > Options Reconfigured:
> > cluster.quorum-type: auto
> > cluster.server-quorum-type: server
> > network.remote-dio: enable
> > cluster.eager-lock: enable
> > performance.quick-read: off
> > performance.read-ahead: off
> > performance.io-cache: off
> > performance.stat-prefetch: off
> > features.shard: on
> > features.shard-block-size: 64MB
> > cluster.data-self-heal-algorithm: full
> > performance.readdir-ahead: on
> >
> >
> > As mentioned, I rebooted one of the nodes to test the freezing issue I had
> > on previous versions and appart from the initial timeout, nothing, the website
> > hosted on the VMs keeps working like a charm even during heal.
> > Since it's testing, there isn't any load on it though, and I just tried to refresh
> > the database by importing the production one on the two MySQL VMs, and both of them
> > started doing I/O errors. I tried shutting them down and powering them on again,
> > but same thing, even starting full heals by hand doesn't solve the problem, the disks are
> > corrupted. They still work, but sometimes they remount their partitions read only ..
> >
> > I believe there is a few people already using 3.7.11, no one noticed corruption problems ?
> > Anyone using Proxmox ? As already mentionned in multiple other threads on this mailing list
> > by other users, I also have pretty much always shards in heal info, but nothing "stuck" there,
> > they always go away in a few seconds getting replaced by other shards.
> >
> > Thanks
> >
> > --
> > Kevin Lemonnier
> > PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
>
>
>
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
>
>
> --
> Kevin Lemonnier
> PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
--
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
-------------- next part --------------
[2016-05-09 09:05:35.262203] I [MSGID: 100030] [glusterfsd.c:2332:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.11 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/42ecf8056c8db7918ecbc3de0575911e.socket --xlator-option *replicate*.node-uuid=616a7c35-4483-4ceb-92a5-1ca4a1055589)
[2016-05-09 09:05:35.268593] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-05-09 09:05:35.306807] I [graph.c:269:gf_add_cmdline_options] 0-gluster-replicate-0: adding option 'node-uuid' for volume 'gluster-replicate-0' with value '616a7c35-4483-4ceb-92a5-1ca4a1055589'
[2016-05-09 09:05:35.315326] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-05-09 09:05:35.316842] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-0: parent translators are ready, attempting connect on transport
[2016-05-09 09:05:35.317417] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-1: parent translators are ready, attempting connect on transport
[2016-05-09 09:05:35.317901] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0)
[2016-05-09 09:05:35.317995] I [MSGID: 114020] [client.c:2106:notify] 0-gluster-client-2: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume gluster-client-0
2: type protocol/client
3: option ping-timeout 42
4: option remote-host ipvr2.client
5: option remote-subvolume /mnt/storage/gluster
6: option transport-type socket
7: option username 397434a1-3c7f-41c9-9c95-a1612f5df6e5
8: option password 1fa3e184-dc44-44fb-9998-224be39be2f9
9: option filter-O_DIRECT enable
10: end-volume
11:
12: volume gluster-client-1
13: type protocol/client
14: option ping-timeout 42
15: option remote-host ipvr3.client
16: option remote-subvolume /mnt/storage/gluster
17: option transport-type socket
18: option username 397434a1-3c7f-41c9-9c95-a1612f5df6e5
19: option password 1fa3e184-dc44-44fb-9998-224be39be2f9
20: option filter-O_DIRECT enable
21: end-volume
22:
23: volume gluster-client-2
24: type protocol/client
25: option ping-timeout 42
26: option remote-host ipvr50.client
[2016-05-09 09:05:35.318833] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
27: option remote-subvolume /mnt/storage/gluster
28: option transport-type socket
29: option username 397434a1-3c7f-41c9-9c95-a1612f5df6e5
30: option password 1fa3e184-dc44-44fb-9998-224be39be2f9
31: option filter-O_DIRECT enable
32: end-volume
33:
34: volume gluster-replicate-0
35: type cluster/replicate
36: option node-uuid 616a7c35-4483-4ceb-92a5-1ca4a1055589
37: option background-self-heal-count 0
38: option metadata-self-heal on
39: option data-self-heal on
40: option entry-self-heal on
41: option self-heal-daemon enable
42: option data-self-heal-algorithm full
43: option eager-lock enable
44: option quorum-type auto
45: option iam-self-heal-daemon yes
46: subvolumes gluster-client-0 gluster-client-1 gluster-client-2
47: end-volume
48:
[2016-05-09 09:05:35.318984] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0)
49: volume glustershd
50: type debug/io-stats
51: option log-level INFO
52: subvolumes gluster-replicate-0
53: end-volume
54:
+------------------------------------------------------------------------------+
[2016-05-09 09:05:35.319072] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-gluster-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2016-05-09 09:05:35.319213] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'.
[2016-05-09 09:05:35.319239] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-09 09:05:35.319311] I [MSGID: 108005] [afr-common.c:4007:afr_notify] 0-gluster-replicate-0: Subvolume 'gluster-client-0' came back up; going online.
[2016-05-09 09:05:35.319349] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-2: disconnected from gluster-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-09 09:05:35.319438] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1
[2016-05-09 09:05:35.319917] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-09 09:05:35.320255] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'.
[2016-05-09 09:05:35.320276] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-09 09:05:35.320487] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1
[2016-05-09 09:05:39.264204] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-2: changing port to 49153 (from 0)
[2016-05-09 09:05:39.264874] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-09 09:05:39.265212] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-2: Connected to gluster-client-2, attached to remote volume '/mnt/storage/gluster'.
[2016-05-09 09:05:39.265237] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-09 09:05:39.265436] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-2: Server lk version = 1
[2016-05-09 09:05:40.897410] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on f6461925-fb17-4a70-b4a3-0d234b8dd784. source=0 sinks=2
[2016-05-09 09:05:41.198990] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 11a98b03-f3db-4c53-9d34-550478ce46ff. source=0 sinks=2
[2016-05-09 09:05:42.194661] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 9abb42f0-0001-4c94-ad23-2c8a38d8b96a. source=0 sinks=2
[2016-05-09 09:05:43.333208] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 040a2ce9-3c9f-46fa-bd7d-f1cd6a7399ad. source=0 sinks=2
[2016-05-09 09:05:44.739776] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 592bce74-3d66-4e6b-bb82-e830f8a187b7. source=0 sinks=2
[2016-05-09 09:05:47.773703] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 1c8a9b1d-055d-472d-a01a-8d9cfed0d595. source=0 sinks=2
[2016-05-09 09:05:50.131017] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 334d1a06-1e0d-4346-9a65-1f03d5a842b7. source=0 sinks=2
[2016-05-09 09:05:51.902703] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 657b50a9-104f-451c-839f-f0f28b9bd510. source=0 sinks=2
[2016-05-09 09:05:09.200297] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 356b0404-8eab-4119-a4f9-c3e21f581027. source=0 sinks=2
[2016-05-09 09:05:12.293262] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 4fb01b63-6cc1-4dbb-ab41-6d3875167ab6. source=0 sinks=2
[2016-05-09 09:05:12.692064] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 6a428a65-ac5c-4bf8-b2f9-389b02b8baea. source=0 sinks=2
[2016-05-09 09:05:17.406277] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 9463b510-8cba-42c6-8ebe-db27a2ce9e9c. source=0 sinks=2
[2016-05-09 09:05:19.453367] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on a784b5b3-df70-4302-9d5e-945315e6826d. source=0 sinks=2
[2016-05-09 09:05:22.244544] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on d99f8136-4feb-4921-a289-7c05dde50bb0. source=0 sinks=2
[2016-05-12 11:12:09.013850] W [socket.c:589:__socket_rwv] 0-gluster-client-0: readv on 172.16.0.2:49152 failed (Connection timed out)
[2016-05-12 11:12:09.013908] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-0: disconnected from gluster-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-12 11:14:26.625815] E [socket.c:2279:socket_connect_finish] 0-gluster-client-0: connection to 172.16.0.2:24007 failed (Connection timed out)
[2016-05-12 11:14:28.420820] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-0: changing port to 49152 (from 0)
[2016-05-12 11:14:28.421696] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-12 11:14:28.422027] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-0: Connected to gluster-client-0, attached to remote volume '/mnt/storage/gluster'.
[2016-05-12 11:14:28.422058] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-12 11:14:28.422277] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-0: Server lk version = 1
[2016-05-12 11:14:29.451016] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on d09d5b5e-e0df-48df-80db-f29869f7a125. source=2 sinks=0
[2016-05-12 11:14:31.524424] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on e503cfb6-3730-4a23-826f-57da1024da58. source=2 sinks=0
[2016-05-12 11:14:33.923977] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 038be2a6-1f8d-4845-b418-3193ecc96506. source=2 sinks=0
[2016-05-12 11:14:36.040807] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on a08f373b-27bb-4a30-8c61-3ce13a829c7d. source=2 sinks=0
[2016-05-12 11:14:38.042780] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 34a31401-9d84-4bee-b5f7-3f218f5cfe5e. source=2 sinks=0
[2016-05-12 11:14:40.018479] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 653435d3-a3e3-4bdb-8cd8-0857c21e391a. source=2 sinks=0
[2016-05-12 11:14:42.640346] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on e141c4a2-7385-44a6-ae04-6e9241232a9b. source=2 sinks=0
[2016-05-12 11:14:44.597815] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on d4f923cd-2266-40a2-9de4-6e2907acdbce. source=2 sinks=0
[2016-05-12 11:14:44.785336] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 41891616-086e-40ef-b9c9-0bd0b7928c09. source=2 sinks=0
[2016-05-12 11:14:46.189454] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on bb34bb67-2b08-4a0b-87d2-59fab4bf63ff. source=2 sinks=0
[2016-05-12 11:14:48.025142] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 1e8705d3-0c5f-4b8d-b13b-d85cf3af6605. source=2 sinks=0
[2016-05-12 11:14:49.809322] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 63d544c7-8c0e-4c9d-87c2-88c943e51fa3. source=2 sinks=0
[2016-05-12 11:14:52.103854] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 51af71e3-ff6f-4f56-b5d7-e02bfd0798db. source=2 sinks=0
[2016-05-12 11:14:54.057043] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 964f9ec3-2bb7-41bb-875c-f780ddcc6f9e. source=2 sinks=0
[2016-05-12 11:14:56.126417] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on b3b3807d-77ac-4be0-8fb1-cbae74972b3d. source=2 sinks=0
[2016-05-12 11:14:58.012489] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 1e9e155a-7493-4566-b000-2f2bd992b8d3. source=2 sinks=0
[2016-05-12 11:14:59.312682] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 29a7aa82-0d6e-4100-8ec1-e58995b917ff. source=2 sinks=0
[2016-05-12 11:15:00.404408] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on df744cca-3a9f-4749-b990-7b4e66e380fa. source=2 sinks=0
[2016-05-12 11:15:01.638656] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on d68da2b9-ddd2-40b6-9f85-ca885190d632. source=2 sinks=0
[2016-05-12 11:15:03.009716] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 040a2ce9-3c9f-46fa-bd7d-f1cd6a7399ad. source=2 sinks=0
[2016-05-12 11:15:04.660177] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on db1382eb-73df-4f61-9809-2344f70f7efa. source=2 sinks=0
[2016-05-12 11:15:06.539039] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 5a619729-b8a7-4bc8-b6a5-45d70f4c4b5c. source=2 sinks=0
[2016-05-12 11:15:08.253000] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on bfb24bd9-0778-4860-8ce0-75d122684991. source=2 sinks=0
[2016-05-12 11:15:09.490381] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 4a59c70a-cfb7-47a3-8cb2-0d20aa6d6c69. source=2 sinks=0
[2016-05-12 11:15:09.495745] I [MSGID: 108026] [afr-self-heal-entry.c:589:afr_selfheal_entry_do] 0-gluster-replicate-0: performing entry selfheal on be318638-e8a0-4c6d-977d-7a937aa84806
[2016-05-12 11:15:30.220951] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed entry selfheal on be318638-e8a0-4c6d-977d-7a937aa84806. source=-1 sinks=0 1 2
[2016-05-12 11:24:17.269857] W [socket.c:589:__socket_rwv] 0-gluster-client-1: readv on 172.16.0.3:49152 failed (Connection timed out)
[2016-05-12 11:24:17.269920] I [MSGID: 114018] [client.c:2030:client_rpc_notify] 0-gluster-client-1: disconnected from gluster-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2016-05-12 11:25:34.295683] I [MSGID: 108026] [afr-self-heal-entry.c:589:afr_selfheal_entry_do] 0-gluster-replicate-0: performing entry selfheal on be318638-e8a0-4c6d-977d-7a937aa84806
[2016-05-12 11:25:44.971323] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed entry selfheal on be318638-e8a0-4c6d-977d-7a937aa84806. source=-1 sinks=0 2
[2016-05-12 11:26:34.817787] E [socket.c:2279:socket_connect_finish] 0-gluster-client-1: connection to 172.16.0.3:24007 failed (Connection timed out)
[2016-05-12 11:26:36.504578] I [rpc-clnt.c:1868:rpc_clnt_reconfig] 0-gluster-client-1: changing port to 49152 (from 0)
[2016-05-12 11:26:36.505404] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluster-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-05-12 11:26:36.505780] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluster-client-1: Connected to gluster-client-1, attached to remote volume '/mnt/storage/gluster'.
[2016-05-12 11:26:36.505808] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluster-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-05-12 11:26:36.506002] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluster-client-1: Server lk version = 1
[2016-05-12 11:26:38.626732] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 40ff8750-3eb5-44f1-a017-e293545e652c. source=2 sinks=1
[2016-05-12 11:26:40.581711] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 51af71e3-ff6f-4f56-b5d7-e02bfd0798db. source=2 sinks=1
[2016-05-12 11:26:43.243298] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on bfb24bd9-0778-4860-8ce0-75d122684991. source=2 sinks=1
[2016-05-12 11:26:45.866992] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on a08f373b-27bb-4a30-8c61-3ce13a829c7d. source=2 sinks=1
[2016-05-12 11:26:48.666049] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on e77c008c-bfa9-48c7-ad2a-b7dde70579ee. source=2 sinks=1
[2016-05-12 11:26:50.353314] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 915fcdb6-bbe3-443e-85ab-0bfd6f04354e. source=2 sinks=1
[2016-05-12 11:26:51.691916] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 1d41bd26-73e5-402f-b241-36fd89f3c667. source=2 sinks=1
[2016-05-12 11:26:54.039485] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 7a3d3fbe-b9f5-430e-b9a1-8e6ab1f14e2b. source=2 sinks=1
[2016-05-12 11:26:56.410813] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on f8ed71df-fb86-4122-92a7-a054387d2295. source=2 sinks=1
[2016-05-12 11:26:58.091799] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 9abb42f0-0001-4c94-ad23-2c8a38d8b96a. source=2 sinks=1
[2016-05-12 11:26:59.297555] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 592bce74-3d66-4e6b-bb82-e830f8a187b7. source=2 sinks=1
[2016-05-12 11:26:59.835113] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 4a59c70a-cfb7-47a3-8cb2-0d20aa6d6c69. source=2 sinks=1
[2016-05-12 11:27:01.817605] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 94a8cf3d-9ad1-414c-bc17-d837d408d28e. source=2 sinks=1
[2016-05-12 11:27:03.937204] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 3358fce4-fdb9-46eb-9b55-afc45c9d11b7. source=2 sinks=1
[2016-05-12 11:27:05.639365] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 4e2b9ca0-bdc8-4105-8baf-b29116d7a17d. source=2 sinks=1
[2016-05-12 11:27:07.765151] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 679af5b0-2b7e-477d-a9c5-4832ab9214e9. source=2 sinks=1
[2016-05-12 11:27:10.131138] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 1224198a-26a4-42df-be79-db43ff847db8. source=2 sinks=1
[2016-05-12 11:27:12.512463] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on fb2a6b00-a92a-4623-a6f7-73a4d059f722. source=2 sinks=1
[2016-05-12 11:27:14.843567] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 86d86a77-d74f-4951-92fd-f8ef7faefe14. source=2 sinks=1
[2016-05-12 11:27:16.139008] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 7736566d-7345-4d0a-b948-44d39a337204. source=2 sinks=1
[2016-05-12 11:27:17.339428] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on db1382eb-73df-4f61-9809-2344f70f7efa. source=2 sinks=1
[2016-05-12 11:27:18.781522] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 843760a4-bba7-4baa-b3d5-e5b7c97ce579. source=2 sinks=1
[2016-05-12 11:27:19.831135] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on c96ff7a3-e051-4d68-8cc0-536b089c9811. source=2 sinks=1
[2016-05-12 11:27:20.788650] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 20c70a52-a7f2-48d2-8ea9-bc239c8a37ba. source=2 sinks=1
[2016-05-12 11:27:22.455848] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on ad7ae9c4-4ee4-4e83-ad65-102fbe5e6f40. source=2 sinks=1
[2016-05-12 11:27:24.097484] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 9e8c9767-da11-48e6-85b0-3463042d4ad9. source=2 sinks=1
[2016-05-12 11:27:25.407941] I [MSGID: 108026] [afr-self-heal-common.c:747:afr_log_selfheal] 0-gluster-replicate-0: Completed data selfheal on 9463b510-8cba-42c6-8ebe-db27a2ce9e9c. source=2 sinks=1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160512/3186158d/attachment.sig>
More information about the Gluster-users
mailing list