[Bugs] [Bug 1365265] New: Glusterd not operational due to snapshot conflicting with nfs-ganesha export file in "/var/lib/glusterd/snaps"
bugzilla at redhat.com
bugzilla at redhat.com
Mon Aug 8 18:06:34 UTC 2016
https://bugzilla.redhat.com/show_bug.cgi?id=1365265
Bug ID: 1365265
Summary: Glusterd not operational due to snapshot conflicting
with nfs-ganesha export file in
"/var/lib/glusterd/snaps"
Product: GlusterFS
Version: 3.8.2
Component: ganesha-nfs
Keywords: Triaged
Severity: high
Priority: high
Assignee: bugs at gluster.org
Reporter: jthottan at redhat.com
CC: amukherj at redhat.com, bugs at gluster.org,
jthottan at redhat.com, mmalhotr at redhat.com,
nlevinki at redhat.com, olim at redhat.com,
rjoseph at redhat.com, storage-qa-internal at redhat.com
Depends On: 1318000, 1318591
+++ This bug was initially created as a clone of Bug #1318591 +++
+++ This bug was initially created as a clone of Bug #1318000 +++
Description of problem:
Glusterd is not operational on one node in the cluster (lvp87) due to snapshot
conflicting with nfs-ganesha
"export.Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01.conf" in
"/var/lib/glusterd/snaps"
Glusterd logs:
[2016-03-15 07:21:49.063861] E [MSGID: 101032]
[store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to
/var/lib/glusterd/snaps/Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01/export.Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01.conf/info.
[Not a directory]
[2016-03-15 07:21:49.063869] D [MSGID: 0]
[store.c:440:gf_store_handle_retrieve] 0-: Returning -1
[2016-03-15 07:21:49.063873] E [MSGID: 106200]
[glusterd-store.c:2550:glusterd_store_update_volinfo] 0-management: volinfo
handle is NULL
[2016-03-15 07:21:49.063878] E [MSGID: 106207]
[glusterd-store.c:2848:glusterd_store_retrieve_volume] 0-management: Failed to
update volinfo for
export.Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01.conf volume
[2016-03-15 07:21:49.063883] D [MSGID: 0]
[glusterd-utils.c:893:glusterd_volume_brickinfos_delete] 0-management:
Returning 0
[2016-03-15 07:21:49.063888] D [MSGID: 0] [store.c:461:gf_store_handle_destroy]
0-: Returning 0
[2016-03-15 07:21:49.063897] D [MSGID: 0]
[glusterd-utils.c:937:glusterd_volinfo_delete] 0-management: Returning 0
[2016-03-15 07:21:49.063902] E [MSGID: 106201]
[glusterd-store.c:3046:glusterd_store_retrieve_volumes] 0-management: Unable to
restore volume:
export.Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01.conf
[2016-03-15 07:21:49.063927] D [MSGID: 0]
[glusterd-store.c:3071:glusterd_store_retrieve_volumes] 0-management: Returning
with -1
[2016-03-15 07:21:49.063937] E [MSGID: 106195]
[glusterd-store.c:3439:glusterd_store_retrieve_snap] 0-management: Failed to
retrieve snap volumes for snap
Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01
[2016-03-15 07:21:49.063942] E [MSGID: 106043]
[glusterd-store.c:3593:glusterd_store_retrieve_snaps] 0-management: Unable to
restore snapshot: Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01
[2016-03-15 07:21:49.063948] D [MSGID: 0]
[glusterd-store.c:3611:glusterd_store_retrieve_snaps] 0-management: Returning
with -1
[2016-03-15 07:21:49.063953] D [MSGID: 0]
[glusterd-store.c:4343:glusterd_restore] 0-management: Returning -1
[2016-03-15 07:21:49.063967] E [MSGID: 101019] [xlator.c:428:xlator_init]
0-management: Initialization of volume 'management' failed, review your volfile
again
[2016-03-15 07:21:49.063973] E [graph.c:322:glusterfs_graph_init] 0-management:
initializing translator failed
[2016-03-15 07:21:49.063977] E [graph.c:661:glusterfs_graph_activate] 0-graph:
init failed
[2016-03-15 07:21:49.064238] D [logging.c:1764:gf_log_flush_extra_msgs]
0-logging-infra: Log buffer size reduced. About to flush 5 extra log messages
[2016-03-15 07:21:49.064249] D [logging.c:1767:gf_log_flush_extra_msgs]
0-logging-infra: Just flushed 5 extra log messages
[2016-03-15 07:21:49.064417] W [glusterfsd.c:1236:cleanup_and_exit]
(-->/usr/sbin/glusterd(glusterfs_volumes_init+0xfd) [0x7f0a7d57e2fd]
-->/usr/sbin/glusterd(glusterfs_process_volfp+0x126) [0x7f0a7d57e1a6]
-->/usr/sbin/glusterd(cleanup_and_exit+0x69) [0x7f0a7d57d789] ) 0-: received
signum (0), shutting down
[2016-03-15 07:21:49.064435] D
[glusterfsd-mgmt.c:2355:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper
signout arguments not given
Volume status:
Status of volume: certsd
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_certsd/brick_certsd_lvp88 49160 0 Y 2005
NFS Server on localhost 2049 0 Y 3584
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume certsd
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: gluster_shared_storage
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/var/lib/glus
terd/ss_brick 49161 0 Y 3564
NFS Server on localhost 2049 0 Y 3584
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume gluster_shared_storage
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv01-sknd3
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv01-sknd3/brick_pv01-sknd3_lvp88 49154 0 Y 2006
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv01-sknd3
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv02-ddfe5
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv02-ddfe5/brick_pv02-ddfe5_lvp88 49155 0 Y 2016
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv02-ddfe5
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv03-ed6fc
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv03-ed6fc/brick_pv03-ed6fc_lvp88 49156 0 Y 2022
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv03-ed6fc
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv04-1fr6e
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv04-1fr6e/brick_pv04-1fr6e_lvp88 49157 0 Y 2028
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv04-1fr6e
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv05-ku56u
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv05-ku56u/brick_pv05-ku56u_lvp88 49158 0 Y 2030
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv05-ku56u
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv06-m6o8i
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv06-m6o8i/brick_pv06-m6o8i_lvp88 49159 0 Y 2045
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv06-m6o8i
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: registry
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_registry/brick_registry_lvp88 49153 0 Y 2052
Self-heal Daemon on localhost N/A N/A Y 3592
Volume info:
Volume Name: pv06-m6o8i
Type: Replicate
Volume ID: b9ecd956-a10d-427a-ad95-81f735c58050
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1:
sv-2000lvp87.paas.local:/bricks/thin_brick_pv06-m6o8i/brick_pv06-m6o8i_lvp87
Brick2:
sv-2000lvp88.paas.local:/bricks/thin_brick_pv06-m6o8i/brick_pv06-m6o8i_lvp88
Options Reconfigured:
performance.readdir-ahead: on
nfs.disable: true
snap-activate-on-create: enable
auto-delete: enable
cluster.enable-shared-storage: enable
Volume Name: certsd
Type: Replicate
Volume ID: 17839849-381f-4299-8088-a1e62765e09c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: sv-2000lvp87.paas.local:/bricks/thin_brick_certsd/brick_certsd_lvp87
Brick2: sv-2000lvp88.paas.local:/bricks/thin_brick_certsd/brick_certsd_lvp88
Options Reconfigured:
performance.readdir-ahead: on
snap-activate-on-create: enable
auto-delete: enable
cluster.enable-shared-storage: enable
[root at sv-2000lvp88 ~]# gluster v info registry
Volume Name: registry
Type: Replicate
Volume ID: 1619f2f4-892d-48d2-9c7a-431a8b57a67e
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1:
sv-2000lvp87.paas.local:/bricks/thin_brick_registry/brick_registry_lvp87
Brick2:
sv-2000lvp88.paas.local:/bricks/thin_brick_registry/brick_registry_lvp88
Options Reconfigured:
features.barrier: disable
performance.readdir-ahead: on
nfs.disable: true
snap-activate-on-create: enable
auto-delete: enable
cluster.enable-shared-storage: enable
[root at sv-2000lvp88 ~]# gluster v info gluster_shared_storage
Volume Name: gluster_shared_storage
Type: Replicate
Volume ID: c160df0e-4472-47e0-80dd-e118b5dddc3f
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: sv-2000lvp88.paas.local:/var/lib/glusterd/ss_brick
Brick2: sv-2000lvp87:/var/lib/glusterd/ss_brick
Options Reconfigured:
performance.readdir-ahead: on
snap-activate-on-create: enable
auto-delete: enable
cluster.enable-shared-storage: enable
Version-Release number of selected component (if applicable):
RHGS 3.1.2
RHEL 7.2
How reproducible:
Always
Steps to Reproduce:
* Stop glusterd on both the nodes in the cluster
* Removed nf-ganesha package on both nodes than removed the exported conf file
"export.Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01.conf" from
"/var/lib/glusterd/snaps"
* Start Glusterd on both the nodes
Actual results:
Due to conflict, glusterd is not operational.
Expected results:
There should be no conflict & glusterd should be operational.
Additional info:
--- Additional comment from Mukul Malhotra on 2016-03-15 14:18 EDT ---
--- Additional comment from Mukul Malhotra on 2016-03-15 14:19 EDT ---
--- Additional comment from Mukul Malhotra on 2016-03-16 10:35:49 EDT ---
Hello,
Also, please improve the logging related to nfs-ganesha in the logs so that the
same should be visible & would help in troubleshooting the issue.
Thanks
Mukul
--- Additional comment from Vijay Bellur on 2016-03-17 10:02:15 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export
configuration properly) posted (#1) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-03-17 10:02:19 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : Skip invalid entries in
"vols" directory) posted (#1) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-03-18 05:36:22 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export
configuration properly) posted (#2) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-03-18 06:37:47 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : Skip invalid entries in
"vols" directory) posted (#2) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-03-24 06:45:01 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories
inside vols) posted (#3) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-03-24 06:45:05 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export
configuration properly) posted (#3) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Mike McCune on 2016-03-28 18:50:27 EDT ---
This bug was accidentally moved from POST to MODIFIED via an error in
automation, please see mmccune at redhat.com with any questions
--- Additional comment from Vijay Bellur on 2016-03-29 07:11:14 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories
inside vols) posted (#4) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-03-29 07:11:18 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export
configuration properly) posted (#4) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-03-30 02:59:09 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories
inside vols) posted (#5) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-03-30 02:59:12 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export
configuration properly) posted (#5) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-06-23 05:00:28 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export
configuration properly) posted (#6) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-06-23 13:17:20 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories
inside vols) posted (#6) for review on master by Atin Mukherjee
(amukherj at redhat.com)
--- Additional comment from Vijay Bellur on 2016-07-08 07:54:24 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories
inside vols) posted (#7) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-07-08 08:11:16 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export
configuration properly) posted (#7) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-07-08 08:18:21 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export
configuration properly) posted (#8) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-08-05 05:05:27 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories
inside vols) posted (#8) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-08-05 08:07:14 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : skip non directories inside
/var/lib/glusterd/vols) posted (#9) for review on master by jiffin tony Thottan
(jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-08-08 02:48:55 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : skip non directories inside
/var/lib/glusterd/vols) posted (#10) for review on master by jiffin tony
Thottan (jthottan at redhat.com)
--- Additional comment from Vijay Bellur on 2016-08-08 10:23:54 EDT ---
COMMIT: http://review.gluster.org/13764 committed in master by Atin Mukherjee
(amukherj at redhat.com)
------
commit 720b63c24b07ee64e1338db28de602b9abbef0a1
Author: Jiffin Tony Thottan <jthottan at gmail.com>
Date: Thu Mar 17 18:53:13 2016 +0530
glusterd : skip non directories inside /var/lib/glusterd/vols
Right now glusterd won't come up if vols directory contains an invalid
entry.
Instead of doing that with this change a message will be logged and then
skip
that entry
Change-Id: I665b5c35291b059cf054622da0eec4db44ec5f68
BUG: 1318591
Signed-off-by: Jiffin Tony Thottan <jthottan at redhat.com>
Reviewed-on: http://review.gluster.org/13764
Reviewed-by: Prashanth Pai <ppai at redhat.com>
Reviewed-by: Atin Mukherjee <amukherj at redhat.com>
Smoke: Gluster Build System <jenkins at build.gluster.org>
CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1318000
[Bug 1318000] [GSS] - Glusterd not operational due to snapshot conflicting
with nfs-ganesha export file in "/var/lib/glusterd/snaps"
https://bugzilla.redhat.com/show_bug.cgi?id=1318591
[Bug 1318591] Glusterd not operational due to snapshot conflicting with
nfs-ganesha export file in "/var/lib/glusterd/snaps"
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list