[Gluster-users] Trying to recover a gluster volume upon reinstall
Pat Haley
phaley at mit.edu
Thu Jun 2 20:50:44 UTC 2016
we have a machine that previously had centos 6.8 and gluster 3.7.10-1
with 2 bricks. The machine had to be rebuilt with centos 6.8 and the 2
bricks were not formatted. Gluster 3.7.11 was installed with the new
OS, and we can start the service, create the volume with the 2 bricks
and mount the gluster share.
The folder name (gluster-data) in it is correct, but we are getting error:
ls /data
ls: cannot access /data/gluster-data: No such file or directory
gluster-data
The data and directories are still there (i.e. we can still see them
looking at the underlying file systems) but gluster isn't serving them.
Looking in the log file for each brick we sea the same errors:
[2016-06-03 04:30:07.494068] I [MSGID: 100030] [glusterfsd.c:2332:main]
0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version
3.7.11 (args: /usr/sbin/glusterfsd -s mseas-data2 --volfile-id
data-volume.mseas-data2.mnt-brick1 -p
/var/lib/glusterd/vols/data-volume/run/mseas-data2-mnt-brick1.pid -S
/var/run/gluster/aa572e87933c930cb53983de35bdccbe.socket --brick-name
/mnt/brick1 -l /var/log/glusterfs/bricks/mnt-brick1.log --xlator-option
*-posix.glusterd-uuid=c1110fd9-cb99-4ca1-b18a-536a122d67ef --brick-port
49152 --xlator-option data-volume-server.listen-port=49152)
[2016-06-03 04:30:07.510671] I [MSGID: 101190]
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2016-06-03 04:30:07.519040] I [graph.c:269:gf_add_cmdline_options]
0-data-volume-server: adding option 'listen-port' for volume
'data-volume-server' with value '49152'
[2016-06-03 04:30:07.519089] I [graph.c:269:gf_add_cmdline_options]
0-data-volume-posix: adding option 'glusterd-uuid' for volume
'data-volume-posix' with value 'c1110fd9-cb99-4ca1-b18a-536a122d67ef'
[2016-06-03 04:30:07.519479] I [MSGID: 101190]
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 2
[2016-06-03 04:30:07.519486] I [MSGID: 115034]
[server.c:403:_check_for_auth_option] 0-/mnt/brick1: skip format check
for non-addr auth option auth.login./mnt/brick1.allow
[2016-06-03 04:30:07.519537] I [MSGID: 115034]
[server.c:403:_check_for_auth_option] 0-/mnt/brick1: skip format check
for non-addr auth option
auth.login.0016d59c-9691-4bb2-bc44-b1d8b19dd230.password
[2016-06-03 04:30:07.520926] I
[rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:
Configured rpc.outstanding-rpc-limit with value 64
[2016-06-03 04:30:07.521003] W [MSGID: 101002]
[options.c:957:xl_opt_validate] 0-data-volume-server: option
'listen-port' is deprecated, preferred is
'transport.socket.listen-port', continuing with correction
[2016-06-03 04:30:07.523056] I [MSGID: 121050]
[ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is
disabled.
[2016-06-03 04:30:07.523077] W [MSGID: 101105]
[gfdb_sqlite3.h:239:gfdb_set_sql_params]
0-data-volume-changetimerecorder: Failed to retrieve sql-db-pagesize
from params.Assigning default value: 4096
[2016-06-03 04:30:07.523086] W [MSGID: 101105]
[gfdb_sqlite3.h:239:gfdb_set_sql_params]
0-data-volume-changetimerecorder: Failed to retrieve sql-db-journalmode
from params.Assigning default value: wal
[2016-06-03 04:30:07.523095] W [MSGID: 101105]
[gfdb_sqlite3.h:239:gfdb_set_sql_params]
0-data-volume-changetimerecorder: Failed to retrieve sql-db-sync from
params.Assigning default value: off
[2016-06-03 04:30:07.523102] W [MSGID: 101105]
[gfdb_sqlite3.h:239:gfdb_set_sql_params]
0-data-volume-changetimerecorder: Failed to retrieve sql-db-autovacuum
from params.Assigning default value: none
[2016-06-03 04:30:07.523280] I [trash.c:2369:init] 0-data-volume-trash:
no option specified for 'eliminate', using NULL
[2016-06-03 04:30:07.523910] W [graph.c:357:_log_if_unknown_option]
0-data-volume-server: option 'rpc-auth.auth-glusterfs' is not recognized
[2016-06-03 04:30:07.523937] W [graph.c:357:_log_if_unknown_option]
0-data-volume-server: option 'rpc-auth.auth-unix' is not recognized
[2016-06-03 04:30:07.523955] W [graph.c:357:_log_if_unknown_option]
0-data-volume-server: option 'rpc-auth.auth-null' is not recognized
[2016-06-03 04:30:07.523989] W [graph.c:357:_log_if_unknown_option]
0-data-volume-quota: option 'timeout' is not recognized
[2016-06-03 04:30:07.524031] W [graph.c:357:_log_if_unknown_option]
0-data-volume-trash: option 'brick-path' is not recognized
[2016-06-03 04:30:07.529994] W [MSGID: 113036]
[posix.c:2211:posix_rename] 0-data-volume-posix: found directory at
/mnt/brick1/.trashcan/ while expecting ENOENT [File exists]
Final graph:
+------------------------------------------------------------------------------+
1: volume data-volume-posix
2: type storage/posix
3: option glusterd-uuid c1110fd9-cb99-4ca1-b18a-536a122d67ef
4: option directory /mnt/brick1
5: option volume-id c54b2a60-ffdc-4d82-9db1-890e41002e28
6: end-volume
7:
8: volume data-volume-trash
9: type features/trash
10: option trash-dir .trashcan
11: option brick-path /mnt/brick1
12: option trash-internal-op off
13: subvolumes data-volume-posix
14: end-volume
15:
16: volume data-volume-changetimerecorder
17: type features/changetimerecorder
18: option db-type sqlite3
19: option hot-brick off
20: option db-name brick1.db
21: option db-path /mnt/brick1/.glusterfs/
22: option record-exit off
23: option ctr_link_consistency off
24: option ctr_lookupheal_link_timeout 300
25: option ctr_lookupheal_inode_timeout 300
26: option record-entry on
27: option ctr-enabled off
28: option record-counters off
29: option ctr-record-metadata-heat off
30: option sql-db-cachesize 1000
31: option sql-db-wal-autocheckpoint 1000
32: subvolumes data-volume-trash
33: end-volume
34:
35: volume data-volume-changelog
36: type features/changelog
37: option changelog-brick /mnt/brick1
38: option changelog-dir /mnt/brick1/.glusterfs/changelogs
39: option changelog-barrier-timeout 120
40: subvolumes data-volume-changetimerecorder
41: end-volume
42:
43: volume data-volume-bitrot-stub
44: type features/bitrot-stub
45: option export /mnt/brick1
46: subvolumes data-volume-changelog
47: end-volume
48:
49: volume data-volume-access-control
50: type features/access-control
51: subvolumes data-volume-bitrot-stub
52: end-volume
53:
54: volume data-volume-locks
55: type features/locks
56: subvolumes data-volume-access-control
57: end-volume
58:
59: volume data-volume-upcall
60: type features/upcall
61: option cache-invalidation off
62: subvolumes data-volume-locks
63: end-volume
64:
65: volume data-volume-io-threads
66: type performance/io-threads
67: subvolumes data-volume-upcall
68: end-volume
69:
70: volume data-volume-marker
71: type features/marker
72: option volume-uuid c54b2a60-ffdc-4d82-9db1-890e41002e28
73: option timestamp-file
/var/lib/glusterd/vols/data-volume/marker.tstamp
74: option quota-version 0
75: option xtime off
76: option gsync-force-xtime off
77: option quota off
78: option inode-quota off
79: subvolumes data-volume-io-threads
80: end-volume
81:
82: volume data-volume-barrier
83: type features/barrier
84: option barrier disable
85: option barrier-timeout 120
86: subvolumes data-volume-marker
87: end-volume
88:
89: volume data-volume-index
90: type features/index
91: option index-base /mnt/brick1/.glusterfs/indices
92: subvolumes data-volume-barrier
93: end-volume
94:
95: volume data-volume-quota
96: type features/quota
97: option volume-uuid data-volume
98: option server-quota off
99: option timeout 0
100: option deem-statfs off
101: subvolumes data-volume-index
102: end-volume
103:
104: volume data-volume-worm
105: type features/worm
106: option worm off
107: subvolumes data-volume-quota
108: end-volume
109:
110: volume data-volume-read-only
111: type features/read-only
112: option read-only off
113: subvolumes data-volume-worm
114: end-volume
115:
116: volume /mnt/brick1
117: type debug/io-stats
118: option log-level INFO
119: option latency-measurement off
120: option count-fop-hits off
121: subvolumes data-volume-read-only
122: end-volume
123:
124: volume data-volume-server
125: type protocol/server
126: option transport.socket.listen-port 49152
127: option rpc-auth.auth-glusterfs on
128: option rpc-auth.auth-unix on
129: option rpc-auth.auth-null on
130: option rpc-auth-allow-insecure on
131: option transport-type tcp
132: option auth.login./mnt/brick1.allow
0016d59c-9691-4bb2-bc44-b1d8b19dd230
133: option auth.login.0016d59c-9691-4bb2-bc44-b1d8b19dd230.password
b021dbcf-e114-4c23-ad9f-968a2d93dd61
134: option auth.addr./mnt/brick1.allow *
135: subvolumes /mnt/brick1
136: end-volume
137:
+------------------------------------------------------------------------------+
[2016-06-03 04:30:07.583590] I [login.c:81:gf_auth] 0-auth/login:
allowed user names: 0016d59c-9691-4bb2-bc44-b1d8b19dd230
[2016-06-03 04:30:07.583640] I [MSGID: 115029]
[server-handshake.c:690:server_setvolume] 0-data-volume-server: accepted
client from
mseas-data2-2383-2016/06/03-04:30:07:127671-data-volume-client-0-0-0
(version: 3.7.11)
[2016-06-03 04:30:40.124584] I [login.c:81:gf_auth] 0-auth/login:
allowed user names: 0016d59c-9691-4bb2-bc44-b1d8b19dd230
[2016-06-03 04:30:40.124628] I [MSGID: 115029]
[server-handshake.c:690:server_setvolume] 0-data-volume-server: accepted
client from
mseas-data2-2500-2016/06/03-04:30:40:46064-data-volume-client-0-0-0
(version: 3.7.11)
[2016-06-03 04:30:43.265342] W [MSGID: 101182]
[inode.c:174:__foreach_ancestor_dentry] 0-data-volume-server: per dentry
fn returned 1
[2016-06-03 04:30:43.265393] C [MSGID: 101184]
[inode.c:228:__is_dentry_cyclic] 0-/mnt/brick1/inode: detected cyclic
loop formation during inode linkage. inode
(00000000-0000-0000-0000-000000000001) linking under itself as gluster-data
[2016-06-03 04:30:43.269197] W [MSGID: 101182]
[inode.c:174:__foreach_ancestor_dentry] 0-data-volume-server: per dentry
fn returned 1
[2016-06-03 04:30:43.269241] C [MSGID: 101184]
[inode.c:228:__is_dentry_cyclic] 0-/mnt/brick1/inode: detected cyclic
loop formation during inode linkage. inode
(00000000-0000-0000-0000-000000000001) linking under itself as gluster-data
[2016-06-03 04:30:43.270689] W [MSGID: 101182]
[inode.c:174:__foreach_ancestor_dentry] 0-data-volume-server: per dentry
fn returned 1
[2016-06-03 04:30:43.270733] C [MSGID: 101184]
[inode.c:228:__is_dentry_cyclic] 0-/mnt/brick1/inode: detected cyclic
loop formation during inode linkage. inode
(00000000-0000-0000-0000-000000000001) linking under itself as gluster-data
This is a distributed volume, not replicated. Can we delete the gluster
volume, remove the .glusterfs folders from each brick and recreate the
volume? Will it re-index the files on both bricks?
Note:
From the last lines of log file,
there is a soft link at
/mnt/brick1/.glusterfs/00/00/00000000-0000-0000-0000-000000000001 >>>>
../../..
We have tried removing the link and restarting the service, no change in
behavior. It replaces/rebuilds the link on service startup.
Any advice you can give will be appreciated.
Thanks
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160602/d4e212eb/attachment.html>
More information about the Gluster-users
mailing list