[Gluster-users] Cluster files are only partly visible.

Sat Sep 21 02:17:05 UTC 2013

I have a SL 6.1, newly upgraded 5-node 12-brick 3.3.2 cluster.  ext4 is
the base filesystem.  The clients are the same release.

During a "remove-brick" drain of a failing array one of the other arrays
failed. It was replaced.
The filesystem is now no longer read-only, but it is in a mess.

What can I do to try to retrieve the situation?

Thank you for your time,
James Bellinger

# ls /data/uwa
ls: cannot access /data/uwa/naoko: Invalid argument
ls: cannot access /data/uwa/IC79Oscillations: Invalid argument
ls: cannot access /data/uwa/desiati: Invalid argument
ls: cannot access /data/uwa/kopper: Invalid argument
ls: cannot access /data/uwa/omurchadha: Invalid argument
ls: cannot access /data/uwa/mfbaker: Invalid argument
ls: cannot access /data/uwa/cweaver: Invalid argument
ls: cannot access /data/uwa/hoshina: Invalid argument
ls: cannot access /data/uwa/pfendner: Invalid argument
briedel  ckopper  cweaver  dima     hskarlupka        jauffenb  jfeintzeig
 karle   krasberg    mahlers  mmerck  nwhitehorn  pettus    rasha    
rmaruyama  swesterhoff
chwendt  cprice   desiati  hoshina  IC79Oscillations  jeisch    jvansanten
 kopper  lost+found  mfbaker  naoko   omurchadha  pfendner  richards 
santander  sybenzvi

Yet some file are accessible:
# wc /data/uwa/jvansanten/sim/sweep.sh
 10  31 153 /data/uwa/jvansanten/sim/sweep.sh

The configuration is the same as it was before, modulo the removal of the
brick I was trying to drain and the replacement of the failed brick, and
the automatic changes going from 3.2.3 to 3.3.2 by rpm install.

Samples from the log files

Client:  /var/log/glusterfs/data-uwa/log
[2013-09-20 20:44:44.673716] E
[client-handshake.c:1298:client_dump_version_cbk] 0-scratch-client-2:
server doesn't support the version
[2013-09-20 20:44:44.673778] I [client.c:1883:client_rpc_notify]
0-scratch-client-2: disconnected
[2013-09-20 20:44:44.679971] E
[client-handshake.c:1298:client_dump_version_cbk] 0-scratch-client-3:
server doesn't support the version
[2013-09-20 20:44:44.680008] I [client.c:1883:client_rpc_notify]
0-scratch-client-3: disconnected

Server:  /var/log/glusterfs/brick/sda:
613:[2013-09-18 20:43:27.922169] I
[server-handshake.c:571:server_setvolume] 0-scratch-server: accepted
client from
npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-4-0
(version: 3.3.2)
664:[2013-09-19 13:03:06.957171] I
[server-handshake.c:571:server_setvolume] 0-scratch-server: accepted
client from
npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-3-1
(version: 3.3.2)
798:[2013-09-19 13:03:53.802320] I [server.c:703:server_rpc_notify]
0-scratch-server: disconnecting connectionfrom
npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-4-0
799:[2013-09-19 13:03:53.802356] I
[server-helpers.c:741:server_connection_put] 0-scratch-server: Shutting
down connection
npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-4-0
800:[2013-09-19 13:03:53.802386] I
[server-helpers.c:629:server_connection_destroy] 0-scratch-server:
destroyed connection of
npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-4-0
801:[2013-09-19 13:04:10.383589] I [server.c:703:server_rpc_notify]
0-scratch-server: disconnecting connectionfrom
npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-3-1
802:[2013-09-19 13:04:10.383672] I
[server-helpers.c:741:server_connection_put] 0-scratch-server: Shutting
down connection
npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-3-1
803:[2013-09-19 13:04:10.383708] I
[server-helpers.c:629:server_connection_destroy] 0-scratch-server:
destroyed connection of
npx4.icecube.wisc.edu-30969-2013/09/18-20:43:23:857292-scratch-client-3-1
804:[2013-09-19 13:04:17.228206] I
[server-handshake.c:571:server_setvolume] 0-scratch-server: accepted
client from
npx4.icecube.wisc.edu-13204-2013/09/19-13:04:13:172868-scratch-client-3-0
(version: 3.3.2)

data-uwa.log:
[2012-12-26 10:40:14.893300] E [common-utils.c:125:gf_resolve_ip6]
0-resolver: getaddrinfo failed (Name or service not known)
[2012-12-26 10:40:14.893343] E
[name.c:253:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS
resolution failed on host gfs-npx
[2012-12-26 10:40:14.893364] E [glusterfsd-mgmt.c:740:mgmt_rpc_notify]
0-glusterfsd-mgmt: failed to connect with remote-host: Success
[2012-12-26 10:40:14.893429] W [glusterfsd.c:727:cleanup_and_exit]
(-->/lib64/libpthread.so.0() [0x30fd2077f1]
(-->/opt/glusterfs/3.2.3/lib64/libglusterfs.so.0(gf_timer_proc+0xb9)
[0x7f61d4931089] (-->/opt/glusterfs/3.2.3/sbin/glusterfs() [0x407dbe])))
0-: received signum (1), shutting down

(The DNS resolution complaint is at least a year old, and seems to have
gone away when I hardwired the /etc/hosts file.  It makes no difference.)

etc-glusterfs-glusterd.vol.log:
[2013-09-21 01:53:02.499976] I [socket.c:1798:socket_event_handler]
0-transport: disconnecting now