[Gluster-users] distributed files disappearing

Mon Sep 8 14:27:45 UTC 2014

Hello all,

I'm running gluster 3.4.2-1 on rhel 6.5 (don't worry, we'll be moving to RHS soon ;)
I just added a brick to an existing volume and suddenly my files can no longer be accessed by the client.
This is what I did:

[root at lpr-nas01 in]# gluster vol info prd_yearlyextract
Volume Name: prd_yearlyextract
Type: Distribute
Volume ID: d3849bfa-7d1e-43bb-9dcf-af6ff0746c19
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: lpr-nas01:/brick-xiv2/prd_yearlyextract

[root at lpr-nas01 in]# gluster vol brick-add prd_yearlyextract lpr-nas01:/brick-xiv1/prd_yearlyextract
[root at lpr-nas01 in]# gluster vol rebalance prd_yearlyextract

Now in the two bricks I have the following folders:

[root at lpr-nas01 in]# ll /brick-xiv1/prd_yearlyextract/sigedis/in/
total 0
---------T 2 444 ftpsiged 129337731  8. Sep 15:40 YCR_20130121_44.xml.DONE
---------T 2 444 ftpsiged 128549601  8. Sep 15:40 YCR_20130121_48.xml
---------T 2 444 ftpsiged 128166675  8. Sep 15:40 YCR_20130121_50.xml
---------T 2 444 ftpsiged 128227308  8. Sep 15:40 YCR_20130121_52.xml
<TRUNCATED>
---------T 2 998 ftpsiged  10544189  8. Sep 15:40 YCR_20140128_98.xml.GZ

[root at lpr-nas01 in]# ll /brick-xiv2/prd_yearlyextract/sigedis/in/
total 5648632
-rw-rw-r-- 2 444 ftpsiged 129337731  8. Mai 2013  YCR_20130121_44.xml.DONE
-rw-rw-r-- 2 444 ftpsiged 125619849  8. Mai 2013  YCR_20130121_45.xml
-rw-rw-r-- 2 444 ftpsiged 129315886  8. Mai 2013  YCR_20130121_46.xml
-rw-rw-r-- 2 444 ftpsiged 125900420  8. Mai 2013  YCR_20130121_47.xml
-rw-rw-r-- 2 444 ftpsiged 128549601  8. Mai 2013  YCR_20130121_48.xml
-rw-rw-r-- 2 444 ftpsiged 126278398  8. Mai 2013  YCR_20130121_49.xml
-rw-rw-r-- 2 444 ftpsiged 128166675  8. Mai 2013  YCR_20130121_50.xml
<TRUNCATED>
-rw-rw-r-- 2 998 ftpsiged  10425421 12. Feb 2014  YCR_20140128_99.xml.GZ

On the client I can only see the files that were in the original brick (brick2). The files that were put there afterwards are gone.
In the rebalance log I lines like this:
+------------------------------------------------------------------------------+
[2014-09-08 12:14:41.426033] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-prd_yearlyextract-client-0: changing port to 49282 (from 0)
[2014-09-08 12:14:41.426069] W [socket.c:514:__socket_rwv] 0-prd_yearlyextract-client-0: readv failed (No data available)
[2014-09-08 12:14:41.428805] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-prd_yearlyextract-client-1: changing port to 49321 (from 0)
[2014-09-08 12:14:41.428854] W [socket.c:514:__socket_rwv] 0-prd_yearlyextract-client-1: readv failed (No data available)
[2014-09-08 12:14:41.431634] I [client-handshake.c:1659:select_server_supported_programs] 0-prd_yearlyextract-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-09-08 12:14:41.431698] I [client-handshake.c:1659:select_server_supported_programs] 0-prd_yearlyextract-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2014-09-08 12:14:41.431895] I [client-handshake.c:1456:client_setvolume_cbk] 0-prd_yearlyextract-client-0: Connected to 192.168.151.21:49282, attached to remote volume '/brick-xiv2/prd_yearlyextract'.
[2014-09-08 12:14:41.431908] I [client-handshake.c:1468:client_setvolume_cbk] 0-prd_yearlyextract-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2014-09-08 12:14:41.431991] I [client-handshake.c:1456:client_setvolume_cbk] 0-prd_yearlyextract-client-1: Connected to 192.168.151.21:49321, attached to remote volume '/brick-xiv1/prd_yearlyextract'.
[2014-09-08 12:14:41.432001] I [client-handshake.c:1468:client_setvolume_cbk] 0-prd_yearlyextract-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2014-09-08 12:14:41.432104] I [client-handshake.c:450:client_set_lk_version_cbk] 0-prd_yearlyextract-client-0: Server lk version = 1
[2014-09-08 12:14:41.432130] I [client-handshake.c:450:client_set_lk_version_cbk] 0-prd_yearlyextract-client-1: Server lk version = 1
[2014-09-08 12:14:41.437758] I [dht-common.c:2618:dht_setxattr] 0-prd_yearlyextract-dht: fixing the layout of /
[2014-09-08 12:14:41.438260] I [dht-rebalance.c:1085:gf_defrag_migrate_data] 0-prd_yearlyextract-dht: migrate data called on /
[2014-09-08 12:14:41.445939] I [dht-rebalance.c:1296:gf_defrag_migrate_data] 0-prd_yearlyextract-dht: Migration operation on dir / took 0.01 secs
[2014-09-08 12:14:41.448304] I [dht-common.c:2618:dht_setxattr] 0-prd_yearlyextract-dht: fixing the layout of /sigedis
[2014-09-08 12:14:41.448800] I [dht-rebalance.c:1085:gf_defrag_migrate_data] 0-prd_yearlyextract-dht: migrate data called on /sigedis
[2014-09-08 12:14:41.475828] I [dht-rebalance.c:1296:gf_defrag_migrate_data] 0-prd_yearlyextract-dht: Migration operation on dir /sigedis took 0.03 secs
[2014-09-08 12:14:41.478187] I [dht-common.c:2618:dht_setxattr] 0-prd_yearlyextract-dht: fixing the layout of /sigedis/in
[2014-09-08 12:14:41.492970] I [dht-rebalance.c:1085:gf_defrag_migrate_data] 0-prd_yearlyextract-dht: migrate data called on /sigedis/in
[2014-09-08 12:14:41.924592] I [dht-rebalance.c:666:dht_migrate_file] 0-prd_yearlyextract-dht: /sigedis/in/YCR_20130121_44.xml.DONE: attempting to move from prd_yearlyextract-client-0 to prd_yearlyextract-client-1
[2014-09-08 12:14:41.925840] W [dht-rebalance.c:382:__dht_check_free_space] 0-prd_yearlyextract-dht: data movement attempted from node (prd_yearlyextract-client-0) with higher disk space to a node (prd_yearlyextract-client-1) with lesser disk space (/sigedis/in/YCR_20130121_44.xml.DONE)
[2014-09-08 12:14:41.928050] I [dht-rebalance.c:666:dht_migrate_file] 0-prd_yearlyextract-dht: /sigedis/in/YCR_20140128_112.xml.GZ: attempting to move from prd_yearlyextract-client-0 to prd_yearlyextract-client-1
[2014-09-08 12:14:41.929562] W [dht-rebalance.c:382:__dht_check_free_space] 0-prd_yearlyextract-dht: data movement attempted from node (prd_yearlyextract-client-0) with higher disk space to a node (prd_yearlyextract-client-1) with lesser disk space (/sigedis/in/YCR_20140128_112.xml.GZ)

The lines at the bottom concern the files that are invisible. What did I do wrong?

Wannes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140908/25f321cd/attachment.html>