[Gluster-users] Would difference in size (and content) of a file on replicated bricks be healed?
Ove K. Pettersen
ove.k.pettersen at gmail.com
Sun Feb 5 21:12:08 UTC 2012
Hi...
Started playing with gluster. And the heal functions is my "target" for
testing.
Short description of my test
----------------------------
* 4 replicas on single machine
* glusterfs mounted locally
* Create file on glusterfs-mounted directory: date >data.txt
* Append to file on one of the bricks: hostname >>data.txt
* Trigger a self-heal with: stat data.txt
=> Modified file on the brick is still different from the three other
bricks.
Q1: Is the modified file supposed to get corrected in my case?
Q2: Is my test-case invalid for what gluster is supposed to handle?
I would be very happy to receive an opinion on this.
All the details are put at the bottom
Regards,
Ove
Details and logs
----------------
My environment
* single machine with OpenSuse 12.1
* gluster 3.3beta2
* xfs is used for the brick-fs (each one to a separate disk)
Volume Name: d1
Type: Replicate
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: glusterdev:/b1
Brick2: glusterdev:/b2
Brick3: glusterdev:/b3
Brick4: glusterdev:/b4
Options Reconfigured:
diagnostics.brick-log-level: DEBUG
diagnostics.client-log-level: DEBUG
* My glustervolume is mountet as glusterfs on /d1
glusterdev:/d1 52403200 33024 52370176 1% /d1
1) I put a file on the glusterfs
date >/d1/data.txt
2) Checking the storage on the bricks: ls -l /b1 /b2 /b3 /b4
/b1:
total 8
-rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
/b2:
total 8
-rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
/b3:
total 8
-rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
/b4:
total 8
-rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
Everything is OK.
3) Then append a line on to /b1/data.txt
hostname >>/b1/data.txt
By just appending, I keep the inode and all the extended attributes
(checked with attr -l on the /b1/data.txt file).
4) Check again: ls -l /b1 /b2 /b3 /b4
/b1:
total 8
-rw-r--r-- 1 root root 40 Feb 5 21:40 data.txt
/b2:
total 8
-rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
/b3:
total 8
-rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
/b4:
total 8
-rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
Now we have a mismatch between the bricks. Both in contents and also the
size of the file.
4) Trigger a self-heal by using stat
cd /d1 ; stat data.txt
File: `data.txt'
Size: 29 Blocks: 16 IO Block: 131072 regular file
Device: 1fh/31d Inode: 18446744070822815228 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2012-02-05 21:40:32.382157697 +0100
Modify: 2012-02-05 21:40:32.388157658 +0100
Change: 2012-02-05 21:40:32.389157657 +0100
Birth: -
5) stat gives the correct size of 29 which was the size of the file
stored on the glusterfs.
6) DEBUG-log from the client when running this stat-command below.
7) Check again the bricks: ls -l /b1 /b2 /b3 /b4
/b1:
total 8
-rw-r--r-- 1 root root 40 Feb 5 21:40 data.txt
/b2:
total 8
-rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
/b3:
total 8
-rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
/b4:
total 8
-rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
8) Brick /b1 still has a bad copy of the file.
In the logs it looks like the difference in size is detected.
It also starts some self-healing actions.
But then for some reason finds out no healing is needed.
Marked a few lines with <<<<< ???? which looks suspect.
===============CUT Client Log
[2012-02-05 21:41:01.206102] D
[client-handshake.c:179:client_start_ping] 0-d1-client-0: returning as
transport is already disconnected OR there are no frames (0 || 0)
[2012-02-05 21:41:01.206201] D
[client-handshake.c:179:client_start_ping] 0-d1-client-1: returning as
transport is already disconnected OR there are no frames (0 || 0)
[2012-02-05 21:41:01.206224] D
[client-handshake.c:179:client_start_ping] 0-d1-client-2: returning as
transport is already disconnected OR there are no frames (0 || 0)
[2012-02-05 21:41:01.206241] D
[client-handshake.c:179:client_start_ping] 0-d1-client-3: returning as
transport is already disconnected OR there are no frames (0 || 0)
[2012-02-05 21:41:03.279124] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.279182] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.279202] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.279218] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.279232] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 0 character innocent
[2012-02-05 21:41:03.279251] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 1 character innocent
[2012-02-05 21:41:03.279266] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 2 character innocent
[2012-02-05 21:41:03.279280] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 3 character innocent
[2012-02-05 21:41:03.279294] D
[afr-self-heal-common.c:738:afr_mark_sources] 0-d1-replicate-0: Number
of sources: 0
[2012-02-05 21:41:03.279309] D
[afr-self-heal-data.c:836:afr_lookup_select_read_child_by_txn_type]
0-d1-replicate-0: returning read_child: 1
[2012-02-05 21:41:03.279324] D
[afr-common.c:1183:afr_lookup_select_read_child] 0-d1-replicate-0:
Source selected as 1 for /
[2012-02-05 21:41:03.279341] D
[afr-common.c:990:afr_lookup_build_response_params] 0-d1-replicate-0:
Building lookup response from 1
[2012-02-05 21:41:03.279561] D [afr-common.c:1945:afr_lookup]
0-d1-replicate-0: failed to get the gfid from dict
[2012-02-05 21:41:03.280265] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.280299] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.280317] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.280332] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.280346] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 0 character innocent
[2012-02-05 21:41:03.280377] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 1 character innocent
[2012-02-05 21:41:03.280392] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 2 character innocent
[2012-02-05 21:41:03.280405] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 3 character innocent
[2012-02-05 21:41:03.280419] D
[afr-self-heal-common.c:738:afr_mark_sources] 0-d1-replicate-0: Number
of sources: 0
[2012-02-05 21:41:03.280433] D
[afr-self-heal-data.c:836:afr_lookup_select_read_child_by_txn_type]
0-d1-replicate-0: returning read_child: 2
[2012-02-05 21:41:03.280507] D
[afr-common.c:1183:afr_lookup_select_read_child] 0-d1-replicate-0:
Source selected as 2 for /data.txt
[2012-02-05 21:41:03.280530] D
[afr-common.c:990:afr_lookup_build_response_params] 0-d1-replicate-0:
Building lookup response from 2
[2012-02-05 21:41:03.280548] I
[afr-common.c:1076:afr_detect_self_heal_by_iatt] 0-d1-replicate-0: size
differs for /data.txt
[2012-02-05 21:41:03.280576] I [afr-common.c:1225:afr_launch_self_heal]
0-d1-replicate-0: background data self-heal triggered. path: /data.txt
[2012-02-05 21:41:03.280612] D
[afr-self-heal-metadata.c:78:afr_sh_metadata_done] 0-d1-replicate-0:
proceeding to data check on /data.txt
[2012-02-05 21:41:03.281307] D
[afr-lk-common.c:1397:afr_nonblocking_inodelk] 0-d1-replicate-0:
attempting data lock range 0 0 by 140191242417704
[2012-02-05 21:41:03.281734] D
[afr-self-heal-data.c:1136:afr_sh_data_post_nonblocking_inodelk_cbk]
0-d1-replicate-0: Non Blocking data inodelks done for /data.txt by
140191242417704. Proceeding to self-heal
[2012-02-05 21:41:03.282815] D
[afr-self-heal-data.c:650:afr_sh_data_fix] 0-d1-replicate-0: Pending
matrix for: 140191242417704
[2012-02-05 21:41:03.282896] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.282940] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.282978] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.283015] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.283050] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 0 character innocent
[2012-02-05 21:41:03.283095] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 1 character innocent
[2012-02-05 21:41:03.283145] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 2 character innocent
[2012-02-05 21:41:03.283191] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 3 character innocent
[2012-02-05 21:41:03.283229] D
[afr-self-heal-common.c:738:afr_mark_sources] 0-d1-replicate-0: Number
of sources: 0 <<<<<< ?????
[2012-02-05 21:41:03.283264] D
[afr-self-heal-data.c:657:afr_sh_data_fix] 0-d1-replicate-0: No
self-heal needed for /data.txt <<<<<< ?????
[2012-02-05 21:41:03.283306] D
[afr-self-heal-data.c:320:afr_sh_data_finish] 0-d1-replicate-0:
finishing data selfheal of /data.txt
[2012-02-05 21:41:03.283344] D [afr-lk-common.c:410:transaction_lk_op]
0-d1-replicate-0: lk op is for a self heal
[2012-02-05 21:41:03.283379] D [afr-lk-common.c:597:afr_unlock_inodelk]
0-d1-replicate-0: attempting data unlock range 0 0 by 140191242417704
[2012-02-05 21:41:03.283879] I
[afr-self-heal-common.c:2022:afr_self_heal_completion_cbk]
0-d1-replicate-0: background data self-heal completed on /data.txt
[2012-02-05 21:41:03.283960] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.284004] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.284042] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.284098] D
[afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
[2012-02-05 21:41:03.284136] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 0 character innocent
[2012-02-05 21:41:03.284173] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 1 character innocent
[2012-02-05 21:41:03.284216] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 2 character innocent
[2012-02-05 21:41:03.284253] D
[afr-self-heal-common.c:604:afr_find_child_character_type]
0-d1-replicate-0: child 3 character innocent
[2012-02-05 21:41:03.284290] D
[afr-self-heal-common.c:738:afr_mark_sources] 0-d1-replicate-0: Number
of sources: 0
[2012-02-05 21:41:03.284325] D
[afr-self-heal-data.c:836:afr_lookup_select_read_child_by_txn_type]
0-d1-replicate-0: returning read_child: 2
[2012-02-05 21:41:03.284361] D
[afr-common.c:1183:afr_lookup_select_read_child] 0-d1-replicate-0:
Source selected as 2 for /data.txt
[2012-02-05 21:41:03.284398] D
[afr-common.c:990:afr_lookup_build_response_params] 0-d1-replicate-0:
Building lookup response from 2
[2012-02-05 21:41:03.285622] D [client-lk.c:442:delete_granted_locks_fd]
0-d1-client-2: Number of locks cleared=0
[2012-02-05 21:41:03.285690] D [client-lk.c:442:delete_granted_locks_fd]
0-d1-client-1: Number of locks cleared=0
[2012-02-05 21:41:03.285779] D [client-lk.c:442:delete_granted_locks_fd]
0-d1-client-3: Number of locks cleared=0
[2012-02-05 21:41:03.285880] D [client-lk.c:442:delete_granted_locks_fd]
0-d1-client-0: Number of locks cleared=0
===============CUT Client Log
More information about the Gluster-users
mailing list