[Gluster-users] Would difference in size (and content) of a file on replicated bricks be healed?
Ove K. Pettersen
ove.k.pettersen at gmail.com
Sun Feb 5 21:31:15 UTC 2012
Den 05.02.2012 22:20, skrev David Coulson:
>
>
> On 2/5/12 4:12 PM, Ove K. Pettersen wrote:
>> Hi...
>>
>> Started playing with gluster. And the heal functions is my "target"
>> for testing.
>>
>> Short description of my test
>> ----------------------------
>> * 4 replicas on single machine
>> * glusterfs mounted locally
>> * Create file on glusterfs-mounted directory: date >data.txt
> Usually it is bad to write directly to the brick.
I know it is bad to write to a brick directly, but my goal is to try to
simulate a failure on a brick.
And those can occure without a complete loss.
Maybe a file is destroyed/modified, but still exist.
>
>> * Append to file on one of the bricks: hostname >>data.txt
>> * Trigger a self-heal with: stat data.txt
>> => Modified file on the brick is still different from the three other
>> bricks.
> What a surprise :-)
>>
>> Q1: Is the modified file supposed to get corrected in my case?
>>
>> Q2: Is my test-case invalid for what gluster is supposed to handle?
> Probably, since the file has no gfid attributes, gluster has no clue
> what to do with it. If you delete it, it should self-heal.
As I stated in my details, I did an append to the file.
I checked before and after I did the append to the file.
It had the same extended attributes also after me appending.
Attribute "gfid" has a 16 byte value for data.txt
Attribute "afr.d1-client-1" has a 12 byte value for data.txt
Attribute "afr.d1-client-0" has a 12 byte value for data.txt
Attribute "afr.d1-client-3" has a 12 byte value for data.txt
Attribute "afr.d1-client-2" has a 12 byte value for data.txt
So I believe gluster should still consider the file as "a gluster-owned
file".
>>
>> I would be very happy to receive an opinion on this.
>>
>> All the details are put at the bottom
>>
>>
>> Regards,
>>
>> Ove
>>
>>
>>
>>
>>
>>
>> Details and logs
>> ----------------
>> My environment
>> * single machine with OpenSuse 12.1
>> * gluster 3.3beta2
>> * xfs is used for the brick-fs (each one to a separate disk)
>>
>> Volume Name: d1
>> Type: Replicate
>> Status: Started
>> Number of Bricks: 4
>> Transport-type: tcp
>> Bricks:
>> Brick1: glusterdev:/b1
>> Brick2: glusterdev:/b2
>> Brick3: glusterdev:/b3
>> Brick4: glusterdev:/b4
>> Options Reconfigured:
>> diagnostics.brick-log-level: DEBUG
>> diagnostics.client-log-level: DEBUG
>>
>> * My glustervolume is mountet as glusterfs on /d1
>> glusterdev:/d1 52403200 33024 52370176 1% /d1
>>
>>
>>
>> 1) I put a file on the glusterfs
>> date >/d1/data.txt
>>
>> 2) Checking the storage on the bricks: ls -l /b1 /b2 /b3 /b4
>> /b1:
>> total 8
>> -rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
>>
>> /b2:
>> total 8
>> -rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
>>
>> /b3:
>> total 8
>> -rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
>>
>> /b4:
>> total 8
>> -rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
>>
>>
>> Everything is OK.
>>
>> 3) Then append a line on to /b1/data.txt
>> hostname >>/b1/data.txt
>>
>> By just appending, I keep the inode and all the extended attributes
>> (checked with attr -l on the /b1/data.txt file).
>>
>>
>> 4) Check again: ls -l /b1 /b2 /b3 /b4
>> /b1:
>> total 8
>> -rw-r--r-- 1 root root 40 Feb 5 21:40 data.txt
>>
>> /b2:
>> total 8
>> -rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
>>
>> /b3:
>> total 8
>> -rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
>>
>> /b4:
>> total 8
>> -rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
>>
>> Now we have a mismatch between the bricks. Both in contents and also
>> the size of the file.
>>
>> 4) Trigger a self-heal by using stat
>> cd /d1 ; stat data.txt
>>
>> File: `data.txt'
>> Size: 29 Blocks: 16 IO Block: 131072 regular file
>> Device: 1fh/31d Inode: 18446744070822815228 Links: 1
>> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
>> Access: 2012-02-05 21:40:32.382157697 +0100
>> Modify: 2012-02-05 21:40:32.388157658 +0100
>> Change: 2012-02-05 21:40:32.389157657 +0100
>> Birth: -
>>
>>
>> 5) stat gives the correct size of 29 which was the size of the file
>> stored on the glusterfs.
>> 6) DEBUG-log from the client when running this stat-command below.
>>
>> 7) Check again the bricks: ls -l /b1 /b2 /b3 /b4
>> /b1:
>> total 8
>> -rw-r--r-- 1 root root 40 Feb 5 21:40 data.txt
>>
>> /b2:
>> total 8
>> -rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
>>
>> /b3:
>> total 8
>> -rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
>>
>> /b4:
>> total 8
>> -rw-r--r-- 1 root root 29 Feb 5 21:40 data.txt
>>
>>
>> 8) Brick /b1 still has a bad copy of the file.
>>
>>
>>
>> In the logs it looks like the difference in size is detected.
>> It also starts some self-healing actions.
>> But then for some reason finds out no healing is needed.
>> Marked a few lines with <<<<< ???? which looks suspect.
>>
>> ===============CUT Client Log
>> [2012-02-05 21:41:01.206102] D
>> [client-handshake.c:179:client_start_ping] 0-d1-client-0: returning
>> as transport is already disconnected OR there are no frames (0 || 0)
>> [2012-02-05 21:41:01.206201] D
>> [client-handshake.c:179:client_start_ping] 0-d1-client-1: returning
>> as transport is already disconnected OR there are no frames (0 || 0)
>> [2012-02-05 21:41:01.206224] D
>> [client-handshake.c:179:client_start_ping] 0-d1-client-2: returning
>> as transport is already disconnected OR there are no frames (0 || 0)
>> [2012-02-05 21:41:01.206241] D
>> [client-handshake.c:179:client_start_ping] 0-d1-client-3: returning
>> as transport is already disconnected OR there are no frames (0 || 0)
>> [2012-02-05 21:41:03.279124] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.279182] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.279202] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.279218] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.279232] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 0 character innocent
>> [2012-02-05 21:41:03.279251] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 1 character innocent
>> [2012-02-05 21:41:03.279266] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 2 character innocent
>> [2012-02-05 21:41:03.279280] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 3 character innocent
>> [2012-02-05 21:41:03.279294] D
>> [afr-self-heal-common.c:738:afr_mark_sources] 0-d1-replicate-0:
>> Number of sources: 0
>> [2012-02-05 21:41:03.279309] D
>> [afr-self-heal-data.c:836:afr_lookup_select_read_child_by_txn_type]
>> 0-d1-replicate-0: returning read_child: 1
>> [2012-02-05 21:41:03.279324] D
>> [afr-common.c:1183:afr_lookup_select_read_child] 0-d1-replicate-0:
>> Source selected as 1 for /
>> [2012-02-05 21:41:03.279341] D
>> [afr-common.c:990:afr_lookup_build_response_params] 0-d1-replicate-0:
>> Building lookup response from 1
>> [2012-02-05 21:41:03.279561] D [afr-common.c:1945:afr_lookup]
>> 0-d1-replicate-0: failed to get the gfid from dict
>> [2012-02-05 21:41:03.280265] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.280299] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.280317] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.280332] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.280346] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 0 character innocent
>> [2012-02-05 21:41:03.280377] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 1 character innocent
>> [2012-02-05 21:41:03.280392] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 2 character innocent
>> [2012-02-05 21:41:03.280405] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 3 character innocent
>> [2012-02-05 21:41:03.280419] D
>> [afr-self-heal-common.c:738:afr_mark_sources] 0-d1-replicate-0:
>> Number of sources: 0
>> [2012-02-05 21:41:03.280433] D
>> [afr-self-heal-data.c:836:afr_lookup_select_read_child_by_txn_type]
>> 0-d1-replicate-0: returning read_child: 2
>> [2012-02-05 21:41:03.280507] D
>> [afr-common.c:1183:afr_lookup_select_read_child] 0-d1-replicate-0:
>> Source selected as 2 for /data.txt
>> [2012-02-05 21:41:03.280530] D
>> [afr-common.c:990:afr_lookup_build_response_params] 0-d1-replicate-0:
>> Building lookup response from 2
>> [2012-02-05 21:41:03.280548] I
>> [afr-common.c:1076:afr_detect_self_heal_by_iatt] 0-d1-replicate-0:
>> size differs for /data.txt
>> [2012-02-05 21:41:03.280576] I
>> [afr-common.c:1225:afr_launch_self_heal] 0-d1-replicate-0:
>> background data self-heal triggered. path: /data.txt
>> [2012-02-05 21:41:03.280612] D
>> [afr-self-heal-metadata.c:78:afr_sh_metadata_done] 0-d1-replicate-0:
>> proceeding to data check on /data.txt
>> [2012-02-05 21:41:03.281307] D
>> [afr-lk-common.c:1397:afr_nonblocking_inodelk] 0-d1-replicate-0:
>> attempting data lock range 0 0 by 140191242417704
>> [2012-02-05 21:41:03.281734] D
>> [afr-self-heal-data.c:1136:afr_sh_data_post_nonblocking_inodelk_cbk]
>> 0-d1-replicate-0: Non Blocking data inodelks done for /data.txt by
>> 140191242417704. Proceeding to self-heal
>> [2012-02-05 21:41:03.282815] D
>> [afr-self-heal-data.c:650:afr_sh_data_fix] 0-d1-replicate-0: Pending
>> matrix for: 140191242417704
>> [2012-02-05 21:41:03.282896] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.282940] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.282978] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.283015] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.283050] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 0 character innocent
>> [2012-02-05 21:41:03.283095] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 1 character innocent
>> [2012-02-05 21:41:03.283145] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 2 character innocent
>> [2012-02-05 21:41:03.283191] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 3 character innocent
>> [2012-02-05 21:41:03.283229] D
>> [afr-self-heal-common.c:738:afr_mark_sources] 0-d1-replicate-0:
>> Number of sources: 0 <<<<<< ?????
>> [2012-02-05 21:41:03.283264] D
>> [afr-self-heal-data.c:657:afr_sh_data_fix] 0-d1-replicate-0: No
>> self-heal needed for /data.txt <<<<<< ?????
>> [2012-02-05 21:41:03.283306] D
>> [afr-self-heal-data.c:320:afr_sh_data_finish] 0-d1-replicate-0:
>> finishing data selfheal of /data.txt
>> [2012-02-05 21:41:03.283344] D
>> [afr-lk-common.c:410:transaction_lk_op] 0-d1-replicate-0: lk op is
>> for a self heal
>> [2012-02-05 21:41:03.283379] D
>> [afr-lk-common.c:597:afr_unlock_inodelk] 0-d1-replicate-0: attempting
>> data unlock range 0 0 by 140191242417704
>> [2012-02-05 21:41:03.283879] I
>> [afr-self-heal-common.c:2022:afr_self_heal_completion_cbk]
>> 0-d1-replicate-0: background data self-heal completed on /data.txt
>> [2012-02-05 21:41:03.283960] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.284004] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.284042] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.284098] D
>> [afr-self-heal-common.c:139:afr_sh_print_pending_matrix]
>> 0-d1-replicate-0: pending_matrix: [ 0 0 0 0 ]
>> [2012-02-05 21:41:03.284136] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 0 character innocent
>> [2012-02-05 21:41:03.284173] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 1 character innocent
>> [2012-02-05 21:41:03.284216] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 2 character innocent
>> [2012-02-05 21:41:03.284253] D
>> [afr-self-heal-common.c:604:afr_find_child_character_type]
>> 0-d1-replicate-0: child 3 character innocent
>> [2012-02-05 21:41:03.284290] D
>> [afr-self-heal-common.c:738:afr_mark_sources] 0-d1-replicate-0:
>> Number of sources: 0
>> [2012-02-05 21:41:03.284325] D
>> [afr-self-heal-data.c:836:afr_lookup_select_read_child_by_txn_type]
>> 0-d1-replicate-0: returning read_child: 2
>> [2012-02-05 21:41:03.284361] D
>> [afr-common.c:1183:afr_lookup_select_read_child] 0-d1-replicate-0:
>> Source selected as 2 for /data.txt
>> [2012-02-05 21:41:03.284398] D
>> [afr-common.c:990:afr_lookup_build_response_params] 0-d1-replicate-0:
>> Building lookup response from 2
>> [2012-02-05 21:41:03.285622] D
>> [client-lk.c:442:delete_granted_locks_fd] 0-d1-client-2: Number of
>> locks cleared=0
>> [2012-02-05 21:41:03.285690] D
>> [client-lk.c:442:delete_granted_locks_fd] 0-d1-client-1: Number of
>> locks cleared=0
>> [2012-02-05 21:41:03.285779] D
>> [client-lk.c:442:delete_granted_locks_fd] 0-d1-client-3: Number of
>> locks cleared=0
>> [2012-02-05 21:41:03.285880] D
>> [client-lk.c:442:delete_granted_locks_fd] 0-d1-client-0: Number of
>> locks cleared=0
>>
>> ===============CUT Client Log
>>
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list