[Gluster-devel] question on glustershd

[Emmanuel Dreyfus]

Krutika Dhananjay <kdhananj at redhat.com> wrote:

> Could you provide the trace logs you had added with this failure? 

A bit different since I had tossed out the previous trace logs.

The file that cannot heal is gfid dae15ff0-0b16-4ca0-b27d-734f9d2290f8. 
You can see that the sweep completes, but the file is not reported as healed.


[afr-self-heald.c:676:afr_shd_full_healer] 0-patchy-replicate-0: starting full sweep on subvol patchy-client-0
[afr-self-heald.c:676:afr_shd_full_healer] 0-patchy-replicate-0: starting full sweep on subvol patchy-client-1
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ...
[client-rpc-fops.c:1525:client3_3_inodelk_cbk] 0-patchy-client-1: remote operation failed: Resource temporarily unavailable
[client-rpc-fops.c:1525:client3_3_inodelk_cbk] 0-patchy-client-0: remote operation failed: Resource temporarily unavailable
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = dae15ff0-0b16-4ca0-b27d-734f9d2290f8 lock->l_type F_UNLCK ...
[afr-self-heal-data.c:754:afr_selfheal_data] 0-patchy-replicate-0: calling afr_selfheal_tryinodelk inode->gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0:self-heal, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0:self-heal, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ...
[afr-self-heal-data.c:757:afr_selfheal_data] 0-patchy-replicate-0: calling afr_selfheal_tryinodelk inode->gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 => 2
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ...
[afr-self-heal-data.c:754:afr_selfheal_data] 0-patchy-replicate-0: calling afr_selfheal_tryinodelk inode->gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0:self-heal, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0:self-heal, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ...
[afr-self-heal-data.c:757:afr_selfheal_data] 0-patchy-replicate-0: calling afr_selfheal_tryinodelk inode->gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 => 2
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ...
[afr-self-heal-common.c:476:afr_log_selfheal] 0-patchy-replicate-0: Completed data selfheal on 994f616e-b6a8-4026-a1b2-176cb8220a18. source=1 sinks=0 
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0:self-heal, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0:self-heal, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ...
[afr-self-heal-metadata.c:54:__afr_selfheal_metadata_do] 0-patchy-replicate-0: performing metadata selfheal on 994f616e-b6a8-4026-a1b2-176cb8220a18
[afr-self-heal-common.c:476:afr_log_selfheal] 0-patchy-replicate-0: Completed data selfheal on beb51df4-9670-4b77-8a59-7d0ae7f84b28. source=0 sinks=1 
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0:self-heal, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0:self-heal, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_RWLCK ...
[afr-self-heal-metadata.c:54:__afr_selfheal_metadata_do] 0-patchy-replicate-0: performing metadata selfheal on beb51df4-9670-4b77-8a59-7d0ae7f84b28
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = 994f616e-b6a8-4026-a1b2-176cb8220a18 lock->l_type F_UNLCK ...
[afr-self-heal-common.c:476:afr_log_selfheal] 0-patchy-replicate-0: Completed metadata selfheal on 994f616e-b6a8-4026-a1b2-176cb8220a18. source=1 sinks=0 
[client.c:1668:client_inodelk] 0-patchy-client-0: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ...
[client.c:1668:client_inodelk] 0-patchy-client-1: volume = patchy-replicate-0, gfid = beb51df4-9670-4b77-8a59-7d0ae7f84b28 lock->l_type F_UNLCK ...
[afr-self-heal-common.c:476:afr_log_selfheal] 0-patchy-replicate-0: Completed metadata selfheal on beb51df4-9670-4b77-8a59-7d0ae7f84b28. source=0 sinks=1 
[afr-self-heald.c:686:afr_shd_full_healer] 0-patchy-replicate-0: finished full sweep on subvol patchy-client-1
[afr-self-heald.c:686:afr_shd_full_healer] 0-patchy-replicate-0: finished full sweep on subvol patchy-client-0

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu at netbsd.org