[Gluster-users] Volume can not write to data if this volume quota limits capacity and mount itself volume on arm64（aarch64） architecture
陈曦
chenxi at shudun.com
Sun Jan 14 07:43:00 UTC 2018
Thanks for reading this email。I found a problem while using Glusterfs。
First，I created a Distributed Dispersed volume on three nodes，and Limit the volume capacity use quota command（this volume is auto mounted on /run/gluster/VOLUME_NAME. This volume can be read and written normally。
After, I manually mounted the volume in another path to provide data storage of SAMBA and ISCSI services, after which the volume can not be read and written normal。
（What is the auto mount parameter （the /run/gluster/VOLUME_NAME）？ can I write it to /etc/ fstab and use this mount point directly）
this is my commond and environment
[root at f08n29glusterfs]# uname -a
Linux f08n29 4.2.0 #1 SMP Tue Jun 7 01:18:20 CST 2016 aarch64 aarch64 aarch64 GNU/Linux
[root at f08n29glusterfs]# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (AltArch)
[root at f08n25test_vol]# glusterfs --version
glusterfs 3.7.20
root at f08n25test_vol]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.33.0.7 f08n25
10.33.0.8 f08n33
10.33.0.30 f08n29
[root at f08n29glusterfs]# gluster pool list
UUID Hostname State
05b203e0-6a2d-44bf-bce0-c1a5dbf2e81f 10.33.0.8 Connected
c72285e0-b98d-4a50-afa7-677916d1b2bf 10.33.0.7 Connected
a3de3c39-7574-4f6c-9429-de8ee24a97ee localhost Connected
[root at f08n29glusterfs]# gluster volume create test_vol disperse 3 redundancy 1 BRICK_LIST_ON_THREE_NODE force
[root at f08n29glusterfs]# gluster volume start test_vol
[root at f08n29glusterfs]# gluster volume quota test_vol enable
[root at f08n29glusterfs]# gluster volume quota test_vol limit-usage / 51000.0GB
[root at f08n29glusterfs]# mount.glusterfs 10.33.0.7:/test_vol /data/gluster/test_vol
[root at f08n29glusterfs]# cd /data/gluster/test_vol
[root at f08n25glusterfs]# mount
......
localhost:test_vol on /run/gluster/test_vol type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
localhost:/test_vol on /data/gluster/test_vol type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root at f08n25glusterfs]# gluster volume status
Status of volume: test_vol
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.33.0.30:/data/gluster/11ddcbb9-069
c-4de8-b3c1-cb11c9c1f770/test_vol 49210 0 Y 17870
......
NFS Server on localhost 2049 0 Y 78928
Self-heal Daemon on localhost N/A N/A Y 78935
Quota Daemon on localhost N/A N/A Y 78972
NFS Server on f08n29 2049 0 Y 18100
Self-heal Daemon on f08n29 N/A N/A Y 18107
Quota Daemon on f08n29 N/A N/A Y 18211
NFS Server on 10.33.0.8 2049 0 Y 90323
Self-heal Daemon on 10.33.0.8 N/A N/A Y 90330
Quota Daemon on 10.33.0.8 N/A N/A Y 90383
Task Status of Volume test_vol
------------------------------------------------------------------------------
There are no active volume tasks
[root at f08n29mnt]# mkdir 111
mkdir: cannot create directory ‘111’: Invalid argument
[root at f08n25test_vol]# ls
ls: cannot access 111: No such file or directory
111
[root at f08n25test_vol]# touch file
touch: setting times of ‘file’: No such file or directory
Using other nodes also can not write data normally
this is some log
[root at f08n25glusterfs]# tail -n 100 test_vol-quota-crawl.log
[2018-02-02 11:23:31.419152] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-0: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.419274] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-0: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.419330] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-0: Heal failed [Invalid argument]
[2018-02-02 11:23:31.419703] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-1: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.419744] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-1: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.419772] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-1: Heal failed [Invalid argument]
[2018-02-02 11:23:31.420085] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-2: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.420141] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-2: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.420184] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-2: Heal failed [Invalid argument]
[2018-02-02 11:23:31.420401] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-3: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.420516] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-3: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.420551] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-3: Heal failed [Invalid argument]
[2018-02-02 11:23:31.420924] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-4: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.420965] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-4: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.420996] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-4: Heal failed [Invalid argument]
[2018-02-02 11:23:31.421487] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-6: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.421527] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-6: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.421554] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-6: Heal failed [Invalid argument]
[2018-02-02 11:23:31.421596] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-7: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.421696] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-7: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.421730] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-7: Heal failed [Invalid argument]
[2018-02-02 11:23:31.421901] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-8: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.421957] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-8: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.421991] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-8: Heal failed [Invalid argument]
[2018-02-02 11:23:31.422084] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-9: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.422138] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-9: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.422171] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-9: Heal failed [Invalid argument]
[2018-02-02 11:23:31.422224] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-5: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.422263] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.422291] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:23:31.422360] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-10: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.422400] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-10: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.422427] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-10: Heal failed [Invalid argument]
[2018-02-02 11:23:31.422588] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-11: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.422629] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-11: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.422661] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-11: Heal failed [Invalid argument]
[2018-02-02 11:23:31.422726] I [MSGID: 109063] [dht-layout.c:718:dht_layout_normalize] 0-test_vol-dht: Found anomalies in /.trashcan (gfid = 00000000-0000-0000-0000-000000000005). Holes=1 overlaps=0
[2018-02-02 11:23:31.435685] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-0: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.435809] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-0: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.436116] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-1: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.436156] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-1: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.436461] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-2: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.436514] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-2: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.436898] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-3: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.436995] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-3: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.437325] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-4: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.437373] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-4: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.437779] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-5: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.437820] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.438006] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-6: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.438059] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-6: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.438152] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-7: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.438241] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-7: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.438451] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-8: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.438497] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-8: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.438603] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-9: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.438746] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-9: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.438752] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-10: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.438880] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-10: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.439070] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-11: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.439126] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-11: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.439301] I [MSGID: 109036] [dht-common.c:9082:dht_log_new_layout_for_dir_selfheal] 0-test_vol-dht: Setting layout of /.trashcan with [Subvol_name: test_vol-disperse-0, Err: -1 , Start: 2863311528 , Stop: 3221225468 , Hash: 1 ], [Subvol_name: test_vol-disperse-1, Err: -1 , Start: 3221225469 , Stop: 3579139409 , Hash: 1 ], [Subvol_name: test_vol-disperse-10, Err: -1 , Start: 3579139410 , Stop: 3937053350 , Hash: 1 ], [Subvol_name: test_vol-disperse-11, Err: -1 , Start: 3937053351 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: test_vol-disperse-2, Err: -1 , Start: 0 , Stop: 357913940 , Hash: 1 ], [Subvol_name: test_vol-disperse-3, Err: -1 , Start: 357913941 , Stop: 715827881 , Hash: 1 ], [Subvol_name: test_vol-disperse-4, Err: -1 , Start: 715827882 , Stop: 1073741822 , Hash: 1 ], [Subvol_name: test_vol-disperse-5, Err: -1 , Start: 1073741823 , Stop: 1431655763 , Hash: 1 ], [Subvol_name: test_vol-disperse-6, Err: -1 , Start: 1431655764 , Stop: 1789569704 , Hash: 1 ], [Subvol_name: test_vol-disperse-7, Err: -1 , Start: 1789569705 , Stop: 2147483645 , Hash: 1 ], [Subvol_name: test_vol-disperse-8, Err: -1 , Start: 2147483646 , Stop: 2505397586 , Hash: 1 ], [Subvol_name: test_vol-disperse-9, Err: -1 , Start: 2505397587 , Stop: 2863311527 , Hash: 1 ],
[2018-02-02 11:23:31.441413] I [MSGID: 122058] [ec-heal.c:2341:ec_heal_do] 0-test_vol-disperse-1: /.trashcan: name heal successful on 7
[2018-02-02 11:23:31.442588] I [MSGID: 122058] [ec-heal.c:2341:ec_heal_do] 0-test_vol-disperse-4: /.trashcan: name heal successful on 7
[2018-02-02 11:23:31.445959] I [MSGID: 122058] [ec-heal.c:2341:ec_heal_do] 0-test_vol-disperse-3: /.trashcan: name heal successful on 7
[2018-02-02 11:23:31.476532] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-1: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.476584] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-1: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.476615] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-1: Heal failed [Invalid argument]
[2018-02-02 11:23:31.478545] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-10: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.478606] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-10: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.478644] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-10: Heal failed [Invalid argument]
[2018-02-02 11:23:31.478823] I [MSGID: 109063] [dht-layout.c:718:dht_layout_normalize] 0-test_vol-dht: Found anomalies in /.trashcan/internal_op (gfid = 00000000-0000-0000-0000-000000000006). Holes=1 overlaps=0
[2018-02-02 11:23:31.491605] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-0: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.491665] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-0: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.491967] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-1: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.492007] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-1: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.492301] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-2: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.492341] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-2: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.492723] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-3: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.492768] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-3: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.493121] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-4: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.493162] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-4: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.493558] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-5: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.493599] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.494023] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-6: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.494067] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-6: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.494107] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-7: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.494154] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-7: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.494393] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-8: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.494493] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-8: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.494798] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-9: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.494952] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-10: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.494991] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-10: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.494998] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-9: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.495365] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-test_vol-disperse-11: Mismatching xdata in answers of 'LOOKUP'
[2018-02-02 11:23:31.495406] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-11: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:23:31.495542] I [MSGID: 109036] [dht-common.c:9082:dht_log_new_layout_for_dir_selfheal] 0-test_vol-dht: Setting layout of /.trashcan/internal_op with [Subvol_name: test_vol-disperse-0, Err: -1 , Start: 715827882 , Stop: 1073741822 , Hash: 1 ], [Subvol_name: test_vol-disperse-1, Err: -1 , Start: 1073741823 , Stop: 1431655763 , Hash: 1 ], [Subvol_name: test_vol-disperse-10, Err: -1 , Start: 1431655764 , Stop: 1789569704 , Hash: 1 ], [Subvol_name: test_vol-disperse-11, Err: -1 , Start: 1789569705 , Stop: 2147483645 , Hash: 1 ], [Subvol_name: test_vol-disperse-2, Err: -1 , Start: 2147483646 , Stop: 2505397586 , Hash: 1 ], [Subvol_name: test_vol-disperse-3, Err: -1 , Start: 2505397587 , Stop: 2863311527 , Hash: 1 ], [Subvol_name: test_vol-disperse-4, Err: -1 , Start: 2863311528 , Stop: 3221225468 , Hash: 1 ], [Subvol_name: test_vol-disperse-5, Err: -1 , Start: 3221225469 , Stop: 3579139409 , Hash: 1 ], [Subvol_name: test_vol-disperse-6, Err: -1 , Start: 3579139410 , Stop: 3937053350 , Hash: 1 ], [Subvol_name: test_vol-disperse-7, Err: -1 , Start: 3937053351 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: test_vol-disperse-8, Err: -1 , Start: 0 , Stop: 357913940 , Hash: 1 ], [Subvol_name: test_vol-disperse-9, Err: -1 , Start: 357913941 , Stop: 715827881 , Hash: 1 ],
[2018-02-02 11:23:31.527002] I [fuse-bridge.c:5016:fuse_thread_proc] 0-fuse: unmounting /tmp/mntEHHMn8
[2018-02-02 11:23:31.527167] W [glusterfsd.c:1251:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7bb0) [0xffff820a7bb0] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0x11c) [0xaaaacbcc9bc0] -->/usr/sbin/glusterfs(cleanup_and_exit+0x78) [0xaaaacbcc99fc] ) 0-: received signum (15), shutting down
[2018-02-02 11:23:31.527206] I [fuse-bridge.c:5720:fini] 0-fuse: Unmounting '/tmp/mntEHHMn8'.
[root at f08n25glusterfs]# tail -n 100 data-gluster-test_vol.log
719: end-volume
720:
721: volume meta-autoload
722: type meta
723: subvolumes test_vol
724: end-volume
725:
+------------------------------------------------------------------------------+
[2018-02-02 11:23:36.282441] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-test_vol-client-33: Connected to test_vol-client-33, attached to remote volume '/data/gluster/d96a32f0-cfb9-4ee4-b3a2-54ae9091fcb0/test_vol'.
[2018-02-02 11:23:36.282517] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-test_vol-client-33: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-02 11:23:36.282887] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-test_vol-client-33: Server lk version = 1
[2018-02-02 11:23:36.283198] I [rpc-clnt.c:1960:rpc_clnt_reconfig] 0-test_vol-client-35: changing port to 49221 (from 0)
[2018-02-02 11:23:36.283668] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-test_vol-client-34: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-02 11:23:36.284446] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-test_vol-client-34: Connected to test_vol-client-34, attached to remote volume '/data/gluster/eca8cd11-d264-40ab-a384-4f8f56fc1e4f/test_vol'.
[2018-02-02 11:23:36.284501] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-test_vol-client-34: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-02 11:23:36.284742] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-test_vol-client-34: Server lk version = 1
[2018-02-02 11:23:36.291994] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-test_vol-client-35: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-02 11:23:36.292683] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-test_vol-client-35: Connected to test_vol-client-35, attached to remote volume '/data/gluster/f6c653ed-1665-43cd-8a0c-8eaa8e23506b/test_vol'.
[2018-02-02 11:23:36.292720] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-test_vol-client-35: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-02 11:23:36.292830] I [MSGID: 122061] [ec.c:313:ec_up] 0-test_vol-disperse-11: Going UP
[2018-02-02 11:23:36.297432] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-test_vol-client-35: Server lk version = 1
[2018-02-02 11:23:36.297606] I [fuse-bridge.c:4083:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.23
[2018-02-02 11:23:36.297662] I [fuse-bridge.c:4768:fuse_graph_sync] 0-fuse: switched to graph 0
[2018-02-02 11:23:39.223315] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2018-02-02 11:23:39.449492] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2018-02-02 11:23:39.516999] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2018-02-02 11:23:39.542305] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2018-02-02 11:24:46.657381] E [MSGID: 114031] [client-rpc-fops.c:321:client3_3_mkdir_cbk] 0-test_vol-client-15: remote operation failed. Path: /111 [Invalid argument]
[2018-02-02 11:24:46.657536] E [MSGID: 114031] [client-rpc-fops.c:321:client3_3_mkdir_cbk] 0-test_vol-client-17: remote operation failed. Path: /111 [Invalid argument]
[2018-02-02 11:24:46.657598] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:46.657630] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:46.657892] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 17: MKDIR() /111 => -1 (Invalid argument)
[2018-02-02 11:24:55.295010] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:55.295077] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:55.298071] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:55.298165] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:55.299100] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:55.299151] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:55.302045] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:55.302083] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:55.303116] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:55.303153] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:55.305829] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:55.305879] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:55.309790] E [MSGID: 114031] [client-rpc-fops.c:321:client3_3_mkdir_cbk] 0-test_vol-client-15: remote operation failed. Path: /111 [Invalid argument]
[2018-02-02 11:24:55.309980] E [MSGID: 114031] [client-rpc-fops.c:321:client3_3_mkdir_cbk] 0-test_vol-client-17: remote operation failed. Path: /111 [Invalid argument]
[2018-02-02 11:24:55.310040] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:55.310070] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:55.310331] W [fuse-bridge.c:470:fuse_entry_cbk] 0-glusterfs-fuse: 20: MKDIR() /111 => -1 (Invalid argument)
[2018-02-02 11:24:59.853932] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:59.853993] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:59.856669] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:59.856709] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:59.857768] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:59.857806] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:59.860562] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:59.860599] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:59.861577] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:59.861613] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:59.864300] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:59.864345] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:59.867964] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-16: remote operation failed. Path: /111 [Is a directory]
[2018-02-02 11:24:59.867992] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-15: remote operation failed. Path: /111 [Invalid argument]
[2018-02-02 11:24:59.868134] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-17: remote operation failed. Path: /111 [Invalid argument]
[2018-02-02 11:24:59.868191] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:24:59.868221] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:24:59.868301] W [fuse-bridge.c:2002:fuse_create_cbk] 0-glusterfs-fuse: 23: /111 => -1 (Invalid argument)
[2018-02-02 11:25:03.461970] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:25:03.462035] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:25:03.464735] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:25:03.464784] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:25:03.465810] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:25:03.465846] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:25:03.468554] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:25:03.468590] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:25:05.167076] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:25:05.167130] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:25:05.169788] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:25:05.169826] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:25:05.170867] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:25:05.170904] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 11:25:05.173580] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 11:25:05.173629] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 12:24:01.295847] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-9: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 12:24:01.295914] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-9: Heal failed [Invalid argument]
[2018-02-02 12:24:19.680348] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 12:24:19.680410] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]
[2018-02-02 12:24:19.694268] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-15: remote operation failed. Path: /111 [Invalid argument]
[2018-02-02 12:24:19.694280] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-16: remote operation failed. Path: /111 [Is a directory]
[2018-02-02 12:24:19.694438] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-17: remote operation failed. Path: /111 [Invalid argument]
[2018-02-02 12:24:19.694555] W [fuse-bridge.c:2002:fuse_create_cbk] 0-glusterfs-fuse: 187: /111 => -1 (Invalid argument)
The message "W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-5: Heal failed [Invalid argument]" repeated 6 times between [2018-02-02 12:24:19.680410] and [2018-02-02 12:24:19.694503]
[2018-02-02 12:24:31.428284] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-3: remote operation failed. Path: /file [Invalid argument]
[2018-02-02 12:24:31.428435] W [MSGID: 114031] [client-rpc-fops.c:2402:client3_3_create_cbk] 0-test_vol-client-5: remote operation failed. Path: /file [Invalid argument]
[2018-02-02 12:24:31.428505] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-1: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)
[2018-02-02 12:24:31.428541] W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-1: Heal failed [Invalid argument]
[2018-02-02 12:24:31.428624] W [fuse-bridge.c:2002:fuse_create_cbk] 0-glusterfs-fuse: 192: /file => -1 (Invalid argument)
The message "W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-9: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)" repeated 3 times between [2018-02-02 12:24:01.295847] and [2018-02-02 12:24:01.302362]
The message "W [MSGID: 122002] [ec-common.c:71:ec_heal_report] 0-test_vol-disperse-9: Heal failed [Invalid argument]" repeated 3 times between [2018-02-02 12:24:01.295914] and [2018-02-02 12:24:01.302365]
The message "W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-test_vol-disperse-5: Operation failed on some subvolumes (up=7, mask=7, remaining=0, good=5, bad=2)" repeated 6 times between [2018-02-02 12:24:19.680348] and [2018-02-02 12:24:19.694499]
[root at f08n25bricks]# tail -n 100 data-gluster-1045fe63-bf09-42f5-986b-ce2d3d63def0-test_vol.log
[2018-02-02 11:23:28.615522] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:28.615573] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-90322-2018/01/14-06:17:22:557038-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:28.730628] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:28.730683] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-18106-2018/01/14-06:17:25:563091-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:28.820630] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:28.820686] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-18099-2018/01/14-06:17:25:551541-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:29.028758] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.028816] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n25-78971-2018/02/02-11:23:28:906426-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:29.492501] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.492553] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-24929-2018/01/13-10:10:52:662544-test_vol-client-1-19-0 (version: 3.7.20)
[2018-02-02 11:23:29.805720] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2018-02-02 11:23:29.815869] I [graph.c:269:gf_add_cmdline_options] 0-test_vol-server: adding option 'listen-port' for volume 'test_vol-server' with value '49163'
[2018-02-02 11:23:29.815936] I [graph.c:269:gf_add_cmdline_options] 0-test_vol-posix: adding option 'glusterd-uuid' for volume 'test_vol-posix' with value 'c72285e0-b98d-4a50-afa7-677916d1b2bf'
[2018-02-02 11:23:29.816237] I [MSGID: 121037] [changetimerecorder.c:1967:reconfigure] 0-test_vol-changetimerecorder: set
[2018-02-02 11:23:29.816393] I [MSGID: 0] [gfdb_sqlite3.c:1374:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB wal_autocheckpoint : 1000
[2018-02-02 11:23:29.816465] I [MSGID: 0] [gfdb_sqlite3.c:1374:gf_sqlite3_set_pragma] 0-sqlite3: Value set on DB cache_size : 1000
[2018-02-02 11:23:29.816947] W [socket.c:3677:reconfigure] 0-test_vol-quota: NBIO on -1 failed (Bad file descriptor)
[2018-02-02 11:23:29.817163] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/gluster/1045fe63-bf09-42f5-986b-ce2d3d63def0/test_vol: skip format check for non-addr auth option auth.login./data/gluster/1045fe63-bf09-42f5-986b-ce2d3d63def0/test_vol.allow
[2018-02-02 11:23:29.817167] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/gluster/1045fe63-bf09-42f5-986b-ce2d3d63def0/test_vol: skip format check for non-addr auth option auth.login.ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16.password
[2018-02-02 11:23:29.817424] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817472] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817515] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817554] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817594] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817633] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817675] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817717] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817763] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817808] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817850] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817896] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817941] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.817985] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.818026] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.864818] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.864868] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-13570-2018/01/13-10:33:03:552250-test_vol-client-1-13-1 (version: 3.7.20)
[2018-02-02 11:23:29.869630] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.869665] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-7618-2018/01/13-10:11:59:173084-test_vol-client-1-15-1 (version: 3.7.20)
[2018-02-02 11:23:29.942042] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:29.942100] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n25-78979-2018/02/02-11:23:29:813103-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:30.627852] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:30.627912] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-18210-2018/01/14-06:17:27:450728-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:30.645858] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:30.645908] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-90382-2018/01/14-06:17:24:698757-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:30.895296] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:30.895347] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-13495-2018/01/13-10:33:01:87712-test_vol-client-1-15-0 (version: 3.7.20)
[2018-02-02 11:23:31.260628] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:31.260673] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-25170-2018/01/13-10:12:01:772309-test_vol-client-1-17-0 (version: 3.7.20)
[2018-02-02 11:23:31.531469] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-test_vol-server: disconnecting connection from f08n25-78979-2018/02/02-11:23:29:813103-test_vol-client-1-0-0
[2018-02-02 11:23:31.531543] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000001 held by {client=0xffff9c105920, pid=-100 lk-owner=6c005a69ffff0000}
[2018-02-02 11:23:31.531577] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000005 held by {client=0xffff9c105920, pid=-100 lk-owner=68055a69ffff0000}
[2018-02-02 11:23:31.531604] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000006 held by {client=0xffff9c105920, pid=-100 lk-owner=640a5a69ffff0000}
[2018-02-02 11:23:31.531651] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /
[2018-02-02 11:23:31.531717] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan
[2018-02-02 11:23:31.531767] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan/internal_op
[2018-02-02 11:23:31.531829] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-test_vol-server: Shutting down connection f08n25-78979-2018/02/02-11:23:29:813103-test_vol-client-1-0-0
[2018-02-02 11:23:31.650679] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:31.650736] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-24929-2018/01/13-10:10:52:662544-test_vol-client-1-23-0 (version: 3.7.20)
[2018-02-02 11:23:31.680844] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:31.680891] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-18222-2018/01/14-06:17:28:502143-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:31.696391] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:31.696440] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-7618-2018/01/13-10:11:59:173084-test_vol-client-1-21-0 (version: 3.7.20)
[2018-02-02 11:23:31.715252] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:31.715301] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-7595-2018/01/13-10:10:50:88945-test_vol-client-1-23-0 (version: 3.7.20)
[2018-02-02 11:23:31.746750] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:31.746797] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-90410-2018/01/14-06:17:25:759911-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:31.826695] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:31.826752] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-13570-2018/01/13-10:33:03:552250-test_vol-client-1-19-0 (version: 3.7.20)
[2018-02-02 11:23:32.101199] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:32.101245] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-13495-2018/01/13-10:33:01:87712-test_vol-client-1-19-0 (version: 3.7.20)
[2018-02-02 11:23:32.178380] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:32.178428] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-25170-2018/01/13-10:12:01:772309-test_vol-client-1-21-0 (version: 3.7.20)
[2018-02-02 11:23:32.863582] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:32.863672] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n25-79117-2018/02/02-11:23:32:733518-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:35.796028] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:35.796087] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-18341-2018/01/14-06:17:32:639576-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:35.802029] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:35.802064] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n25-79181-2018/02/02-11:23:35:671964-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:35.829726] I [login.c:81:gf_auth] 0-auth/login: allowed user names: ff3d40c1-1ef1-4eb3-a448-19ec3cb60f16
[2018-02-02 11:23:35.829775] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n33-90605-2018/01/14-06:17:29:876725-test_vol-client-1-0-0 (version: 3.7.20)
[2018-02-02 11:23:36.842225] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-test_vol-server: disconnecting connection from f08n29-18222-2018/01/14-06:17:28:502143-test_vol-client-1-0-0
[2018-02-02 11:23:36.842297] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000005 held by {client=0xffff9c109f50, pid=-100 lk-owner=6c00af7effff0000}
[2018-02-02 11:23:36.842330] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000006 held by {client=0xffff9c109f50, pid=-100 lk-owner=6805af7effff0000}
[2018-02-02 11:23:36.842366] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /
[2018-02-02 11:23:36.842422] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan
[2018-02-02 11:23:36.842524] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan/internal_op
[2018-02-02 11:23:36.842621] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-test_vol-server: Shutting down connection f08n29-18222-2018/01/14-06:17:28:502143-test_vol-client-1-0-0
[2018-02-02 11:23:36.946546] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-test_vol-server: disconnecting connection from f08n33-90410-2018/01/14-06:17:25:759911-test_vol-client-1-0-0
[2018-02-02 11:23:36.946631] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000001 held by {client=0xffff9c1123e0, pid=-100 lk-owner=6c006587ffff0000}
[2018-02-02 11:23:36.946664] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000005 held by {client=0xffff9c1123e0, pid=-100 lk-owner=68056587ffff0000}
[2018-02-02 11:23:36.946690] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-test_vol-server: releasing lock on 00000000-0000-0000-0000-000000000006 held by {client=0xffff9c1123e0, pid=-100 lk-owner=640a6587ffff0000}
[2018-02-02 11:23:36.946733] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /
[2018-02-02 11:23:36.946788] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan
[2018-02-02 11:23:36.946948] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-test_vol-server: fd cleanup on /.trashcan/internal_op
[2018-02-02 11:23:36.947037] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-test_vol-server: Shutting down connection f08n33-90410-2018/01/14-06:17:25:759911-test_vol-client-1-0-0
[2018-02-02 11:23:39.222539] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2018-02-02 11:23:39.448907] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2018-02-02 11:23:39.493807] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2018-02-02 11:23:39.519128] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2018-02-02 11:28:57.534578] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-test_vol-server: accepted client from f08n29-19277-2018/01/14-06:22:54:356470-test_vol-client-1-0-0 (version: 3.7.20)
[root at f08n25glusterfs]# tail -n 100 etc-glusterfs-glusterd.vol.log
[2018-02-02 11:21:56.188669] I [MSGID: 106540] [glusterd-utils.c:4409:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered MOUNTV3 successfully
[2018-02-02 11:21:56.190907] I [MSGID: 106540] [glusterd-utils.c:4418:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered MOUNTV1 successfully
[2018-02-02 11:21:56.193112] I [MSGID: 106540] [glusterd-utils.c:4427:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NFSV3 successfully
[2018-02-02 11:21:56.195264] I [MSGID: 106540] [glusterd-utils.c:4436:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NLM v4 successfully
[2018-02-02 11:21:56.197410] I [MSGID: 106540] [glusterd-utils.c:4445:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NLM v1 successfully
[2018-02-02 11:21:56.199545] I [MSGID: 106540] [glusterd-utils.c:4454:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered ACL v3 successfully
[2018-02-02 11:21:58.201432] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2018-02-02 11:21:58.201589] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2018-02-02 11:22:00.302648] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:22:32.998693] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:23:24.792241] W [common-utils.c:1685:gf_string2boolean] (-->/usr/lib64/glusterfs/3.7.20/xlator/mgmt/glusterd.so(gd_mgmt_v3_commit_fn+0x1f0) [0xffff789b7230] -->/usr/lib64/glusterfs/3.7.20/xlator/mgmt/glusterd.so(glusterd_op_start_volume+0x290) [0xffff789816fc] -->/lib64/libglusterfs.so.0(gf_string2boolean+0x158) [0xffff82e676f0] ) 0-management: argument invalid [Invalid argument]
[2018-02-02 11:23:24.967600] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/1045fe63-bf09-42f5-986b-ce2d3d63def0/test_vol on port 49163
[2018-02-02 11:23:24.971147] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:25.080827] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/22125620-6c93-40c3-9df0-962db4a347ac/test_vol on port 49164
[2018-02-02 11:23:25.082455] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:25.229069] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/223ff77c-2828-4f8e-ba38-21778227d135/test_vol on port 49165
[2018-02-02 11:23:25.230542] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:25.395235] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/22ed6b17-125a-4a17-b712-e7c9c4831105/test_vol on port 49166
[2018-02-02 11:23:25.396795] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:25.536292] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/2401811b-351a-4083-933a-d990dd11ef06/test_vol on port 49167
[2018-02-02 11:23:25.537817] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:25.693177] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/2fc12dde-54e1-4af2-a51e-70243bff72bd/test_vol on port 49168
[2018-02-02 11:23:25.694751] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:25.798878] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/37295c8b-fa3e-41c4-b10f-e7eba647bb35/test_vol on port 49169
[2018-02-02 11:23:25.800429] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:25.923224] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/4fe51006-a5bd-49c3-9eb7-ef2886184b46/test_vol on port 49170
[2018-02-02 11:23:25.924810] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:26.074144] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/58c1f8f7-7725-42d8-ab12-24b54476f48c/test_vol on port 49171
[2018-02-02 11:23:26.075499] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:26.181729] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/a08c683c-02bb-4395-bc19-8e1a0e03f62a/test_vol on port 49172
[2018-02-02 11:23:26.183043] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:26.332312] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/a2847ee4-d987-48ae-8fad-e17bd3e3298b/test_vol on port 49173
[2018-02-02 11:23:26.333655] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:26.435919] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind] 0-pmap: adding brick /data/gluster/eca8cd11-d264-40ab-a384-4f8f56fc1e4f/test_vol on port 49174
[2018-02-02 11:23:26.437205] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-02 11:23:26.515934] I [rpc-clnt.c:1046:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2018-02-02 11:23:26.516500] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2018-02-02 11:23:26.562053] W [socket.c:3045:socket_connect] 0-nfs: Ignore failed connection attempt on /var/run/gluster/e377f3a659bcefeef4591aaf11f965a3.socket, (No such file or directory)
[2018-02-02 11:23:26.566840] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2018-02-02 11:23:26.574335] W [socket.c:3045:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/4e1c4a574d86933847ffe01f15d99b7b.socket, (No such file or directory)
[2018-02-02 11:23:26.574831] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2018-02-02 11:23:26.575059] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2018-02-02 11:23:26.579561] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd.
[2018-02-02 11:23:26.579698] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd.
[2018-02-02 11:23:28.897881] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2018-02-02 11:23:39.449912] W [dict.c:1452:dict_get_with_ref] (-->/usr/lib64/glusterfs/3.7.20/xlator/mgmt/glusterd.so(build_shd_graph+0x64) [0xffff78955628] -->/lib64/libglusterfs.so.0(dict_get_str_boolean+0x34) [0xffff82e41b30] -->/lib64/libglusterfs.so.0(+0x1f4ec) [0xffff82e3f4ec] ) 0-dict: dict OR key (graph-check) is NULL [Invalid argument]
The message "I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 3 times between [2018-02-02 11:22:32.998693] and [2018-02-02 11:23:30.704852]
[2018-02-02 11:23:39.491019] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2018-02-02 11:23:39.491341] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2018-02-02 11:23:40.395142] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:25:01.103516] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:26:31.489920] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:28:01.886988] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:29:32.202222] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:31:02.603808] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:32:33.004987] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:34:03.400885] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:35:33.734868] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:37:04.132280] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:38:34.524441] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:40:04.854724] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:41:35.233183] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:43:05.637442] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:44:35.954183] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:46:06.349265] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:47:36.747630] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:49:07.143301] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:50:37.537960] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:52:07.934435] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:53:38.277513] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:55:08.620253] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:56:38.924622] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:58:09.319604] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 11:59:39.712464] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:01:10.103154] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:02:40.500445] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:04:10.858003] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:05:41.184019] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:07:11.571314] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:08:41.966829] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:10:12.363095] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:11:42.757728] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:13:13.155581] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:14:43.553737] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:16:13.910490] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:17:44.306432] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:19:14.645538] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:20:44.962022] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:22:15.359878] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:23:45.739255] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:25:16.140107] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:26:46.539513] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:28:16.880023] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:29:47.257474] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:31:17.578904] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:32:47.898360] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:34:18.285139] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:35:48.677987] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:37:19.076224] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-02-02 12:38:49.425375] I [MSGID: 106487] [glusterd-handler.c:1472:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180114/35ff7d36/attachment.html>
More information about the Gluster-users
mailing list