[Gluster-users] after upgrade to 3.6.7 : Internal error xfs_attr3_leaf_write_verify
Saravanakumar Arumugam
sarumuga at redhat.com
Sun Dec 6 18:00:34 UTC 2015
Hi,
This seems like XFS filesystem issue.
Can you communicate this error to xfs mailing list?
Thanks,
Saravana
On 12/06/2015 05:23 AM, Julius Thomas wrote:
> Dear Gluster Users,
>
> after fixing the problem in the last mail from my colleague by
> upgrading to kernel 3.19.0-39-generic in case of changes with this bug
> in the xfs tree,
> the xfs filesystem crashes again after 4 - 5 hours on several peers.
>
> Has anyone recommendations for fixing this problems?
> Are there known issues with xfs and ubuntu 14.04?
>
> What is the latest stable release of gluster3, v3.6.3?
>
You can find latest gluster here.
http://download.gluster.org/pub/gluster/glusterfs/LATEST/
and follow the link here for Ubuntu:
http://download.gluster.org/pub/gluster/glusterfs/LATEST/Ubuntu/
> Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018838] XFS (sdc1):
> Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe5/0x100
> [xfs], block 0x44458e670
> Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018879] XFS (sdc1):
> Unmount and run xfs_repair
> Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018895] XFS (sdc1):
> First 64 bytes of corrupted metadata buffer:
> Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018916]
> ffff880417ff3000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00
> ................
> Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018956]
> ffff880417ff3010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00
> ..... ..........
> Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.018984]
> ffff880417ff3020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ................
> Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.019011]
> ffff880417ff3030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ................
> Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.019041] XFS (sdc1):
> xfs_do_force_shutdown(0x8) called from line 1249 of file
> /build/linux-lts-vivid-1jarlV/linux-lts-vivid-3.19.0/fs/xfs/xfs_buf.c.
> Return address = 0xffffffffc02bbd22
> Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.019044] XFS (sdc1):
> Corruption of in-memory data detected. Shutting down filesystem
> Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.019069] XFS (sdc1):
> Please umount the filesystem and rectify the problem(s)
> Dec 5 21:14:48 gluster-ger-ber-11 kernel: [16564.069906] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:15:08 gluster-ger-ber-11 gluster-export[4447]: [2015-12-05
> 21:15:08.797327] M
> [posix-helpers.c:1559:posix_health_check_thread_proc]
> 0-ger-ber-01-posix: health-check failed, going down
> Dec 5 21:15:18 gluster-ger-ber-11 kernel: [16594.068660] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:15:38 gluster-ger-ber-11 gluster-export[4447]: [2015-12-05
> 21:15:38.797422] M
> [posix-helpers.c:1564:posix_health_check_thread_proc]
> 0-ger-ber-01-posix: still alive! -> SIGTERM
> Dec 5 21:15:48 gluster-ger-ber-11 kernel: [16624.119428] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:16:18 gluster-ger-ber-11 kernel: [16654.170134] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:16:48 gluster-ger-ber-11 kernel: [16684.220834] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:17:01 gluster-ger-ber-11 CRON[17656]: (root) CMD ( cd / &&
> run-parts --report /etc/cron.hourly)
> Dec 5 21:17:18 gluster-ger-ber-11 kernel: [16714.271507] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:17:48 gluster-ger-ber-11 kernel: [16744.322244] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:18:18 gluster-ger-ber-11 kernel: [16774.372948] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:18:48 gluster-ger-ber-11 kernel: [16804.423650] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:19:18 gluster-ger-ber-11 kernel: [16834.474365] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:19:48 gluster-ger-ber-11 kernel: [16864.525082] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:20:18 gluster-ger-ber-11 kernel: [16894.575778] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:20:49 gluster-ger-ber-11 kernel: [16924.626464] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:21:19 gluster-ger-ber-11 kernel: [16954.677161] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:21:49 gluster-ger-ber-11 kernel: [16984.727791] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:22:19 gluster-ger-ber-11 kernel: [17014.778570] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:22:49 gluster-ger-ber-11 kernel: [17044.829240] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:23:19 gluster-ger-ber-11 kernel: [17074.880003] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:23:49 gluster-ger-ber-11 kernel: [17104.930643] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:24:19 gluster-ger-ber-11 kernel: [17134.981336] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:24:49 gluster-ger-ber-11 kernel: [17165.032049] XFS (sdc1):
> xfs_log_force: error -5 returned.
> Dec 5 21:25:19 gluster-ger-ber-11 kernel: [17195.082689] XFS (sdc1):
> xfs_log_force: error -5 returned.
>
> On 03.12.2015 12:06, Dietmar Putz wrote:
>> Hello all,
>>
>> on 1st december i upgraded two 6 node cluster from glusterfs 3.5.6 to
>> 3.6.7.
>> all of them are equal in hw, os and patchlevel, currently running
>> ubuntu 14.04 lts by an do-release-upgrade from 12.04 lts (this was
>> done before gfs upgrade to 3.5.6, not directly before upgrading to
>> 3.6.7).
>> because of a geo-replication issue all of the nodes have rsync
>> 3.1.1.3 installed instead 3.1.0 which comes by the repositories.
>> this is the only deviation from ubuntu repositories for 14.04 lts.
>> since upgrade to gfs 3.6.7 the glusterd on two nodes of the same
>> cluster are going offline after getting an
>> xfs_attr3_leaf_write_verify error for the underlying bricks as shown
>> below.
>> this happens about every 4-5 hours after the problem was solved by an
>> umount / remount of the brick. it makes no difference to run a
>> xfs_check / xfs_repair before remount.
>> xfs_check / xfs_repair did not show any faults. the underlying hw is
>> a raid 5 vol on lsi-9271 8i. megacli does not show any errors.
>> the syslog does not show more than the dmesg output below.
>> every time the same two nodes of the same cluster are affected.
>> as shown in dmesg and syslog, the system recognizes the
>> xfs_attr_leaf_write_verify error about 38 min. before finally giving
>> up. for both events i can not found corresponding events in gluster
>> logs.
>> this is strange...the gluster is historical grown from 3.2.5, 3.3, to
>> 3.4.6/7 which was running well for month, gfs 3.5.6 was running for
>> about two weeks and upgrade to 3.6.7 was done because of a geo-repl
>> log-flood.
>> even when i have no hint/evidence that this is caused by gfs 3.6.7
>> somehow i believe that this is the case...
>> does anybody experienced such an error or have some hints to getting
>> out of this big problem...?
>> unfortunately the affected cluster is the master of a geo-replication
>> which is not well running since update from gfs 3.4.7...fortunately
>> both affected gluster-nodes are not of the same sub-volume.
>>
>> any help is appreciated...
>>
>> best regards
>> dietmar
>>
>>
>>
>>
>> [ 09:32:29 ] - root at gluster-ger-ber-10 /var/log $gluster volume info
>>
>> Volume Name: ger-ber-01
>> Type: Distributed-Replicate
>> Volume ID: 6a071cfa-b150-4f0b-b1ed-96ab5d4bd671
>> Status: Started
>> Number of Bricks: 3 x 2 = 6
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster-ger-ber-11-int:/gluster-export
>> Brick2: gluster-ger-ber-12-int:/gluster-export
>> Brick3: gluster-ger-ber-09-int:/gluster-export
>> Brick4: gluster-ger-ber-10-int:/gluster-export
>> Brick5: gluster-ger-ber-07-int:/gluster-export
>> Brick6: gluster-ger-ber-08-int:/gluster-export
>> Options Reconfigured:
>> changelog.changelog: on
>> geo-replication.ignore-pid-check: on
>> cluster.min-free-disk: 200GB
>> geo-replication.indexing: on
>> auth.allow:
>> 10.0.1.*,188.138.82.*,188.138.123.*,82.193.249.198,82.193.249.200,31.7.178.137,31.7.178.135,31.7.180.109,31.7.180.98,82.199.147.*,104.155.22.202,104.155.30.201,104.155.5.117,104.155.11.253,104.155.15.34,104.155.25.145,146.148.120.255,31.7.180.148
>> nfs.disable: off
>> performance.cache-refresh-timeout: 2
>> performance.io-thread-count: 32
>> performance.cache-size: 1024MB
>> performance.read-ahead: on
>> performance.cache-min-file-size: 0
>> network.ping-timeout: 10
>> [ 09:32:52 ] - root at gluster-ger-ber-10 /var/log $
>>
>>
>>
>>
>> [ 19:10:55 ] - root at gluster-ger-ber-10 /var/log $gluster volume status
>> Status of volume: ger-ber-01
>> Gluster process Port Online Pid
>> ------------------------------------------------------------------------------
>>
>> Brick gluster-ger-ber-11-int:/gluster-export 49152 Y 15994
>> Brick gluster-ger-ber-12-int:/gluster-export N/A N N/A
>> Brick gluster-ger-ber-09-int:/gluster-export 49152 Y 10965
>> Brick gluster-ger-ber-10-int:/gluster-export N/A N N/A
>> Brick gluster-ger-ber-07-int:/gluster-export 49152 Y 18542
>> Brick gluster-ger-ber-08-int:/gluster-export 49152 Y 20275
>> NFS Server on localhost 2049 Y 13658
>> Self-heal Daemon on localhost N/A Y 13666
>> NFS Server on gluster-ger-ber-09-int 2049 Y 13503
>> Self-heal Daemon on gluster-ger-ber-09-int N/A Y 13511
>> NFS Server on gluster-ger-ber-07-int 2049 Y 21526
>> Self-heal Daemon on gluster-ger-ber-07-int N/A Y 21534
>> NFS Server on gluster-ger-ber-08-int 2049 Y 24004
>> Self-heal Daemon on gluster-ger-ber-08-int N/A Y 24011
>> NFS Server on gluster-ger-ber-11-int 2049 Y 18944
>> Self-heal Daemon on gluster-ger-ber-11-int N/A Y 18952
>> NFS Server on gluster-ger-ber-12-int 2049 Y 19138
>> Self-heal Daemon on gluster-ger-ber-12-int N/A Y 19146
>>
>> Task Status of Volume ger-ber-01
>> ------------------------------------------------------------------------------
>>
>> There are no active volume tasks
>>
>> - root at gluster-ger-ber-10 /var/log $
>>
>> - root at gluster-ger-ber-10 /var/log $dmesg -T
>> ...
>> [Wed Dec 2 12:43:47 2015] XFS (sdc1): xfs_log_force: error 5 returned.
>> [Wed Dec 2 12:43:48 2015] XFS (sdc1): xfs_log_force: error 5 returned.
>> [Wed Dec 2 12:45:58 2015] XFS (sdc1): Mounting Filesystem
>> [Wed Dec 2 12:45:58 2015] XFS (sdc1): Starting recovery (logdev:
>> internal)
>> [Wed Dec 2 12:45:59 2015] XFS (sdc1): Ending recovery (logdev:
>> internal)
>> [Wed Dec 2 13:11:53 2015] XFS (sdc1): Mounting Filesystem
>> [Wed Dec 2 13:11:54 2015] XFS (sdc1): Ending clean mount
>> [Wed Dec 2 13:12:29 2015] init: statd main process (25924) killed by
>> KILL signal
>> [Wed Dec 2 13:12:29 2015] init: statd main process ended, respawning
>> [Wed Dec 2 13:13:24 2015] init: statd main process (13433) killed by
>> KILL signal
>> [Wed Dec 2 13:13:24 2015] init: statd main process ended, respawning
>> [Wed Dec 2 17:22:28 2015] ffff8807076b1000: 00 00 00 00 00 00 00 00
>> fb ee 00 00 00 00 00 00 ................
>> [Wed Dec 2 17:22:28 2015] ffff8807076b1010: 10 00 00 00 00 20 0f e0
>> 00 00 00 00 00 00 00 00 ..... ..........
>> [Wed Dec 2 17:22:28 2015] ffff8807076b1020: 00 00 00 00 00 00 00 00
>> 00 00 00 00 00 00 00 00 ................
>> [Wed Dec 2 17:22:28 2015] ffff8807076b1030: 00 00 00 00 00 00 00 00
>> 00 00 00 00 00 00 00 00 ................
>> [Wed Dec 2 17:22:28 2015] XFS (sdc1): Internal error
>> xfs_attr3_leaf_write_verify at line 216 of file
>> /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa01a66f0
>> [Wed Dec 2 17:22:28 2015] CPU: 4 PID: 13162 Comm: xfsaild/sdc1 Not
>> tainted 3.13.0-67-generic #110-Ubuntu
>> [Wed Dec 2 17:22:28 2015] Hardware name: Supermicro
>> X10SLL-F/X10SLL-F, BIOS 1.1b 11/01/2013
>> [Wed Dec 2 17:22:28 2015] 0000000000000001 ffff8801c5691bd0
>> ffffffff817240e0 ffff8801b15c3800
>> [Wed Dec 2 17:22:28 2015] ffff8801c5691be8 ffffffffa01aa6fb
>> ffffffffa01a66f0 ffff8801c5691c20
>> [Wed Dec 2 17:22:28 2015] ffffffffa01aa755 000000d800200200
>> ffff8804a59ac780 ffff8800d917e658
>> [Wed Dec 2 17:22:28 2015] Call Trace:
>> [Wed Dec 2 17:22:28 2015] [<ffffffff817240e0>] dump_stack+0x45/0x56
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01aa6fb>]
>> xfs_error_report+0x3b/0x40 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>] ?
>> _xfs_buf_ioapply+0x70/0x3a0 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01aa755>]
>> xfs_corruption_error+0x55/0x80 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01c7b70>]
>> xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>] ?
>> _xfs_buf_ioapply+0x70/0x3a0 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>] ?
>> xfs_bdstrat_cb+0x55/0xb0 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01a66f0>]
>> _xfs_buf_ioapply+0x70/0x3a0 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffff8109ac90>] ?
>> wake_up_state+0x20/0x20
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>] ?
>> xfs_bdstrat_cb+0x55/0xb0 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01a8336>]
>> xfs_buf_iorequest+0x46/0x90 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01a83d5>]
>> xfs_bdstrat_cb+0x55/0xb0 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01a856b>]
>> __xfs_buf_delwri_submit+0x13b/0x210 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01a9000>] ?
>> xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa0207af0>] ?
>> xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa01a9000>]
>> xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa0207d27>] xfsaild+0x237/0x5c0
>> [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffffa0207af0>] ?
>> xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
>> [Wed Dec 2 17:22:28 2015] [<ffffffff8108b7d2>] kthread+0xd2/0xf0
>> [Wed Dec 2 17:22:28 2015] [<ffffffff8108b700>] ?
>> kthread_create_on_node+0x1c0/0x1c0
>> [Wed Dec 2 17:22:28 2015] [<ffffffff81734c28>] ret_from_fork+0x58/0x90
>> [Wed Dec 2 17:22:28 2015] [<ffffffff8108b700>] ?
>> kthread_create_on_node+0x1c0/0x1c0
>> [Wed Dec 2 17:22:28 2015] XFS (sdc1): Corruption detected. Unmount
>> and run xfs_repair
>> [Wed Dec 2 17:22:28 2015] XFS (sdc1): xfs_do_force_shutdown(0x8)
>> called from line 1320 of file
>> /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_buf.c. Return address =
>> 0xffffffffa01a671c
>> [Wed Dec 2 17:22:28 2015] XFS (sdc1): Corruption of in-memory data
>> detected. Shutting down filesystem
>> [Wed Dec 2 17:22:28 2015] XFS (sdc1): Please umount the filesystem
>> and rectify the problem(s)
>> [Wed Dec 2 17:22:28 2015] XFS (sdc1): xfs_log_force: error 5 returned.
>> [Wed Dec 2 17:22:49 2015] XFS (sdc1): xfs_log_force: error 5 returned.
>> ...
>>
>> [ 19:10:49 ] - root at gluster-ger-ber-10 /var/log $xfs_info
>> /gluster-export
>> meta-data=/dev/sdc1 isize=256 agcount=32,
>> agsize=152596472 blks
>> = sectsz=512 attr=2
>> data = bsize=4096 blocks=4883087099, imaxpct=5
>> = sunit=0 swidth=0 blks
>> naming =version 2 bsize=4096 ascii-ci=0
>> log =internal bsize=4096 blocks=521728, version=2
>> = sectsz=512 sunit=0 blks, lazy-count=1
>> realtime =none extsz=4096 blocks=0, rtextents=0
>> [ 19:10:55 ] - root at gluster-ger-ber-10 /var/log $
>>
>> [ 09:36:37 ] - root at gluster-ger-ber-10 /var/log $stat /gluster-export
>> stat: cannot stat ‘/gluster-export’: Input/output error
>> [ 09:36:45 ] - root at gluster-ger-ber-10 /var/log $
>>
>>
>> [ 08:50:43 ] - root at gluster-ger-ber-10 ~/tmp/syslog $dmesg -T | grep
>> xfs_attr3_leaf_write_verify
>> [Di Dez 1 23:24:53 2015] XFS (sdc1): Internal error
>> xfs_attr3_leaf_write_verify at line 216 of file
>> /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa01a66f0
>> [Di Dez 1 23:24:53 2015] [<ffffffffa01c7b70>]
>> xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> [Mi Dez 2 12:19:16 2015] XFS (sdc1): Internal error
>> xfs_attr3_leaf_write_verify at line 216 of file
>> /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa01a66f0
>> [Mi Dez 2 12:19:16 2015] [<ffffffffa01c7b70>]
>> xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> [Mi Dez 2 17:22:28 2015] XFS (sdc1): Internal error
>> xfs_attr3_leaf_write_verify at line 216 of file
>> /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa01a66f0
>> [Mi Dez 2 17:22:28 2015] [<ffffffffa01c7b70>]
>> xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> [Mi Dez 2 23:06:32 2015] XFS (sdc1): Internal error
>> xfs_attr3_leaf_write_verify at line 216 of file
>> /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa01a66f0
>> [Mi Dez 2 23:06:32 2015] [<ffffffffa01c7b70>]
>> xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>>
>> [ 08:06:28 ] - root at gluster-ger-ber-10
>> /var/log/glusterfs/geo-replication $grep xfs_attr3_leaf_write_verify
>> /root/tmp/syslog/syslog*
>> Dec 2 00:01:50 gluster-ger-ber-10 kernel: [2278489.906268] XFS
>> (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of
>> file /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa01a66f0
>> Dec 2 00:01:50 gluster-ger-ber-10 kernel: [2278489.906448]
>> [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> Dec 2 12:56:57 gluster-ger-ber-10 kernel: [2324952.509891] XFS
>> (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of
>> file /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa01a66f0
>> Dec 2 12:56:57 gluster-ger-ber-10 kernel: [2324952.510414]
>> [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> xfs_check
>> xfs_repair -> no fault
>> Dec 2 18:00:27 gluster-ger-ber-10 kernel: [2343144.298098] XFS
>> (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of
>> file /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa01a66f0
>> Dec 2 18:00:27 gluster-ger-ber-10 kernel: [2343144.298259]
>> [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> Dec 2 23:44:52 gluster-ger-ber-10 kernel: [2363788.969849] XFS
>> (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of
>> file /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa01a66f0
>> Dec 2 23:44:52 gluster-ger-ber-10 kernel: [2363788.970217]
>> [<ffffffffa01c7b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> [ 08:06:37 ] - root at gluster-ger-ber-10
>> /var/log/glusterfs/geo-replication $
>>
>> [ 08:04:51 ] - root at gluster-ger-ber-12 ~/tmp/syslog $grep
>> xfs_attr3_leaf_write_verify syslog*
>> Dec 2 00:01:10 gluster-ger-ber-12 kernel: [2276785.772229] XFS
>> (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of
>> file /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa019a6f0
>> Dec 2 00:01:10 gluster-ger-ber-12 kernel: [2276785.772504]
>> [<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> Dec 2 12:59:08 gluster-ger-ber-12 kernel: [2323418.198659] XFS
>> (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of
>> file /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa019a6f0
>> Dec 2 12:59:08 gluster-ger-ber-12 kernel: [2323418.199085]
>> [<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> xfs_check
>> xfs_repair -> no fault
>> Dec 2 18:30:47 gluster-ger-ber-12 kernel: [2343298.342473] XFS
>> (sdc1): Internal error xfs_attr3_leaf_write_verify at line 216 of
>> file /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa019a6f0
>> Dec 2 18:30:47 gluster-ger-ber-12 kernel: [2343298.342850]
>> [<ffffffffa01bbb70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> Dec 2 23:48:38 gluster-ger-ber-12 kernel: [15001.493190] XFS (sdc1):
>> Internal error xfs_attr3_leaf_write_verify at line 216 of file
>> /build/linux-XHaR1x/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller
>> 0xffffffffa01936f0
>> Dec 2 23:48:38 gluster-ger-ber-12 kernel: [15001.493550]
>> [<ffffffffa01b4b70>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
>> [ 08:05:02 ] - root at gluster-ger-ber-12 ~/tmp/syslog $
>>
>> gluster-ger-ber-10-int:
>> glustershd.log :
>> [2015-12-02 23:45:33.160852] W [socket.c:620:__socket_rwv]
>> 0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data
>> available)
>> [2015-12-02 23:45:33.170590] I [client.c:2203:client_rpc_notify]
>> 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client
>> process will keep trying to connect to glusterd until brick's port is
>> available
>> [2015-12-02 23:45:43.784388] E
>> [client-handshake.c:1496:client_query_portmap_cbk]
>> 0-ger-ber-01-client-3: failed to get the port number for remote
>> subvolume. Please run 'gluster volume status' on server to see if
>> brick process is running.
>> [2015-12-02 23:45:43.784543] I [client.c:2203:client_rpc_notify]
>> 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client
>> process will keep trying to connect to glusterd until brick's port is
>> available
>> [2015-12-02 23:45:50.000203] W
>> [client-rpc-fops.c:1090:client3_3_getxattr_cbk]
>> 0-ger-ber-01-client-3: remote operation failed: Transport endpoint is
>> not connected. Path: / (00000000-0000-0000-0000-000000000001). Key:
>> trusted.glusterfs.pathinfo
>> [2015-12-02 23:49:33.524740] W [socket.c:620:__socket_rwv]
>> 0-ger-ber-01-client-1: readv on 10.0.1.107:49152 failed (No data
>> available)
>> [2015-12-02 23:49:33.524934] I [client.c:2203:client_rpc_notify]
>> 0-ger-ber-01-client-1: disconnected from ger-ber-01-client-1. Client
>> process will keep trying to connect to glusterd until brick's port is
>> available
>> [2015-12-02 23:49:43.882976] E
>> [client-handshake.c:1496:client_query_portmap_cbk]
>> 0-ger-ber-01-client-1: failed to get the port number for remote
>> subvolume. Please run 'gluster volume status' on server to see if
>> brick process is running.
>>
>> sdn.log :
>> [2015-12-02 23:45:33.160963] W [socket.c:620:__socket_rwv]
>> 0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data
>> available)
>> [2015-12-02 23:45:33.168504] I [client.c:2203:client_rpc_notify]
>> 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client
>> process will keep trying to connect to glusterd until brick's port is
>> available
>> [2015-12-02 23:45:43.395787] E
>> [client-handshake.c:1496:client_query_portmap_cbk]
>> 0-ger-ber-01-client-3: failed to get the port number for remote
>> subvolume. Please run 'gluster volume status' on server to see if
>> brick process is running.
>>
>> nfs.log :
>> [2015-12-02 23:45:33.160856] W [socket.c:620:__socket_rwv]
>> 0-ger-ber-01-client-3: readv on 10.0.1.103:49152 failed (No data
>> available)
>> [2015-12-02 23:45:33.180366] I [client.c:2203:client_rpc_notify]
>> 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client
>> process will keep trying to connect to glusterd until brick's port is
>> available
>> [2015-12-02 23:45:43.780186] E
>> [client-handshake.c:1496:client_query_portmap_cbk]
>> 0-ger-ber-01-client-3: failed to get the port number for remote
>> subvolume. Please run 'gluster volume status' on server to see if
>> brick process is running.
>> [2015-12-02 23:45:43.780340] I [client.c:2203:client_rpc_notify]
>> 0-ger-ber-01-client-3: disconnected from ger-ber-01-client-3. Client
>> process will keep trying to connect to glusterd until brick's port is
>> available
>>
>> geo-replication log :
>> [2015-12-02 23:44:34.624957] I
>> [master(/gluster-export):514:crawlwrap] _GMaster: 0 crawls, 0 turns
>> [2015-12-02 23:44:54.798414] E
>> [syncdutils(/gluster-export):270:log_raise_exception] <top>: FAIL:
>> Traceback (most recent call last):
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",
>> line 164, in main main_i()
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/gsyncd.py",
>> line 643, in main_i local.service_loop(*[r for r in [remote] if r])
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
>> line 1325, in service_loop g3.crawlwrap(oneshot=True)
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
>> line 527, in crawlwrap brick_stime = self.xtime('.', self.slave)
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
>> line 362, in xtime return self.xtime_low(rsc, path, **opts)
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/master.py",
>> line 132, in xtime_low xt = rsc.server.stime(path, self.uuid)
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
>> line 1259, in <lambda> uuid + '.' + gconf.slave_id)
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
>> line 322, in ff return f(*a)
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/resource.py",
>> line 510, in stime 8)
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py",
>> line 55, in lgetxattr return cls._query_xattr(path, siz,
>> 'lgetxattr', attr)
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py",
>> line 47, in _query_xattr cls.raise_oserr()
>> File
>> "/usr/lib/x86_64-linux-gnu/glusterfs/python/syncdaemon/libcxattr.py",
>> line 37, in raise_oserr raise OSError(errn, os.strerror(errn))
>> OSError: [Errno 5] Input/output error
>> [2015-12-02 23:44:54.845763] I
>> [syncdutils(/gluster-export):214:finalize] <top>: exiting.
>> [2015-12-02 23:44:54.847527] I [repce(agent):92:service_loop]
>> RepceServer: terminating on reaching EOF.
>> [2015-12-02 23:44:54.847784] I [syncdutils(agent):214:finalize]
>> <top>: exiting.
>> [2015-12-02 23:44:54.849092] I [monitor(monitor):141:set_state]
>> Monitor: new state: faulty
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list