[Gluster-users] Problems with write-behind with large files on Gluster 3.8.4

Jim Prewett download at carc.unm.edu
Mon Feb 26 21:19:03 UTC 2018


Hello,

I'm having problems when write-behind is enabled on Gluster 3.8.4.

I have 2 Gluster servers each with a single brick that is mirrored between 
them.  The code causing these issues reads two data files each approx. 
128G in size.  It opens a third file, mmap()'s that file, and 
subsequently reads and writes to it.  The third file, on sucessful runs 
(without write-behind enabled) is ultimately approx. 224G in size.

The servers have the IP addresses 172.17.2.254 and 172.17.2.255 and the 
client has the IP address 172.17.1.61.  These are all IP over InfiniBand.

I'm attaching logfiles for the brick and for the volume from each of the 
servers and for the client.  I'm also attaching the output of "gluster 
volume info" and "gluster volume get <volume> all".

I have only noticed problems with write-behind being enabled with this one 
particular workload.  When I ran it under strace, I see it seeking all 
over the place and reading and writing little bits of data to/from the 
third file.

For now, I'm leaving write-behind disabled.  What are the performance 
implications of this for jobs that don't have this strange access pattern?

My co-worker who usually maintains the Gluster filesystems here is busy 
having a baby right now and I've gotten it while he's out, so I'm /really/ 
new to Gluster and am not confident that anything is correct in my 
configuration (nor do I have a specific reason to doubt its correctness! :)

I have checked the InfiniBand fabric for errors and do not see any beyond 
the normal PortXmitWait counter.  There is no firewall on any of these 
machines.  Their system clocks seem to all be synchronized.

Is there anything additional I can provide to help diagnose this problem?

Thanks for any help you can provide! :)

Jim

James E. Prewett                    Jim at Prewett.org download at hpc.unm.edu
Systems Team Leader           LoGS: http://www.hpc.unm.edu/~download/LoGS/
Designated Security Officer         OpenPGP key: pub 1024D/31816D93
HPC Systems Engineer III   UNM HPC  505.277.8210
-------------- next part --------------
Option                                  Value                                   
------                                  -----                                   
cluster.lookup-unhashed                 on                                      
cluster.lookup-optimize                 on                                      
cluster.min-free-disk                   10%                                     
cluster.min-free-inodes                 5%                                      
cluster.rebalance-stats                 off                                     
cluster.subvols-per-directory           (null)                                  
cluster.readdir-optimize                off                                     
cluster.rsync-hash-regex                (null)                                  
cluster.extra-hash-regex                (null)                                  
cluster.dht-xattr-name                  trusted.glusterfs.dht                   
cluster.randomize-hash-range-by-gfid    off                                     
cluster.rebal-throttle                  normal                                  
cluster.lock-migration                  off                                     
cluster.local-volume-name               (null)                                  
cluster.weighted-rebalance              on                                      
cluster.switch-pattern                  (null)                                  
cluster.entry-change-log                on                                      
cluster.read-subvolume                  (null)                                  
cluster.read-subvolume-index            -1                                      
cluster.read-hash-mode                  1                                       
cluster.background-self-heal-count      8                                       
cluster.metadata-self-heal              on                                      
cluster.data-self-heal                  on                                      
cluster.entry-self-heal                 on                                      
cluster.self-heal-daemon                on                                      
cluster.heal-timeout                    600                                     
cluster.self-heal-window-size           1                                       
cluster.data-change-log                 on                                      
cluster.metadata-change-log             on                                      
cluster.data-self-heal-algorithm        (null)                                  
cluster.eager-lock                      on                                      
disperse.eager-lock                     on                                      
cluster.quorum-type                     none                                    
cluster.quorum-count                    (null)                                  
cluster.choose-local                    true                                    
cluster.self-heal-readdir-size          1KB                                     
cluster.post-op-delay-secs              1                                       
cluster.ensure-durability               on                                      
cluster.consistent-metadata             no                                      
cluster.heal-wait-queue-length          128                                     
cluster.favorite-child-policy           none                                    
cluster.stripe-block-size               128KB                                   
cluster.stripe-coalesce                 true                                    
diagnostics.latency-measurement         on                                      
diagnostics.dump-fd-stats               off                                     
diagnostics.count-fop-hits              on                                      
diagnostics.brick-log-level             INFO                                    
diagnostics.client-log-level            INFO                                    
diagnostics.brick-sys-log-level         CRITICAL                                
diagnostics.client-sys-log-level        CRITICAL                                
diagnostics.brick-logger                (null)                                  
diagnostics.client-logger               (null)                                  
diagnostics.brick-log-format            (null)                                  
diagnostics.client-log-format           (null)                                  
diagnostics.brick-log-buf-size          5                                       
diagnostics.client-log-buf-size         5                                       
diagnostics.brick-log-flush-timeout     120                                     
diagnostics.client-log-flush-timeout    120                                     
diagnostics.stats-dump-interval         0                                       
diagnostics.fop-sample-interval         0                                       
diagnostics.fop-sample-buf-size         65535                                   
diagnostics.stats-dnscache-ttl-sec      86400                                   
performance.cache-max-file-size         0                                       
performance.cache-min-file-size         0                                       
performance.cache-refresh-timeout       1                                       
performance.cache-priority                                                      
performance.cache-size                  8GB                                     
performance.io-thread-count             24                                      
performance.high-prio-threads           16                                      
performance.normal-prio-threads         16                                      
performance.low-prio-threads            16                                      
performance.least-prio-threads          1                                       
performance.enable-least-priority       on                                      
performance.least-rate-limit            0                                       
performance.cache-size                  8GB                                     
performance.flush-behind                on                                      
performance.nfs.flush-behind            on                                      
performance.write-behind-window-size    8MB                                     
performance.resync-failed-syncs-after-fsyncon                                      
performance.nfs.write-behind-window-size1MB                                     
performance.strict-o-direct             off                                     
performance.nfs.strict-o-direct         off                                     
performance.strict-write-ordering       off                                     
performance.nfs.strict-write-ordering   off                                     
performance.lazy-open                   yes                                     
performance.read-after-open             no                                      
performance.read-ahead-page-count       4                                       
performance.md-cache-timeout            1                                       
performance.cache-swift-metadata        true                                    
features.encryption                     off                                     
encryption.master-key                   (null)                                  
encryption.data-key-size                256                                     
encryption.block-size                   4096                                    
network.frame-timeout                   1800                                    
network.ping-timeout                    42                                      
network.tcp-window-size                 (null)                                  
features.lock-heal                      off                                     
features.grace-timeout                  10                                      
network.remote-dio                      disable                                 
client.event-threads                    8                                       
network.ping-timeout                    42                                      
network.tcp-window-size                 (null)                                  
network.inode-lru-limit                 16384                                   
auth.allow                              *                                       
auth.reject                             (null)                                  
transport.keepalive                     (null)                                  
server.allow-insecure                   (null)                                  
server.root-squash                      off                                     
server.anonuid                          65534                                   
server.anongid                          65534                                   
server.statedump-path                   /var/run/gluster                        
server.outstanding-rpc-limit            512                                     
features.lock-heal                      off                                     
features.grace-timeout                  10                                      
server.ssl                              (null)                                  
auth.ssl-allow                          *                                       
server.manage-gids                      off                                     
server.dynamic-auth                     on                                      
client.send-gids                        on                                      
server.gid-timeout                      300                                     
server.own-thread                       (null)                                  
server.event-threads                    32                                      
ssl.own-cert                            (null)                                  
ssl.private-key                         (null)                                  
ssl.ca-list                             (null)                                  
ssl.crl-path                            (null)                                  
ssl.certificate-depth                   (null)                                  
ssl.cipher-list                         (null)                                  
ssl.dh-param                            (null)                                  
ssl.ec-curve                            (null)                                  
performance.write-behind                off                                     
performance.read-ahead                  on                                      
performance.readdir-ahead               on                                      
performance.io-cache                    on                                      
performance.quick-read                  on                                      
performance.open-behind                 on                                      
performance.stat-prefetch               on                                      
performance.client-io-threads           on                                      
performance.nfs.write-behind            on                                      
performance.nfs.read-ahead              off                                     
performance.nfs.io-cache                off                                     
performance.nfs.quick-read              off                                     
performance.nfs.stat-prefetch           off                                     
performance.nfs.io-threads              off                                     
performance.force-readdirp              true                                    
features.uss                            off                                     
features.snapshot-directory             .snaps                                  
features.show-snapshot-directory        off                                     
network.compression                     off                                     
network.compression.window-size         -15                                     
network.compression.mem-level           8                                       
network.compression.min-size            0                                       
network.compression.compression-level   -1                                      
network.compression.debug               false                                   
features.limit-usage                    (null)                                  
features.quota-timeout                  0                                       
features.default-soft-limit             80%                                     
features.soft-timeout                   60                                      
features.hard-timeout                   5                                       
features.alert-time                     86400                                   
features.quota-deem-statfs              off                                     
geo-replication.indexing                off                                     
geo-replication.indexing                off                                     
geo-replication.ignore-pid-check        off                                     
geo-replication.ignore-pid-check        off                                     
features.quota                          off                                     
features.inode-quota                    off                                     
features.bitrot                         disable                                 
debug.trace                             off                                     
debug.log-history                       no                                      
debug.log-file                          no                                      
debug.exclude-ops                       (null)                                  
debug.include-ops                       (null)                                  
debug.error-gen                         off                                     
debug.error-failure                     (null)                                  
debug.error-number                      (null)                                  
debug.random-failure                    off                                     
debug.error-fops                        (null)                                  
nfs.enable-ino32                        no                                      
nfs.mem-factor                          15                                      
nfs.export-dirs                         on                                      
nfs.export-volumes                      on                                      
nfs.addr-namelookup                     off                                     
nfs.dynamic-volumes                     off                                     
nfs.register-with-portmap               on                                      
nfs.outstanding-rpc-limit               16                                      
nfs.port                                2049                                    
nfs.rpc-auth-unix                       on                                      
nfs.rpc-auth-null                       on                                      
nfs.rpc-auth-allow                      all                                     
nfs.rpc-auth-reject                     none                                    
nfs.ports-insecure                      off                                     
nfs.trusted-sync                        off                                     
nfs.trusted-write                       off                                     
nfs.volume-access                       read-write                              
nfs.export-dir                                                                  
nfs.disable                             on                                      
nfs.nlm                                 on                                      
nfs.acl                                 on                                      
nfs.mount-udp                           off                                     
nfs.mount-rmtab                         /var/lib/glusterd/nfs/rmtab             
nfs.rpc-statd                           /sbin/rpc.statd                         
nfs.server-aux-gids                     off                                     
nfs.drc                                 off                                     
nfs.drc-size                            0x20000                                 
nfs.read-size                           (1 * 1048576ULL)                        
nfs.write-size                          (1 * 1048576ULL)                        
nfs.readdir-size                        (1 * 1048576ULL)                        
nfs.rdirplus                            on                                      
nfs.exports-auth-enable                 (null)                                  
nfs.auth-refresh-interval-sec           (null)                                  
nfs.auth-cache-ttl-sec                  (null)                                  
features.read-only                      off                                     
features.worm                           off                                     
features.worm-file-level                off                                     
features.default-retention-period       120                                     
features.retention-mode                 relax                                   
features.auto-commit-period             180                                     
storage.linux-aio                       off                                     
storage.batch-fsync-mode                reverse-fsync                           
storage.batch-fsync-delay-usec          0                                       
storage.owner-uid                       -1                                      
storage.owner-gid                       -1                                      
storage.node-uuid-pathinfo              off                                     
storage.health-check-interval           30                                      
storage.build-pgfid                     off                                     
storage.bd-aio                          off                                     
config.transport                        tcp,rdma                                
cluster.server-quorum-type              off                                     
cluster.server-quorum-ratio             0                                       
changelog.changelog                     off                                     
changelog.changelog-dir                 (null)                                  
changelog.encoding                      ascii                                   
changelog.rollover-time                 15                                      
changelog.fsync-interval                5                                       
changelog.changelog-barrier-timeout     120                                     
changelog.capture-del-path              off                                     
features.barrier                        disable                                 
features.barrier-timeout                120                                     
features.trash                          off                                     
features.trash-dir                      .trashcan                               
features.trash-eliminate-path           (null)                                  
features.trash-max-filesize             5MB                                     
features.trash-internal-op              off                                     
cluster.enable-shared-storage           disable                                 
cluster.write-freq-threshold            0                                       
cluster.read-freq-threshold             0                                       
cluster.tier-pause                      off                                     
cluster.tier-promote-frequency          120                                     
cluster.tier-demote-frequency           3600                                    
cluster.watermark-hi                    90                                      
cluster.watermark-low                   75                                      
cluster.tier-mode                       cache                                   
cluster.tier-max-promote-file-size      0                                       
cluster.tier-max-mb                     4000                                    
cluster.tier-max-files                  10000                                   
features.ctr-enabled                    off                                     
features.record-counters                off                                     
features.ctr-record-metadata-heat       off                                     
features.ctr_link_consistency           off                                     
features.ctr_lookupheal_link_timeout    300                                     
features.ctr_lookupheal_inode_timeout   300                                     
features.ctr-sql-db-cachesize           1000                                    
features.ctr-sql-db-wal-autocheckpoint  1000                                    
locks.trace                             off                                     
locks.mandatory-locking                 off                                     
cluster.disperse-self-heal-daemon       enable                                  
cluster.quorum-reads                    no                                      
client.bind-insecure                    (null)                                  
ganesha.enable                          off                                     
features.shard                          off                                     
features.shard-block-size               4MB                                     
features.scrub-throttle                 lazy                                    
features.scrub-freq                     biweekly                                
features.scrub                          false                                   
features.expiry-time                    120                                     
features.cache-invalidation             off                                     
features.cache-invalidation-timeout     60                                      
features.leases                         off                                     
features.lease-lock-recall-timeout      60                                      
disperse.background-heals               8                                       
disperse.heal-wait-qlength              128                                     
cluster.heal-timeout                    600                                     
dht.force-readdirp                      on                                      
disperse.read-policy                    round-robin                             
cluster.shd-max-threads                 1                                       
cluster.shd-wait-qlength                1024                                    
cluster.locking-scheme                  full                                    
cluster.granular-entry-heal             no                                      
-------------- next part --------------
Volume Name: scratch
Type: Stripe
Volume ID: 38f6c032-b6bf-4923-8751-5bd93650e2d0
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp,rdma
Bricks:
Brick1: server01ib:/data/brick1/scratch
Brick2: server02ib:/data/brick1/scratch
Options Reconfigured:
config.transport: tcp,rdma
performance.cache-size: 8GB
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
performance.readdir-ahead: on
cluster.lookup-optimize: on
client.event-threads: 8
performance.client-io-threads: on
performance.io-thread-count: 24
performance.write-behind-window-size: 8MB
server.event-threads: 32
performance.write-behind: off
performance.stat-prefetch: on
performance.resync-failed-syncs-after-fsync: on
server.outstanding-rpc-limit: 512
-------------- next part --------------
[2018-02-22 18:07:45.241970] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.242018] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.242681] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.242704] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.242728] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.242737] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.242753] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.242762] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.242786] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.242795] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243058] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.243071] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243090] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.243098] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243116] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.243124] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243141] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.243149] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243384] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.243396] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243416] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f6462b85000, wc.byte_len = 45056, post->reused = 1
[2018-02-22 18:07:45.243692] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f6462b85000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:45.244143] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f6461e73000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.159515] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f6462b7d000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.171016] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f6462b75000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.171548] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f6462b77000, wc.byte_len = 131072, post->reused = 1
-------------- next part --------------
[2018-02-22 18:07:45.241029] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023
[2018-02-22 18:07:45.241095] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.241269] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023
[2018-02-22 18:07:45.241333] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.241375] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023
[2018-02-22 18:07:45.241394] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.241461] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023
[2018-02-22 18:07:45.241482] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.241512] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023
[2018-02-22 18:07:45.241529] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.241560] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f10e7351000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:45.242274] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f10e7349000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.158767] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f10e733d000, wc.byte_len = 131072, post->reused = 1
-------------- next part --------------
[2018-02-22 18:07:45.221277] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.221309] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.221320] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54987 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.221334] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.221365] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.221368] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:45.221576] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.210946 (xid=0x54974)
[2018-02-22 18:07:45.221596] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.224781] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54988 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.224902] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54989 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225011] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225074] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225630] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225697] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225753] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225801] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225830] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54990 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.226338] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54991 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.226383] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54992 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.226452] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54993 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.226947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54994 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227452] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54995 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227491] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54996 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227524] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54997 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227568] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54998 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227615] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54999 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227654] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227738] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227801] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227863] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227920] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227972] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228767] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228803] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790331: READ => -1 gfid=3a86afe5-7392-49a5-b60a-e0c93b050c01 fd=0x2b2f23f023b4 (Transport endpoint is not connected)
[2018-02-22 18:07:45.228826] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228864] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790330: READ => -1 gfid=3a86afe5-7392-49a5-b60a-e0c93b050c01 fd=0x2b2f23f023b4 (Transport endpoint is not connected)
[2018-02-22 18:07:45.228872] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228919] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228966] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228989] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.229009] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229050] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229102] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229122] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.220065 (xid=0x54986)
[2018-02-22 18:07:45.229151] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229193] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229408] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549aa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 35 times between [2018-02-22 18:07:45.221365] and [2018-02-22 18:07:45.229445]
[2018-02-22 18:07:45.229449] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.229488] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ab Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 37 times between [2018-02-22 18:07:45.221334] and [2018-02-22 18:07:45.229507]
[2018-02-22 18:07:45.229524] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.229511] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.229549] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54385 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.229561] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.229572] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ac Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 37 times between [2018-02-22 18:07:45.221309] and [2018-02-22 18:07:45.229571]
[2018-02-22 18:07:45.229587] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.229583] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.229620] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:45.229623] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)" repeated 38 times between [2018-02-22 18:07:45.221277] and [2018-02-22 18:07:45.229677]
[2018-02-22 18:07:45.229678] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.229702] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ad Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229713] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.229736] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.220053 (xid=0x5437b)
[2018-02-22 18:07:45.229753] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.229769] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.229800] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.229812] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ae Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229866] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.220767 (xid=0x5437c)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.229623] and [2018-02-22 18:07:45.229838]
[2018-02-22 18:07:45.229883] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.229867] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.229903] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549af Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.229713] and [2018-02-22 18:07:45.229913]
[2018-02-22 18:07:45.229928] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.229978] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.229991] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 2 times between [2018-02-22 18:07:45.229800] and [2018-02-22 18:07:45.229990]
[2018-02-22 18:07:45.230004] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.222057 (xid=0x5437d)
[2018-02-22 18:07:45.230037] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230004] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230040] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230083] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230088] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230126] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230130] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.222694 (xid=0x5437e)
[2018-02-22 18:07:45.230129] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230149] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230163] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54386 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.230165] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230176] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.230004] and [2018-02-22 18:07:45.230183]
[2018-02-22 18:07:45.230193] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.230187] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.230088] and [2018-02-22 18:07:45.230195]
[2018-02-22 18:07:45.230220] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230214] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230236] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230242] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54387 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.230234] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230245] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230257] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230267] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.224869 (xid=0x5437f)
[2018-02-22 18:07:45.230273] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230264] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.230274] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230308] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230314] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230344] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 2 times between [2018-02-22 18:07:45.230257] and [2018-02-22 18:07:45.230367]
[2018-02-22 18:07:45.230368] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230377] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230389] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.225018 (xid=0x54380)
The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)" repeated 2 times between [2018-02-22 18:07:45.230314] and [2018-02-22 18:07:45.230420]
[2018-02-22 18:07:45.230423] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230422] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230424] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230454] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230466] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230476] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54388 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.230497] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230502] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.226409 (xid=0x54381)
[2018-02-22 18:07:45.230514] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54389 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 7 times between [2018-02-22 18:07:45.229883] and [2018-02-22 18:07:45.230485]
[2018-02-22 18:07:45.230492] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230513] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230536] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230535] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.230542] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230557] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.230565] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230570] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230588] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230572] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230600] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 5 times between [2018-02-22 18:07:45.230245] and [2018-02-22 18:07:45.230600]
[2018-02-22 18:07:45.230624] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 2 times between [2018-02-22 18:07:45.230422] and [2018-02-22 18:07:45.230623]
[2018-02-22 18:07:45.230612] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230625] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230618] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230635] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230652] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.227685 (xid=0x54382)
[2018-02-22 18:07:45.230665] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.230659] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230660] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.230612] and [2018-02-22 18:07:45.230674]
The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)" repeated 2 times between [2018-02-22 18:07:45.230565] and [2018-02-22 18:07:45.230688]
[2018-02-22 18:07:45.230677] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230690] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230717] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ba Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230728] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230740] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230763] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230764] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.228029 (xid=0x54383)
[2018-02-22 18:07:45.230762] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230773] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230784] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230782] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230786] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230788] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230804] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.230677] and [2018-02-22 18:07:45.230802]
[2018-02-22 18:07:45.230829] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230819] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230845] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230864] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230867] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230852] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.230874] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230898] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.229052 (xid=0x54384)
[2018-02-22 18:07:45.230900] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230914] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230914] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230901] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.230660] and [2018-02-22 18:07:45.230923]
[2018-02-22 18:07:45.230928] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230946] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549be Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230956] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230961] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790385: READ => -1 gfid=3a86afe5-7392-49a5-b60a-e0c93b050c01 fd=0x2b2f23f023b4 (Transport endpoint is not connected)
[2018-02-22 18:07:45.230960] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230972] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230986] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230990] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.231005] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230990] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.231008] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.231015] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231032] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.231042] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.231029] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231031] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231073] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231080] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231078] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.231085] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.231104] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231099] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231112] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231123] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231130] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231130] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231144] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231136] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231164] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231148] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231149] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231175] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231184] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231207] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231209] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231210] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231211] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231219] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231238] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231231] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231255] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231264] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54390 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231261] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231261] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231275] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231283] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231279] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231297] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231303] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231306] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231310] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231324] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231333] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54391 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231326] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231340] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231341] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231335] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231377] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231388] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231383] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231381] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t1_0001.fits (3a86afe5-7392-49a5-b60a-e0c93b050c01) [Transport endpoint is not connected]
[2018-02-22 18:07:45.231417] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231419] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231432] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231434] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t1_0001.fits (3a86afe5-7392-49a5-b60a-e0c93b050c01) [Transport endpoint is not connected]
[2018-02-22 18:07:45.231446] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231452] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231463] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-scratch-dht: Found anomalies in /download/work/kstovall/57910/drx_57910_B0656+14_b2t1_0001.fits (gfid = 3a86afe5-7392-49a5-b60a-e0c93b050c01). Holes=1 overlaps=0
[2018-02-22 18:07:45.231456] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231471] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231488] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: 3a86afe5-7392-49a5-b60a-e0c93b050c01: failed to resolve (Transport endpoint is not connected)
[2018-02-22 18:07:45.231490] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231486] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231501] E [fuse-bridge.c:796:fuse_getattr_resume] 0-glusterfs-fuse: 790393: GETATTR 47480857159808 (3a86afe5-7392-49a5-b60a-e0c93b050c01) resolution failed
[2018-02-22 18:07:45.231494] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231509] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231516] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54392 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231515] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231520] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231540] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231538] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231542] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231564] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231570] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54393 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231580] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231587] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231594] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231604] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231617] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231610] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231626] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231638] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231638] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231664] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54394 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231653] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231667] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231688] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231694] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231696] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231714] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231715] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231721] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231747] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231751] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231765] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231789] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231800] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231801] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231816] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231823] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54395 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231824] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ca Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231822] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231833] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231828] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231850] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231843] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231865] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231863] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231873] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231879] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231885] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231899] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231917] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231919] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231922] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231928] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54396 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231925] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231954] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231959] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231949] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231967] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231985] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232000] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54398 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231999] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232002] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232021] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232039] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232048] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232050] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232060] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54397 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.232057] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232064] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232075] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232073] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232092] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232098] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232100] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232104] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232114] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ce Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232121] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232136] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54399 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.232138] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232159] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232156] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232154] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232183] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232194] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232203] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232214] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232216] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232226] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232226] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232241] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232249] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232262] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232267] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232268] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232279] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.232276] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232302] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232303] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232314] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232320] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232329] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232328] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232335] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232344] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.232334] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232356] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232364] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232400] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232404] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232448] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232456] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232471] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232479] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232476] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232493] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.232496] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.232268] and [2018-02-22 18:07:45.232525]
[2018-02-22 18:07:45.232532] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232552] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232564] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232571] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.232364] and [2018-02-22 18:07:45.232579]
[2018-02-22 18:07:45.232592] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232592] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232632] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232654] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232668] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232680] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232708] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232703] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232706] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232726] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232757] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.233456] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.233454] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.233478] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.233485] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.233494] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.233483] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.233498] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233513] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233540] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.233532] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.233580] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.233592] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.233602] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.233614] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233630] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.233670] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.233692] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.233690] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.233706] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233739] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.233756] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.233755] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.233766] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233756] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.233795] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.233833] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.233835] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.233860] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.233871] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233892] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.233900] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.233908] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.233926] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.233936] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233937] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.233939] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.233960] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.233967] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.233978] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233996] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234011] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234023] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234016] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234035] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234065] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234077] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.234079] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.234096] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.234100] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234106] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234119] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.234115] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234136] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234146] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234154] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234157] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234174] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234184] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234197] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234220] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234218] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234231] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.234227] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234247] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.234256] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549da Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.234262] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop" repeated 3 times between [2018-02-22 18:07:45.234065] and [2018-02-22 18:07:45.234264]
[2018-02-22 18:07:45.234283] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.234511] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234513] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234530] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234540] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234556] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234567] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234572] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234583] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234593] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234613] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234622] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234629] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234638] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234646] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234670] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234677] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234692] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234707] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234716] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234725] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234738] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234746] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543aa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234751] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234754] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234784] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234828] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234834] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234842] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234859] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ab Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234867] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234879] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234880] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234881] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234907] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ac Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234917] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234929] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234923] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234954] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ad Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234953] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234965] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234988] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234996] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234998] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235015] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ae Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235024] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235037] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235059] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235075] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235083] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543af Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235094] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235119] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235182] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235197] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235196] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235205] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235231] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235258] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235260] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235273] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235280] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235297] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235321] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235336] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235344] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235352] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235362] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235399] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235413] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235412] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235423] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235436] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235444] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235460] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235465] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235486] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235495] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235501] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235514] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235529] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235537] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235558] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.236105] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:45.236245] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234482 (xid=0x549dc)
[2018-02-22 18:07:45.236264] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.236558] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234491 (xid=0x549de)
[2018-02-22 18:07:45.236672] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234655 (xid=0x549df)
[2018-02-22 18:07:45.236899] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234699 (xid=0x549e0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2018-02-22 18:07:45.236264] and [2018-02-22 18:07:45.236914]
[2018-02-22 18:07:45.236931] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237010] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234804 (xid=0x549e1)
[2018-02-22 18:07:45.237025] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237045] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237061] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237069] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ee Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237076] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237135] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234822 (xid=0x549e2)
[2018-02-22 18:07:45.237357] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(PORTMAP) op(PORTBYBRICK(1)) called at 2018-02-22 18:07:45.234954 (xid=0x549e3)
[2018-02-22 18:07:45.237150] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237431] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237430] W [MSGID: 114032] [client-handshake.c:1508:client_query_portmap_cbk] 2-scratch-client-0: received RPC status error, try again later [Transport endpoint is not connected]
[2018-02-22 18:07:45.237450] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237458] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ef Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237466] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.234283] and [2018-02-22 18:07:45.237488]
[2018-02-22 18:07:45.237510] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237529] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237541] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237541] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234979 (xid=0x549e4)
[2018-02-22 18:07:45.237549] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237555] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237566] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237554] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237585] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237584] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237594] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237623] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237657] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237658] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237675] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237673] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235016 (xid=0x549e5)
[2018-02-22 18:07:45.237690] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237696] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237698] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237710] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237721] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237736] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237748] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237763] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237771] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237765] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237784] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237799] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235054 (xid=0x549e6)
[2018-02-22 18:07:45.237797] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237813] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237827] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237814] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237835] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237842] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237862] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237886] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237888] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237905] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237913] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237926] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237934] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235159 (xid=0x549e7)
[2018-02-22 18:07:45.237942] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237941] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237953] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237950] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237965] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.237862] and [2018-02-22 18:07:45.237976]
[2018-02-22 18:07:45.237986] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238005] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238012] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.237950] and [2018-02-22 18:07:45.238020]
[2018-02-22 18:07:45.238034] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238038] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.238033] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238070] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238068] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238079] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235589 (xid=0x549e8)
[2018-02-22 18:07:45.238080] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238097] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238100] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238110] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238122] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238111] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238129] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238147] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238174] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238183] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238182] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238195] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238207] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238208] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238220] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238214] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235705 (xid=0x549e9)
[2018-02-22 18:07:45.238229] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.238195] and [2018-02-22 18:07:45.238243]
[2018-02-22 18:07:45.238247] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238253] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238266] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238278] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238287] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238297] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238302] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238303] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238327] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238337] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238357] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235740 (xid=0x549ea)
[2018-02-22 18:07:45.238381] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238392] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238411] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238418] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ff Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238424] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.238297] and [2018-02-22 18:07:45.238446]
[2018-02-22 18:07:45.238469] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235777 (xid=0x549eb)
[2018-02-22 18:07:45.238465] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238485] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238491] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238507] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a00 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238516] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238530] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238536] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238531] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238556] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a01 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238579] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235812 (xid=0x549ec)
[2018-02-22 18:07:45.238566] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238583] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238579] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238595] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238610] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238621] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a02 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238629] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238643] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238651] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238644] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238672] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a03 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238682] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238692] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238696] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238709] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235840 (xid=0x549ed)
[2018-02-22 18:07:45.238711] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238729] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a05 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238739] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238752] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238752] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238761] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238788] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a04 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238798] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238812] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238823] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238824] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238837] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a06 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238847] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238862] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238869] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238865] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238885] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a07 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238893] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238903] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238907] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238922] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238935] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238943] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238956] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238964] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238958] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238977] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a09 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238990] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239004] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239007] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239024] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239038] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a08 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239047] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239060] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239069] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239062] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239081] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239090] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239100] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239105] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239119] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239132] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239140] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239153] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239160] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239155] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239174] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239183] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239190] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239207] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239208] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239221] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239233] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239243] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239244] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239254] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239259] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239268] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.239279] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239270] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.239286] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239296] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ce Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.239293] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239307] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a10 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239305] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239309] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:45.239328] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239333] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.239345] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239330] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239361] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a11 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239383] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2018-02-22 18:07:45.239233] and [2018-02-22 18:07:45.239384]
[2018-02-22 18:07:45.239410] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239401] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239428] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a12 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239432] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235590 (xid=0x543b8)
[2018-02-22 18:07:45.239449] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239453] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239457] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239459] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239483] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a13 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239484] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239500] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239506] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239523] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239528] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239541] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a14 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239551] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239557] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235614 (xid=0x543b9)
[2018-02-22 18:07:45.239573] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239573] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239574] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239589] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239607] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a17 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239616] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239625] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.239633] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239638] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.239644] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239650] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239657] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543cf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.239666] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a16 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239676] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239682] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235622 (xid=0x543ba)
[2018-02-22 18:07:45.239685] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239696] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239704] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.239712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239733] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239748] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239758] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a15 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239769] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239789] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239799] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235647 (xid=0x543bb)
[2018-02-22 18:07:45.239798] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.239803] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239826] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.239824] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.239833] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a18 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239797] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239827] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239835] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239856] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.239851] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239881] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239884] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239899] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a19 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239898] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239911] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239921] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239925] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.239932] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239936] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.239947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.239947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239950] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239954] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235673 (xid=0x543bc)
[2018-02-22 18:07:45.239956] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239969] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239969] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239988] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239994] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239998] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240006] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240016] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240026] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240036] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240035] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.240043] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240039] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240058] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240062] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240073] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240077] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240082] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235687 (xid=0x543bd)
[2018-02-22 18:07:45.240100] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240099] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240084] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240115] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240098] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240127] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240136] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240149] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240164] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240166] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240173] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240184] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240193] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240203] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240210] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240211] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240220] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240222] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235708 (xid=0x543be)
[2018-02-22 18:07:45.240237] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240239] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240243] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240257] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240267] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a20 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240275] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240286] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240291] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240297] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240311] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a21 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240316] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.240321] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240335] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240339] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240342] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235728 (xid=0x543bf)
[2018-02-22 18:07:45.240353] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240368] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240368] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240357] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240406] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.240360] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240424] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240426] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240428] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240438] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a22 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240435] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240449] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240452] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240460] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240471] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240473] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.240484] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a23 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240490] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240487] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235746 (xid=0x543c0)
[2018-02-22 18:07:45.240482] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240507] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240513] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240490] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240517] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240493] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240533] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240541] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240548] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240558] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240561] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240567] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240580] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240589] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240592] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a24 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240602] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed" repeated 2 times between [2018-02-22 18:07:45.240473] and [2018-02-22 18:07:45.240601]
[2018-02-22 18:07:45.240616] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235760 (xid=0x543c1)
[2018-02-22 18:07:45.240611] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240611] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240622] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240633] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240656] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240661] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240668] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a25 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240676] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240677] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240689] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240702] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240710] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a26 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240719] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240719] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240732] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240746] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240757] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a27 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240761] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240764] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235783 (xid=0x543c2)
[2018-02-22 18:07:45.240768] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240777] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240777] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240797] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240815] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.240798] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240828] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240831] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a28 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240835] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240841] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240854] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240857] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240870] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240880] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a29 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240874] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240888] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240899] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240913] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240910] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.236943 (xid=0x543c3)
[2018-02-22 18:07:45.240911] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240920] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240919] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240924] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240950] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240956] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240965] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240986] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240978] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240999] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241006] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241001] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241015] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241033] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241038] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(PORTMAP) op(PORTBYBRICK(1)) called at 2018-02-22 18:07:45.237021 (xid=0x543c4)
[2018-02-22 18:07:45.241034] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241048] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241055] W [MSGID: 114032] [client-handshake.c:1508:client_query_portmap_cbk] 2-scratch-client-1: received RPC status error, try again later [Transport endpoint is not connected]
[2018-02-22 18:07:45.241043] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241068] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543da Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241079] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241080] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241102] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241106] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241113] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241108] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241111] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241130] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543db Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241142] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241149] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.237611 (xid=0x543c5)
[2018-02-22 18:07:45.241149] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241150] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241166] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241173] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241172] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241174] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241187] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241189] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241221] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241245] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241259] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241263] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241273] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241282] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543dc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241282] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241291] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241288] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241295] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241293] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.237634 (xid=0x543c6)
[2018-02-22 18:07:45.241313] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241315] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241321] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241330] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241335] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543dd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241334] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241335] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241351] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241357] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241344] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241348] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241378] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241389] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a30 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241389] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241408] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241405] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241421] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241430] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a31 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241434] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241439] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241442] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241442] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241474] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543de Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241483] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.237740 (xid=0x543c7)
[2018-02-22 18:07:45.241477] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241499] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241485] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241500] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241516] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241528] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543df Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241537] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241539] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241552] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241555] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241568] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241579] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241587] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241595] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241598] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241611] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241612] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241618] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a32 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241628] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241621] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.237917 (xid=0x543c8)
[2018-02-22 18:07:45.241616] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241637] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241648] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241650] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241649] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241636] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241672] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241674] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241680] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790398: READ => -1 gfid=cb243c85-02fe-4e37-b03f-80f9187e72df fd=0x2b2f23f02210 (Transport endpoint is not connected)
[2018-02-22 18:07:45.241690] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a33 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241685] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241704] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241708] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241706] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241722] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241726] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241741] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed" repeated 2 times between [2018-02-22 18:07:45.241612] and [2018-02-22 18:07:45.241740]
[2018-02-22 18:07:45.241741] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241753] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.241754] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241767] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241786] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a34 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241784] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241791] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241784] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241795] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241810] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241820] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241824] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)" repeated 2 times between [2018-02-22 18:07:45.241726] and [2018-02-22 18:07:45.241840]
[2018-02-22 18:07:45.241846] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241855] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241848] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.238059 (xid=0x543c9)
[2018-02-22 18:07:45.241842] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241826] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241881] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241885] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a35 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241886] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241890] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241894] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241910] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241908] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241927] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790399: READ => -1 gfid=cb243c85-02fe-4e37-b03f-80f9187e72df fd=0x2b2f23f02210 (Transport endpoint is not connected)
[2018-02-22 18:07:45.241907] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241940] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a36 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241945] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241962] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241962] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241980] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241974] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241991] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.241998] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242004] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242008] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a37 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242008] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242000] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242023] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242018] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242040] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242049] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242054] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242063] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242067] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242065] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242084] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238080 (xid=0x543ca)
[2018-02-22 18:07:45.242095] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242086] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242101] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242107] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242109] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242125] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242130] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242132] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242137] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242141] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a38 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242156] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242162] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242164] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242176] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242191] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242199] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a39 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242201] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242204] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242223] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238146 (xid=0x543cb)
[2018-02-22 18:07:45.242220] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242249] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242246] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242252] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242277] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242288] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ea Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242299] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.242312] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242313] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242332] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242330] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790424: READ => -1 gfid=cb243c85-02fe-4e37-b03f-80f9187e72df fd=0x2b2f23f02210 (Transport endpoint is not connected)
[2018-02-22 18:07:45.242341] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242354] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543eb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242356] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242366] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.242246] and [2018-02-22 18:07:45.242363]
[2018-02-22 18:07:45.242384] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242369] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242403] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242408] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238325 (xid=0x543cc)
[2018-02-22 18:07:45.242408] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242436] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242435] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242431] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242436] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242460] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242461] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242476] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242468] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242489] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242489] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242493] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ec Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242497] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242526] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242528] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242511] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242530] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242530] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242524] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242552] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242555] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238346 (xid=0x543cd)
[2018-02-22 18:07:45.242567] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242578] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ed Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242571] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242579] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242594] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242607] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242605] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242638] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ee Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.242579] and [2018-02-22 18:07:45.242647]
[2018-02-22 18:07:45.242658] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242680] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ef Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed" repeated 2 times between [2018-02-22 18:07:45.242567] and [2018-02-22 18:07:45.242679]
[2018-02-22 18:07:45.242689] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242701] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242702] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242718] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop" repeated 3 times between [2018-02-22 18:07:45.242607] and [2018-02-22 18:07:45.242733]
[2018-02-22 18:07:45.242748] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242740] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242762] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242761] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242762] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242771] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t2_0001.fits (cb243c85-02fe-4e37-b03f-80f9187e72df) [Transport endpoint is not connected]
[2018-02-22 18:07:45.242782] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.242689] and [2018-02-22 18:07:45.242777]
[2018-02-22 18:07:45.242797] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242795] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242801] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t2_0001.fits (cb243c85-02fe-4e37-b03f-80f9187e72df) [Transport endpoint is not connected]
[2018-02-22 18:07:45.242795] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242817] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242831] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-scratch-dht: Found anomalies in /download/work/kstovall/57910/drx_57910_B0656+14_b2t2_0001.fits (gfid = cb243c85-02fe-4e37-b03f-80f9187e72df). Holes=1 overlaps=0
[2018-02-22 18:07:45.242836] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242845] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242845] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242853] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242857] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: cb243c85-02fe-4e37-b03f-80f9187e72df: failed to resolve (Transport endpoint is not connected)
[2018-02-22 18:07:45.242870] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242845] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242875] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242879] E [fuse-bridge.c:796:fuse_getattr_resume] 0-glusterfs-fuse: 790426: GETATTR 47480857159432 (cb243c85-02fe-4e37-b03f-80f9187e72df) resolution failed
[2018-02-22 18:07:45.242879] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242906] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242913] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242917] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242917] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242927] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242934] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a40 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242937] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242952] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242969] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242979] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242979] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242989] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a41 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242998] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243012] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243018] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243035] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243047] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a42 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243050] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243050] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243060] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243068] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243070] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243084] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a43 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243087] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243077] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243082] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243089] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243106] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243128] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243132] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243140] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243141] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243153] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a44 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243136] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243159] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243168] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243171] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243177] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243188] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243203] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243203] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243215] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243207] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243226] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243236] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243247] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243256] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243258] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243265] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243257] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243287] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a45 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243289] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243294] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243296] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243321] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243318] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243327] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243346] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243355] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a46 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243356] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243346] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243364] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243388] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243383] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243388] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243405] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243419] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a47 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243428] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243442] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243447] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243444] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243466] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a48 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243475] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243487] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243503] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243507] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243520] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a49 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243528] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243535] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243540] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243537] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243559] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243569] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243582] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243574] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243588] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243604] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243598] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243616] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243621] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243624] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243631] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243646] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243649] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243648] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243656] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243679] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243688] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243688] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243698] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243700] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243707] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243717] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243729] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243733] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243735] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243739] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243759] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243764] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243766] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243773] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243785] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243786] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243791] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243797] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243799] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243806] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243816] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243832] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243838] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243833] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243852] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243862] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243872] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243881] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243889] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243889] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243897] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243916] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243918] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243928] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243934] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243940] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243952] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243958] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243962] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243967] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243973] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243985] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243992] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244002] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ff Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244003] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244010] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244023] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244027] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244043] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244057] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54400 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244065] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244075] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244076] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244092] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54401 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244092] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244100] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244112] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244118] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244136] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244149] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54402 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244158] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244167] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244168] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244183] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54403 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244185] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244196] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244204] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244223] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244231] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244228] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244245] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244250] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54404 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244254] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a50 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244264] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244265] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244271] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244279] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244273] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244274] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244293] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54405 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244297] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244302] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244310] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a51 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244314] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244320] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244322] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244325] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244341] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54406 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244344] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244351] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244363] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244364] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244383] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244391] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54407 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244388] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244403] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244413] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244425] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244432] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54408 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244430] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244445] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244443] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244460] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244473] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a52 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244476] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244483] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244496] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244517] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244522] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244533] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a53 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244542] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244552] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244561] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244568] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a54 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244579] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244579] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244590] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244603] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244610] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a55 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244620] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244621] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244633] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244644] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244651] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a56 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244660] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244663] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244674] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244676] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244692] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244699] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54409 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244708] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244724] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244734] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244741] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244745] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244750] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244762] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244775] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244774] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244788] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244784] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244797] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244806] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244824] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244832] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244837] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244843] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244858] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244865] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244866] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244883] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244886] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244899] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244903] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244915] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244918] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244933] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a57 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.244832] and [2018-02-22 18:07:45.244923]
[2018-02-22 18:07:45.244949] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244951] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244941] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244967] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a58 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244965] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244977] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244976] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244991] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244993] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245009] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a59 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245020] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245017] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245017] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.245042] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.245046] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245051] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.245059] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245068] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245078] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245077] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245087] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245097] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.245099] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245101] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245126] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245129] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245134] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245156] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245169] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245177] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245180] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245186] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245196] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245201] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245208] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245212] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.245216] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245225] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.245232] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54410 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.245230] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245248] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245254] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.245248] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245272] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.245281] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54411 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.245288] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.245289] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245298] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.245314] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54412 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.245313] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.245322] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245339] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.245367] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.245389] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245404] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245422] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245432] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245443] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245454] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245461] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245468] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245471] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245499] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245507] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245507] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245522] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a60 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245529] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245567] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245592] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245601] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245608] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a61 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245614] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245654] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.246260] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0)
[2018-02-22 18:07:45.247605] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-1: changing port to 49154 (from 0)
[2018-02-22 18:07:45.254977] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:45.255615] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-0: Connected to scratch-client-0, attached to remote volume '/data/brick1/scratch'.
[2018-02-22 18:07:45.255630] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-22 18:07:45.255641] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-0: 4 fds open - Delaying child_up until they are re-opened
[2018-02-22 18:07:45.255830] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:45.256099] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-02-22 18:07:45.256197] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-0: Server lk version = 1
[2018-02-22 18:07:45.256226] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-1: Connected to scratch-client-1, attached to remote volume '/data/brick1/scratch'.
[2018-02-22 18:07:45.256238] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-22 18:07:45.256247] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-1: 4 fds open - Delaying child_up until they are re-opened
[2018-02-22 18:07:45.256661] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-02-22 18:07:45.256739] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-1: Server lk version = 1
[2018-02-22 18:07:46.186045] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:46.186183] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184758 (xid=0x54a6e)
[2018-02-22 18:07:46.186197] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:46.186286] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184793 (xid=0x54a6f)
[2018-02-22 18:07:46.186379] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184809 (xid=0x54a70)
[2018-02-22 18:07:46.186472] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184824 (xid=0x54a71)
[2018-02-22 18:07:46.186557] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184837 (xid=0x54a72)
[2018-02-22 18:07:46.186641] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184851 (xid=0x54a73)
[2018-02-22 18:07:46.186718] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184881 (xid=0x54a74)
[2018-02-22 18:07:46.186794] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184919 (xid=0x54a75)
[2018-02-22 18:07:46.186879] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184933 (xid=0x54a76)
[2018-02-22 18:07:46.186963] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184948 (xid=0x54a77)
[2018-02-22 18:07:46.187044] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184971 (xid=0x54a78)
[2018-02-22 18:07:46.187128] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184992 (xid=0x54a79)
[2018-02-22 18:07:46.187206] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185049 (xid=0x54a7a)
[2018-02-22 18:07:46.187282] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185090 (xid=0x54a7b)
[2018-02-22 18:07:46.187367] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185110 (xid=0x54a7c)
[2018-02-22 18:07:46.187454] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185124 (xid=0x54a7d)
[2018-02-22 18:07:46.187547] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185137 (xid=0x54a7e)
[2018-02-22 18:07:46.187635] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185164 (xid=0x54a7f)
[2018-02-22 18:07:46.187712] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185186 (xid=0x54a80)
[2018-02-22 18:07:46.187799] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185308 (xid=0x54a81)
[2018-02-22 18:07:46.187886] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185351 (xid=0x54a82)
[2018-02-22 18:07:46.187973] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185391 (xid=0x54a83)
[2018-02-22 18:07:46.188050] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185410 (xid=0x54a84)
[2018-02-22 18:07:46.188126] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185447 (xid=0x54a85)
[2018-02-22 18:07:46.188203] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185462 (xid=0x54a86)
[2018-02-22 18:07:46.188281] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185485 (xid=0x54a87)
[2018-02-22 18:07:46.188369] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185498 (xid=0x54a88)
[2018-02-22 18:07:46.188470] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185514 (xid=0x54a89)
[2018-02-22 18:07:46.188757] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184866 (xid=0x5441f)
[2018-02-22 18:07:46.188678] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:46.188773] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:46.188866] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184890 (xid=0x54420)
[2018-02-22 18:07:46.188960] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184903 (xid=0x54421)
[2018-02-22 18:07:46.189053] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184958 (xid=0x54422)
[2018-02-22 18:07:46.189149] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184985 (xid=0x54423)
[2018-02-22 18:07:46.189243] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185005 (xid=0x54424)
[2018-02-22 18:07:46.189338] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185019 (xid=0x54425)
[2018-02-22 18:07:46.189446] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185032 (xid=0x54426)
[2018-02-22 18:07:46.189543] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185058 (xid=0x54427)
[2018-02-22 18:07:46.189638] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185072 (xid=0x54428)
[2018-02-22 18:07:46.189735] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185103 (xid=0x54429)
[2018-02-22 18:07:46.189834] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185150 (xid=0x5442a)
[2018-02-22 18:07:46.189930] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185173 (xid=0x5442b)
[2018-02-22 18:07:46.190028] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185198 (xid=0x5442c)
[2018-02-22 18:07:46.190126] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185211 (xid=0x5442d)
[2018-02-22 18:07:46.190224] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185224 (xid=0x5442e)
[2018-02-22 18:07:46.190323] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185236 (xid=0x5442f)
[2018-02-22 18:07:46.190427] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185249 (xid=0x54430)
[2018-02-22 18:07:46.190529] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185265 (xid=0x54431)
[2018-02-22 18:07:46.190629] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185281 (xid=0x54432)
[2018-02-22 18:07:46.190729] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185295 (xid=0x54433)
[2018-02-22 18:07:46.190830] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185322 (xid=0x54434)
[2018-02-22 18:07:46.190931] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185334 (xid=0x54435)
[2018-02-22 18:07:46.191033] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185366 (xid=0x54436)
[2018-02-22 18:07:46.191135] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185422 (xid=0x54437)
[2018-02-22 18:07:46.191237] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185437 (xid=0x54438)
[2018-02-22 18:07:46.191342] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185472 (xid=0x54439)
[2018-02-22 18:07:46.191457] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185524 (xid=0x5443a)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 27 times between [2018-02-22 18:07:46.188773] and [2018-02-22 18:07:46.191472]
[2018-02-22 18:07:47.148084] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.148118] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.148126] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 28 times between [2018-02-22 18:07:46.186197] and [2018-02-22 18:07:47.148136]
[2018-02-22 18:07:47.148142] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148200] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.148218] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148217] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.148229] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148259] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:47.148272] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.148271] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:47.148281] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 790447: FLUSH() ERR => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:47.148279] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148310] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148314] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.148346] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.148355] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.148362] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148369] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148410] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148435] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.148436] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.148450] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.148457] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148476] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.148505] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.148513] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.148520] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.148527] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148538] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.148560] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.148568] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.148575] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148596] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148617] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148626] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.148666] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.148678] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148702] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148761] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.148765] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.148807] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148803] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.148816] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.148846] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.148849] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148858] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148872] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148888] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.148920] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.148924] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.148922] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.148954] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.148961] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.148964] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148973] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a90 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148994] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149000] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149008] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.149029] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149045] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149058] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54440 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149077] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149091] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149126] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.149132] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149141] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.149158] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a91 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.149162] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149177] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54441 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149168] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149185] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149185] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.149228] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149275] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54442 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149287] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149325] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149343] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149345] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149379] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54443 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149390] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149420] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149422] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149450] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149479] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54444 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149504] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149572] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149606] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149607] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149627] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54445 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149635] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149668] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149694] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149713] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149723] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54446 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149753] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149797] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149811] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54447 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149810] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149820] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149857] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149891] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149891] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149904] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54448 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149911] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149935] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149952] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149966] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149973] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54449 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149985] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150013] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150057] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150067] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150066] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150078] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150104] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150136] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150138] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150162] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150174] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150193] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150217] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150225] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150219] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150232] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150246] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150259] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150258] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150266] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150262] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150293] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150319] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150321] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150335] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150348] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150369] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150403] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.150410] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150423] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.150433] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a92 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.150430] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150447] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150454] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150467] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150470] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.150484] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150510] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150520] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150527] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54451 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150535] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150558] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150585] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150598] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150604] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54453 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150613] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150640] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150683] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150697] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150707] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54455 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150726] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150738] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150752] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150759] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54454 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150759] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150768] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150794] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.150820] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.150823] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150832] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a93 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.150836] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150854] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54456 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150846] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150868] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.150866] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150890] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150916] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.150925] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.150932] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a94 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.150940] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150940] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150979] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.150980] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.150998] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.151012] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a95 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.151010] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54450 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151007] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151025] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151065] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151078] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.151109] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151120] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151131] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151146] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54452 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151159] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151176] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151196] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151200] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151207] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54457 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151222] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151236] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151251] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151261] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54458 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151262] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151273] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151306] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151311] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.151328] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a96 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.151327] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.151343] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151350] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151384] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.151384] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151396] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54459 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151406] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151414] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151423] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151429] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151434] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151436] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151465] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151471] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151482] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151491] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151504] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151523] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151539] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151539] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151566] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151576] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151598] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151609] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151617] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151635] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151653] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151668] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151686] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151688] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151697] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151710] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151731] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151745] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151753] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54461 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151753] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151779] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151804] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151818] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151819] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151844] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54462 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151857] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151899] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151930] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151948] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151958] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151970] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152001] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152033] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152042] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152053] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54465 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152062] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152089] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152092] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152107] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152116] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54463 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152127] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152140] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152153] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152164] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54464 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152168] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152173] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152207] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152233] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152243] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152250] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54467 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152251] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0)
[2018-02-22 18:07:47.152258] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152270] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152296] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152304] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152310] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54469 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152318] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152330] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152332] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152358] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152375] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54466 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152388] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152422] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152443] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152460] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152469] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54460 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152481] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152508] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152550] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152572] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152582] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc3000, wc.byte_len = 0, post->reused = 2
[2018-02-22 18:07:47.152588] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54468 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152636] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152667] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152681] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152709] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152729] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152734] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc5000, wc.byte_len = 0, post->reused = 3
[2018-02-22 18:07:47.152751] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152791] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152796] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152817] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152829] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152847] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152868] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152909] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152922] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152930] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152942] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152948] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc3000, wc.byte_len = 0, post->reused = 3
[2018-02-22 18:07:47.152970] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.153025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.153060] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.153083] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.153104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.153142] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc3000, wc.byte_len = 0, post->reused = 4
[2018-02-22 18:07:47.153142] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.153646] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.153648] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.153682] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a9f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.153698] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.153722] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.153886] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.153901] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.153908] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.153918] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.153936] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.153978] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.153993] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154004] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154015] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154031] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154087] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154104] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154116] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154128] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154144] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154170] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154188] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154195] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154196] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152215 (xid=0x54a99)
[2018-02-22 18:07:47.154206] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154229] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154230] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154239] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154270] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154282] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154292] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154317] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154320] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154344] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154369] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154382] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154410] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154507] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154506] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152542 (xid=0x54a9a)
[2018-02-22 18:07:47.154518] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154532] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154535] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154542] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154561] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154580] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154596] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154606] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154617] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154632] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154646] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154650] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154653] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154662] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154696] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154701] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154701] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152568 (xid=0x54a9b)
[2018-02-22 18:07:47.154714] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154730] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154758] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154743] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154756] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154792] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154801] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aaa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154813] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154837] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154857] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154858] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154879] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aab Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154893] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154905] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154917] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154926] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aac Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154929] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154918] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152591 (xid=0x54a9c)
[2018-02-22 18:07:47.154940] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154951] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154980] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155029] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155041] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155049] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aad Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155059] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155091] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155087] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152742 (xid=0x54a9d)
[2018-02-22 18:07:47.155110] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155131] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155147] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155157] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aae Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155168] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155206] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155240] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155252] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155262] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aaf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155271] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155284] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153129 (xid=0x54a9e)
[2018-02-22 18:07:47.155312] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155318] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155368] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155382] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155389] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155397] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155425] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155452] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155461] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155468] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155474] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155495] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155526] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155535] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155542] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155553] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155576] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155585] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155593] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:47.155600] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155610] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155621] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155641] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155661] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155666] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155677] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155686] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155707] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155719] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155721] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155727] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155736] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155738] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153148 (xid=0x5446c)
[2018-02-22 18:07:47.155760] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155763] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155766] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155798] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155813] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155827] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155848] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155865] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155866] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155893] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155903] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155913] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153170 (xid=0x5446f)
[2018-02-22 18:07:47.155918] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155938] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155950] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155961] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155967] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155992] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156036] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156073] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156086] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156094] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156100] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2018-02-22 18:07:47.153187 (xid=0x54470)
[2018-02-22 18:07:47.156128] W [MSGID: 114032] [client-handshake.c:1625:client_dump_version_cbk] 2-scratch-client-1: received RPC status error [Transport endpoint is not connected]
[2018-02-22 18:07:47.156129] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.156140] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156158] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.156169] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.156178] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156195] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.156197] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156225] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156238] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aba Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156251] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156272] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156275] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153204 (xid=0x54471)
[2018-02-22 18:07:47.156287] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156309] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156298] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156319] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156303] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156358] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156403] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156414] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156413] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156421] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156454] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156485] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156485] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156501] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156509] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156504] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153220 (xid=0x54472)
[2018-02-22 18:07:47.156529] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156543] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156564] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156583] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156592] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156601] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156636] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156665] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156676] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156686] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156693] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156701] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153237 (xid=0x54473)
[2018-02-22 18:07:47.156723] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156728] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156766] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156774] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156780] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156787] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156790] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156816] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.156824] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.156832] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.156830] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153269 (xid=0x54474)
[2018-02-22 18:07:47.156854] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156855] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156876] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.156865] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.156904] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.156916] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.156923] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156927] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156937] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156936] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156947] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156957] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.156982] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156994] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156991] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153302 (xid=0x54475)
[2018-02-22 18:07:47.157009] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157010] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157025] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157034] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157042] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157043] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157060] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157064] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157085] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157087] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157101] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157101] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153334 (xid=0x54476)
[2018-02-22 18:07:47.157106] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.157107] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157118] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157124] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.157132] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:47.157034] and [2018-02-22 18:07:47.157127]
[2018-02-22 18:07:47.157142] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157138] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157158] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.157171] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.157179] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.157180] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.157179] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157195] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157192] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157201] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157208] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157221] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.157225] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153366 (xid=0x54477)
[2018-02-22 18:07:47.157233] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157248] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157245] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157259] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157266] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157273] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157285] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157295] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157299] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157301] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157312] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157325] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157333] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157338] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157340] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157354] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157357] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153393 (xid=0x54478)
[2018-02-22 18:07:47.157367] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157382] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157384] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157389] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157402] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157459] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157497] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.157498] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.157503] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157514] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54490 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.157519] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157521] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157526] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aca Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157532] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157532] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.157547] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153416 (xid=0x54479)
[2018-02-22 18:07:47.157562] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157565] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:47.157459] and [2018-02-22 18:07:47.157565]
[2018-02-22 18:07:47.157575] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157589] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157600] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157612] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157613] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157628] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157630] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157638] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157656] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.157670] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.157668] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153441 (xid=0x5447a)
[2018-02-22 18:07:47.157676] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54491 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.157685] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157690] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157700] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.157712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.157718] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.157725] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54492 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.157732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157742] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157751] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157757] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:47.157600] and [2018-02-22 18:07:47.157766]
[2018-02-22 18:07:47.157767] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.157788] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157803] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153758 (xid=0x5447b)
[2018-02-22 18:07:47.157809] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157823] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ace Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157822] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157823] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157835] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157881] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157906] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157907] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157919] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157925] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157940] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153782 (xid=0x5447c)
[2018-02-22 18:07:47.157955] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157957] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157973] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157987] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157993] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157999] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158031] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158067] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158067] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153910 (xid=0x5447d)
[2018-02-22 18:07:47.158077] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158084] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158082] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158090] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158108] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158129] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.158137] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.158144] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54493 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.158150] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158186] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.158183] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153928 (xid=0x5447e)
[2018-02-22 18:07:47.158198] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158209] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158218] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158218] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158225] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158264] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158288] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158289] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158305] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158312] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158309] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153945 (xid=0x5447f)
[2018-02-22 18:07:47.158325] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158456] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153964 (xid=0x54480)
[2018-02-22 18:07:47.158474] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158475] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158492] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158498] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158505] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:47.158264] and [2018-02-22 18:07:47.158515]
[2018-02-22 18:07:47.158529] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.158544] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.158550] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54494 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.158556] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158581] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153990 (xid=0x54481)
[2018-02-22 18:07:47.158600] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158600] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.158649] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158658] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158665] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158672] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158704] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158708] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154202 (xid=0x54482)
[2018-02-22 18:07:47.158722] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158735] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158744] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158743] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158750] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158794] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158817] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158817] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158829] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158837] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158839] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154251 (xid=0x54483)
[2018-02-22 18:07:47.158854] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158876] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158919] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158933] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158939] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158945] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158979] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154278 (xid=0x54484)
[2018-02-22 18:07:47.158986] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158997] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.159017] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.159033] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.159039] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.159045] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.159087] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.159103] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.159110] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154297 (xid=0x54485)
[2018-02-22 18:07:47.159118] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.159130] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54495 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.159127] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.159141] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.159178] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.159284] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154326 (xid=0x54486)
[2018-02-22 18:07:47.159298] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.159432] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154349 (xid=0x54487)
[2018-02-22 18:07:47.159569] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154507 (xid=0x54488)
[2018-02-22 18:07:47.159703] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154546 (xid=0x54489)
[2018-02-22 18:07:47.159837] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154685 (xid=0x5448a)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 4 times between [2018-02-22 18:07:47.159298] and [2018-02-22 18:07:47.159852]
[2018-02-22 18:07:47.159919] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.159942] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.159948] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ada Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.160013] W [fuse-bridge.c:2312:fuse_writev_cbk] 0-glusterfs-fuse: 790458: WRITE => -1 gfid=91f97edd-97a3-4948-9d9d-469c2d174853 fd=0x2b2f23f02440 (Transport endpoint is not connected)
[2018-02-22 18:07:47.159959] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.160040] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.161163] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161183] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161195] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54496 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161205] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161225] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161258] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.161260] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.161280] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54adb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.161290] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161318] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.161360] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161395] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161407] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54497 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161418] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161448] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161473] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161492] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161501] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54498 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161510] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161523] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161566] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161580] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54499 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161579] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161589] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161626] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161659] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161660] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161678] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161687] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161734] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161752] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161761] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161770] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161785] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161828] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161841] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161840] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161850] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161882] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161914] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.161925] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.161934] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54adc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.161943] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161970] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162001] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162012] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162021] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54add Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162030] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162052] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162082] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.162083] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.162101] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.162109] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162139] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.162171] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162184] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162194] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ade Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162202] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162231] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162283] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54adf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162292] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162296] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162334] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.162346] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.162355] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.162363] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162401] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.162434] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162447] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162446] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162456] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162497] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162530] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162531] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162549] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162558] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162589] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162614] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.162633] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.162642] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.162650] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162682] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.162714] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162725] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162734] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162742] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162764] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.162816] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.162825] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.162833] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162866] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.162901] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162913] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162912] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162922] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162948] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162976] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.162987] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.162996] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163006] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163040] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163073] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.163074] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.163093] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.163104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163119] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.163147] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.163159] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.163167] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163176] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163212] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163234] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.163253] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.163262] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163270] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163296] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163337] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.163348] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.163356] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.163365] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163415] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.163451] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.163462] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.163471] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.163480] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163518] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.163550] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.163562] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163562] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.163571] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163616] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163649] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.163660] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.163669] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.163677] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163692] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.163720] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.163721] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.163738] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163747] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163787] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163819] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.163831] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.163841] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163855] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163897] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163931] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.163942] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.163953] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.163962] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.164005] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.164029] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.164048] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.164057] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.164066] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.164080] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.164189] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:47.164228] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.164245] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.164254] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.164263] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.164280] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.164343] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.164355] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.164365] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.164382] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.164427] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.164462] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.164474] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.164487] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544aa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.164496] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.164540] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.165116] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165133] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165143] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165155] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165162] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:47.165193] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.165235] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165248] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165247] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165258] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165299] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164108 (xid=0x54aea)
[2018-02-22 18:07:47.165311] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.165316] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165346] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165348] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165367] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165384] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165469] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.165522] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164130 (xid=0x54aeb)
[2018-02-22 18:07:47.165539] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165572] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165589] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165588] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165599] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165621] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.165668] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164175 (xid=0x54aec)
[2018-02-22 18:07:47.165682] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165746] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165748] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165770] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165802] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164210 (xid=0x54aed)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:47.165682] and [2018-02-22 18:07:47.165815]
[2018-02-22 18:07:47.165822] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.165876] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165889] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165888] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165899] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165926] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS Handshake) op(SETVOLUME(1)) called at 2018-02-22 18:07:47.164238 (xid=0x54aee)
[2018-02-22 18:07:47.165941] W [MSGID: 114032] [client-handshake.c:1088:client_setvolume_cbk] 2-scratch-client-0: received RPC status error [Transport endpoint is not connected]
[2018-02-22 18:07:47.165953] I [MSGID: 114051] [client-handshake.c:1259:client_setvolume_cbk] 2-scratch-client-0: sending CHILD_CONNECTING event
[2018-02-22 18:07:47.165963] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.166001] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.166002] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.166020] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.166029] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166037] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164320 (xid=0x54aef)
[2018-02-22 18:07:47.166082] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.166163] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.166185] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164582 (xid=0x54af0)
[2018-02-22 18:07:47.166187] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.166212] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2018-02-22 18:07:47.166029] and [2018-02-22 18:07:47.166224]
[2018-02-22 18:07:47.166282] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164646 (xid=0x54af1)
[2018-02-22 18:07:47.166269] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.166300] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166404] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.166422] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.166421] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.166433] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166456] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.166508] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164697 (xid=0x54af2)
[2018-02-22 18:07:47.166518] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.166531] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.166525] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166552] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.166565] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166653] W [fuse-bridge.c:2312:fuse_writev_cbk] 0-glusterfs-fuse: 790460: WRITE => -1 gfid=91f97edd-97a3-4948-9d9d-469c2d174853 fd=0x2b2f23f02440 (Transport endpoint is not connected)
[2018-02-22 18:07:47.166688] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164725 (xid=0x54af3)
[2018-02-22 18:07:47.166704] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166947] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:47.167042] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b3t1_0001.fits (dace8bc0-f4d7-48b2-98ab-a0f768b3be8c) [Transport endpoint is not connected]
[2018-02-22 18:07:47.167069] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164600 (xid=0x544ac)
[2018-02-22 18:07:47.167085] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167085] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.167099] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b3t1_0001.fits (dace8bc0-f4d7-48b2-98ab-a0f768b3be8c) [Transport endpoint is not connected]
[2018-02-22 18:07:47.167128] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.167126] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.167145] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167152] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-scratch-dht: Found anomalies in /download/work/kstovall/57910/drx_57910_B0656+14_b3t1_0001.fits (gfid = dace8bc0-f4d7-48b2-98ab-a0f768b3be8c). Holes=1 overlaps=0
[2018-02-22 18:07:47.167191] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: dace8bc0-f4d7-48b2-98ab-a0f768b3be8c: failed to resolve (Transport endpoint is not connected)
[2018-02-22 18:07:47.167205] E [fuse-bridge.c:796:fuse_getattr_resume] 0-glusterfs-fuse: 790462: GETATTR 47480857159620 (dace8bc0-f4d7-48b2-98ab-a0f768b3be8c) resolution failed
[2018-02-22 18:07:47.167242] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164622 (xid=0x544ad)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:47.166456] and [2018-02-22 18:07:47.167174]
[2018-02-22 18:07:47.167257] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167304] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.167319] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.167326] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aff Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.167334] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167356] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.167463] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167478] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164669 (xid=0x544ae)
[2018-02-22 18:07:47.167501] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167502] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167554] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 790467: FSTAT() /download/work/kstovall/57910/drx_57910_B0656+14_b3t2_0001.fits => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:47.167560] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.167575] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.167582] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b00 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.167590] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167609] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.167656] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(PORTMAP) op(PORTBYBRICK(1)) called at 2018-02-22 18:07:47.164738 (xid=0x544af)
[2018-02-22 18:07:47.167669] W [MSGID: 114032] [client-handshake.c:1508:client_query_portmap_cbk] 2-scratch-client-1: received RPC status error, try again later [Transport endpoint is not connected]
[2018-02-22 18:07:47.167761] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164755 (xid=0x544b0)
[2018-02-22 18:07:47.167774] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167861] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.167873] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.167880] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b01 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.167890] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167909] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.167910] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165522 (xid=0x544b1)
[2018-02-22 18:07:47.167924] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.168078] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165551 (xid=0x544b2)
[2018-02-22 18:07:47.168222] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165676 (xid=0x544b3)
[2018-02-22 18:07:47.168423] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165710 (xid=0x544b4)
[2018-02-22 18:07:47.168568] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166138 (xid=0x544b5)
[2018-02-22 18:07:47.168712] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166343 (xid=0x544b6)
[2018-02-22 18:07:47.168856] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166378 (xid=0x544b7)
[2018-02-22 18:07:47.169003] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166492 (xid=0x544b8)
[2018-02-22 18:07:47.172418] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 7 times between [2018-02-22 18:07:47.167924] and [2018-02-22 18:07:47.169018]
[2018-02-22 18:07:47.176708] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:47.177249] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-0: Connected to scratch-client-0, attached to remote volume '/data/brick1/scratch'.
[2018-02-22 18:07:47.177261] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-22 18:07:47.177270] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-0: 4 fds open - Delaying child_up until they are re-opened
[2018-02-22 18:07:47.177839] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-02-22 18:07:47.177894] W [MSGID: 114031] [client-rpc-fops.c:799:client3_3_statfs_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.177935] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-0: Server lk version = 1
[2018-02-22 18:07:48.123815] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-02-22 18:07:48.548708] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-1: changing port to 49154 (from 0)
[2018-02-22 18:07:48.124421] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-02-22 18:07:48.553517] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:48.553937] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-1: Connected to scratch-client-1, attached to remote volume '/data/brick1/scratch'.
[2018-02-22 18:07:48.553946] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-22 18:07:48.553953] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-1: 4 fds open - Delaying child_up until they are re-opened
[2018-02-22 18:07:48.554300] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-02-22 18:07:48.554458] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-1: Server lk version = 1
[2018-02-22 18:07:49.139511] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.139545] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.139561] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b2e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.139578] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.139584] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:49.139616] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.139744] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2018-02-22 18:07:49.137922 (xid=0x54b13)
[2018-02-22 18:07:49.139762] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.139816] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.139836] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.139847] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b2f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.139859] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138017 (xid=0x54b14)
[2018-02-22 18:07:49.139861] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.139883] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.139896] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.139960] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.139976] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.139981] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138034 (xid=0x54b15)
[2018-02-22 18:07:49.139986] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b30 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.140001] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140009] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140016] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.140093] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.140094] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138210 (xid=0x54b16)
[2018-02-22 18:07:49.140112] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.140119] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140124] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b31 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.140146] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140164] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.140219] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.140219] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138307 (xid=0x54b17)
[2018-02-22 18:07:49.140236] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.140243] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140248] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b32 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.140270] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140277] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.140343] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138332 (xid=0x54b18)
[2018-02-22 18:07:49.140361] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.140361] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140405] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.140418] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b33 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.140429] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140458] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.140491] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138404 (xid=0x54b19)
[2018-02-22 18:07:49.140505] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140609] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138432 (xid=0x54b1a)
[2018-02-22 18:07:49.140622] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140643] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.140664] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.140671] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b34 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.140678] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140690] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.140711] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138454 (xid=0x54b1b)
[2018-02-22 18:07:49.140726] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140825] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138475 (xid=0x54b1c)
[2018-02-22 18:07:49.140924] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138506 (xid=0x54b1d)
[2018-02-22 18:07:49.141023] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138533 (xid=0x54b1e)
[2018-02-22 18:07:49.141122] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138560 (xid=0x54b1f)
[2018-02-22 18:07:49.141211] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138607 (xid=0x54b20)
[2018-02-22 18:07:49.141311] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138631 (xid=0x54b21)
[2018-02-22 18:07:49.141443] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138654 (xid=0x54b22)
[2018-02-22 18:07:49.141546] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138679 (xid=0x54b23)
[2018-02-22 18:07:49.141646] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138725 (xid=0x54b24)
[2018-02-22 18:07:49.141735] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138747 (xid=0x54b25)
[2018-02-22 18:07:49.141835] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138773 (xid=0x54b26)
[2018-02-22 18:07:49.141934] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138845 (xid=0x54b27)
[2018-02-22 18:07:49.142023] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138891 (xid=0x54b28)
[2018-02-22 18:07:49.142112] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138961 (xid=0x54b29)
[2018-02-22 18:07:49.142211] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138986 (xid=0x54b2a)
[2018-02-22 18:07:49.142312] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139037 (xid=0x54b2b)
[2018-02-22 18:07:49.142433] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139086 (xid=0x54b2c)
[2018-02-22 18:07:49.142523] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139164 (xid=0x54b2d)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 18 times between [2018-02-22 18:07:49.140726] and [2018-02-22 18:07:49.142536]
[2018-02-22 18:07:49.142797] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:49.142922] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2018-02-22 18:07:49.137938 (xid=0x544c6)
[2018-02-22 18:07:49.142939] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.142965] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.142984] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143128] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138060 (xid=0x544c7)
[2018-02-22 18:07:49.143144] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.143145] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143175] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143201] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143219] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143244] W [fuse-bridge.c:471:fuse_entry_cbk] 0-glusterfs-fuse: 790481: LOOKUP() /download => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:49.143247] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138083 (xid=0x544c8)
[2018-02-22 18:07:49.143261] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.143359] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138105 (xid=0x544c9)
[2018-02-22 18:07:49.143482] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138127 (xid=0x544ca)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:49.143261] and [2018-02-22 18:07:49.143503]
[2018-02-22 18:07:49.143515] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143574] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143605] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143639] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143638] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138147 (xid=0x544cb)
[2018-02-22 18:07:49.143670] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.143768] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143801] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143820] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138169 (xid=0x544cc)
[2018-02-22 18:07:49.143840] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143841] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.143875] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143912] W [fuse-bridge.c:471:fuse_entry_cbk] 0-glusterfs-fuse: 790485: LOOKUP() /download => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:49.143999] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138194 (xid=0x544cd)
[2018-02-22 18:07:49.144019] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.144169] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138231 (xid=0x544ce)
[2018-02-22 18:07:49.144295] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138253 (xid=0x544cf)
[2018-02-22 18:07:49.144404] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138273 (xid=0x544d0)
[2018-02-22 18:07:49.144541] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138294 (xid=0x544d1)
[2018-02-22 18:07:49.144695] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138348 (xid=0x544d2)
[2018-02-22 18:07:49.144847] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138582 (xid=0x544d3)
[2018-02-22 18:07:49.145003] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138707 (xid=0x544d4)
[2018-02-22 18:07:49.145155] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138796 (xid=0x544d5)
[2018-02-22 18:07:49.145311] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138819 (xid=0x544d6)
[2018-02-22 18:07:49.145422] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138863 (xid=0x544d7)
[2018-02-22 18:07:49.145567] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138910 (xid=0x544d8)
[2018-02-22 18:07:49.145721] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138936 (xid=0x544d9)
[2018-02-22 18:07:49.145871] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139010 (xid=0x544da)
[2018-02-22 18:07:49.146021] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139057 (xid=0x544db)
[2018-02-22 18:07:49.146173] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139105 (xid=0x544dc)
[2018-02-22 18:07:49.146326] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139130 (xid=0x544dd)
[2018-02-22 18:07:49.146488] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139190 (xid=0x544de)
[2018-02-22 18:07:49.146629] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140064 (xid=0x544df)
[2018-02-22 18:07:49.146783] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140327 (xid=0x544e0)
[2018-02-22 18:07:49.146935] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140501 (xid=0x544e1)
[2018-02-22 18:07:49.147086] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140522 (xid=0x544e2)
[2018-02-22 18:07:49.147236] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140545 (xid=0x544e3)
[2018-02-22 18:07:49.147393] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140564 (xid=0x544e4)
[2018-02-22 18:07:49.147536] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140582 (xid=0x544e5)
[2018-02-22 18:07:49.147641] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140601 (xid=0x544e6)
[2018-02-22 18:07:49.147790] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140623 (xid=0x544e7)
[2018-02-22 18:07:49.147846] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 790480: FLUSH() ERR => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:49.152660] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 26 times between [2018-02-22 18:07:49.144019] and [2018-02-22 18:07:49.147810]
[2018-02-22 18:07:49.156308] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.156348] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.156358] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b37 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.156377] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.156418] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 790496: FLUSH() ERR => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:49.156449] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.157895] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:49.158352] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-0: Connected to scratch-client-0, attached to remote volume '/data/brick1/scratch'.
[2018-02-22 18:07:49.158366] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-22 18:07:49.158708] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-0: 2 fds open - Delaying child_up until they are re-opened
[2018-02-22 18:07:49.159196] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-02-22 18:07:49.159242] W [MSGID: 114031] [client-rpc-fops.c:799:client3_3_statfs_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.159319] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-0: Server lk version = 1
[2018-02-22 18:07:49.181472] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/jp.pbs (e9f57eb0-e8a2-40dc-b56f-59eea34941a6) [Transport endpoint is not connected]
-------------- next part --------------
[2018-02-22 18:07:45.226382] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7fd5ba679000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:45.236646] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-0
[2018-02-22 18:07:45.236796] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:45.236854] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits
[2018-02-22 18:07:45.236910] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:45.236966] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log
[2018-02-22 18:07:45.237007] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-0
[2018-02-22 18:07:45.262237] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-1 (version: 3.8.4)
[2018-02-22 18:07:45.416142] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-1-22-0
[2018-02-22 18:07:45.416218] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OSZICAR
[2018-02-22 18:07:45.416254] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OUTCAR
[2018-02-22 18:07:45.416306] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-1-22-0
[2018-02-22 18:07:46.191329] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7fd5ba799000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.192099] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba8e3000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.192498] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba401000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.192876] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba7c5000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.195455] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-1
[2018-02-22 18:07:46.195537] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:46.195589] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:46.195634] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits
[2018-02-22 18:07:46.195675] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log
[2018-02-22 18:07:46.195742] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-1
[2018-02-22 18:07:48.559889] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-2 (version: 3.8.4)
[2018-02-22 18:07:49.144466] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7fd5bbf7b000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:49.145265] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba53b000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:49.149583] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-2
[2018-02-22 18:07:49.149670] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:49.149726] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:49.149825] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-2
-------------- next part --------------
[2018-02-22 18:07:45.226574] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32cabd1000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:45.228378] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-0
[2018-02-22 18:07:45.228601] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits
[2018-02-22 18:07:45.228681] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:45.228737] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:45.228793] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log
[2018-02-22 18:07:45.230202] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1021)
[2018-02-22 18:07:45.230273] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1021) failed
[2018-02-22 18:07:45.230292] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54974, Program: GlusterFS 3.3, ProgVers: 330, Proc: 12) to rpc-transport (rdma.scratch-server)
[2018-02-22 18:07:45.230386] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x10e82) [0x7f32dc180e82] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x1cac6) [0x7f32d7dd4ac6] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed
[2018-02-22 18:07:45.230463] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-0
[2018-02-22 18:07:45.261717] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-1 (version: 3.8.4)
[2018-02-22 18:07:45.416150] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-0-22-0
[2018-02-22 18:07:45.416244] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OSZICAR
[2018-02-22 18:07:45.416315] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OUTCAR
[2018-02-22 18:07:45.416373] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-0-22-0
[2018-02-22 18:07:46.191486] W [MSGID: 103004] [rdma.c:2327:__gf_rdma_register_local_mr_for_rdma] 0-rpc-transport/rdma: registering memory for IBV_ACCESS_LOCAL_WRITE failed [Invalid argument]
[2018-02-22 18:07:46.191549] W [MSGID: 103004] [rdma.c:3701:gf_rdma_do_reads] 0-rpc-transport/rdma: registering local memory for rdma read failed
[2018-02-22 18:07:46.191570] W [MSGID: 103061] [rdma.c:3969:gf_rdma_recv_request] 0-rpc-transport/rdma: rdma read from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.191600] W [MSGID: 103063] [rdma.c:4084:gf_rdma_process_recv] 0-rpc-transport/rdma: receiving a request from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.191611] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32caced000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.192513] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca41d000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.192880] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca9c7000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.193399] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca7d3000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.193948] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-1
[2018-02-22 18:07:46.194011] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023)
[2018-02-22 18:07:46.194085] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:46.194086] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.194118] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:46.194131] W [MSGID: 103026] [rdma.c:3670:gf_rdma_do_reads] 0-rpc-transport/rdma: transport not connected to peer (172.17.1.61:1023), not doing rdma reads
[2018-02-22 18:07:46.194191] W [MSGID: 103061] [rdma.c:3969:gf_rdma_recv_request] 0-rpc-transport/rdma: rdma read from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.194191] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log
[2018-02-22 18:07:46.194214] W [MSGID: 103063] [rdma.c:4084:gf_rdma_process_recv] 0-rpc-transport/rdma: receiving a request from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.194252] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023)
[2018-02-22 18:07:46.194264] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.194342] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54a72, Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (rdma.scratch-server)
[2018-02-22 18:07:46.194288] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54a6f, Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (rdma.scratch-server)
[2018-02-22 18:07:46.194404] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x112bb) [0x7f32dc1812bb] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x1c651) [0x7f32d7dd4651] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed
[2018-02-22 18:07:46.194452] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x112bb) [0x7f32dc1812bb] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x1c651) [0x7f32d7dd4651] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed
[2018-02-22 18:07:46.194562] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-1
[2018-02-22 18:07:47.170678] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32cabaf000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.170781] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-2 (version: 3.8.4)
[2018-02-22 18:07:47.171160] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca9eb000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.171941] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-2
[2018-02-22 18:07:47.171968] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023)
[2018-02-22 18:07:47.172145] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed
[2018-02-22 18:07:47.172168] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54aee, Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (rdma.scratch-server)
[2018-02-22 18:07:47.172164] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glusterfs: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-2
[2018-02-22 18:07:47.172214] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cf25) [0x7f32d7de4f25] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cec3) [0x7f32d7de4ec3] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed
[2018-02-22 18:07:47.172248] E [server.c:207:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cf25) [0x7f32d7de4f25] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cec3) [0x7f32d7de4ec3] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x882e) [0x7f32d7dc082e] ) 0-: Reply submission failed
[2018-02-22 18:07:47.183288] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-3 (version: 3.8.4)
[2018-02-22 18:07:49.144790] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32ca3b7000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:49.147498] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-3
[2018-02-22 18:07:49.147530] W [MSGID: 103026] [rdma.c:3670:gf_rdma_do_reads] 0-rpc-transport/rdma: transport not connected to peer (172.17.1.61:1023), not doing rdma reads
[2018-02-22 18:07:49.147592] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:49.147596] W [MSGID: 103061] [rdma.c:3969:gf_rdma_recv_request] 0-rpc-transport/rdma: rdma read from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:49.147646] W [MSGID: 103063] [rdma.c:4084:gf_rdma_process_recv] 0-rpc-transport/rdma: receiving a request from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:49.147659] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:49.147655] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023)
[2018-02-22 18:07:49.147797] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed
[2018-02-22 18:07:49.147824] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54b13, Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (rdma.scratch-server)
[2018-02-22 18:07:49.148988] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x12857) [0x7f32dc182857] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2b427) [0x7f32d7de3427] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed
[2018-02-22 18:07:49.149072] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-3
[2018-02-22 18:07:49.164480] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-4 (version: 3.8.4)
[2018-02-22 18:07:46.194027] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits


More information about the Gluster-users mailing list