[Gluster-users] Problems with write-behind with large files on Gluster 3.8.4
Jim Prewett
download at carc.unm.edu
Mon Feb 26 21:19:03 UTC 2018
Hello,
I'm having problems when write-behind is enabled on Gluster 3.8.4.
I have 2 Gluster servers each with a single brick that is mirrored between
them. The code causing these issues reads two data files each approx.
128G in size. It opens a third file, mmap()'s that file, and
subsequently reads and writes to it. The third file, on sucessful runs
(without write-behind enabled) is ultimately approx. 224G in size.
The servers have the IP addresses 172.17.2.254 and 172.17.2.255 and the
client has the IP address 172.17.1.61. These are all IP over InfiniBand.
I'm attaching logfiles for the brick and for the volume from each of the
servers and for the client. I'm also attaching the output of "gluster
volume info" and "gluster volume get <volume> all".
I have only noticed problems with write-behind being enabled with this one
particular workload. When I ran it under strace, I see it seeking all
over the place and reading and writing little bits of data to/from the
third file.
For now, I'm leaving write-behind disabled. What are the performance
implications of this for jobs that don't have this strange access pattern?
My co-worker who usually maintains the Gluster filesystems here is busy
having a baby right now and I've gotten it while he's out, so I'm /really/
new to Gluster and am not confident that anything is correct in my
configuration (nor do I have a specific reason to doubt its correctness! :)
I have checked the InfiniBand fabric for errors and do not see any beyond
the normal PortXmitWait counter. There is no firewall on any of these
machines. Their system clocks seem to all be synchronized.
Is there anything additional I can provide to help diagnose this problem?
Thanks for any help you can provide! :)
Jim
James E. Prewett Jim at Prewett.org download at hpc.unm.edu
Systems Team Leader LoGS: http://www.hpc.unm.edu/~download/LoGS/
Designated Security Officer OpenPGP key: pub 1024D/31816D93
HPC Systems Engineer III UNM HPC 505.277.8210
-------------- next part --------------
Option Value
------ -----
cluster.lookup-unhashed on
cluster.lookup-optimize on
cluster.min-free-disk 10%
cluster.min-free-inodes 5%
cluster.rebalance-stats off
cluster.subvols-per-directory (null)
cluster.readdir-optimize off
cluster.rsync-hash-regex (null)
cluster.extra-hash-regex (null)
cluster.dht-xattr-name trusted.glusterfs.dht
cluster.randomize-hash-range-by-gfid off
cluster.rebal-throttle normal
cluster.lock-migration off
cluster.local-volume-name (null)
cluster.weighted-rebalance on
cluster.switch-pattern (null)
cluster.entry-change-log on
cluster.read-subvolume (null)
cluster.read-subvolume-index -1
cluster.read-hash-mode 1
cluster.background-self-heal-count 8
cluster.metadata-self-heal on
cluster.data-self-heal on
cluster.entry-self-heal on
cluster.self-heal-daemon on
cluster.heal-timeout 600
cluster.self-heal-window-size 1
cluster.data-change-log on
cluster.metadata-change-log on
cluster.data-self-heal-algorithm (null)
cluster.eager-lock on
disperse.eager-lock on
cluster.quorum-type none
cluster.quorum-count (null)
cluster.choose-local true
cluster.self-heal-readdir-size 1KB
cluster.post-op-delay-secs 1
cluster.ensure-durability on
cluster.consistent-metadata no
cluster.heal-wait-queue-length 128
cluster.favorite-child-policy none
cluster.stripe-block-size 128KB
cluster.stripe-coalesce true
diagnostics.latency-measurement on
diagnostics.dump-fd-stats off
diagnostics.count-fop-hits on
diagnostics.brick-log-level INFO
diagnostics.client-log-level INFO
diagnostics.brick-sys-log-level CRITICAL
diagnostics.client-sys-log-level CRITICAL
diagnostics.brick-logger (null)
diagnostics.client-logger (null)
diagnostics.brick-log-format (null)
diagnostics.client-log-format (null)
diagnostics.brick-log-buf-size 5
diagnostics.client-log-buf-size 5
diagnostics.brick-log-flush-timeout 120
diagnostics.client-log-flush-timeout 120
diagnostics.stats-dump-interval 0
diagnostics.fop-sample-interval 0
diagnostics.fop-sample-buf-size 65535
diagnostics.stats-dnscache-ttl-sec 86400
performance.cache-max-file-size 0
performance.cache-min-file-size 0
performance.cache-refresh-timeout 1
performance.cache-priority
performance.cache-size 8GB
performance.io-thread-count 24
performance.high-prio-threads 16
performance.normal-prio-threads 16
performance.low-prio-threads 16
performance.least-prio-threads 1
performance.enable-least-priority on
performance.least-rate-limit 0
performance.cache-size 8GB
performance.flush-behind on
performance.nfs.flush-behind on
performance.write-behind-window-size 8MB
performance.resync-failed-syncs-after-fsyncon
performance.nfs.write-behind-window-size1MB
performance.strict-o-direct off
performance.nfs.strict-o-direct off
performance.strict-write-ordering off
performance.nfs.strict-write-ordering off
performance.lazy-open yes
performance.read-after-open no
performance.read-ahead-page-count 4
performance.md-cache-timeout 1
performance.cache-swift-metadata true
features.encryption off
encryption.master-key (null)
encryption.data-key-size 256
encryption.block-size 4096
network.frame-timeout 1800
network.ping-timeout 42
network.tcp-window-size (null)
features.lock-heal off
features.grace-timeout 10
network.remote-dio disable
client.event-threads 8
network.ping-timeout 42
network.tcp-window-size (null)
network.inode-lru-limit 16384
auth.allow *
auth.reject (null)
transport.keepalive (null)
server.allow-insecure (null)
server.root-squash off
server.anonuid 65534
server.anongid 65534
server.statedump-path /var/run/gluster
server.outstanding-rpc-limit 512
features.lock-heal off
features.grace-timeout 10
server.ssl (null)
auth.ssl-allow *
server.manage-gids off
server.dynamic-auth on
client.send-gids on
server.gid-timeout 300
server.own-thread (null)
server.event-threads 32
ssl.own-cert (null)
ssl.private-key (null)
ssl.ca-list (null)
ssl.crl-path (null)
ssl.certificate-depth (null)
ssl.cipher-list (null)
ssl.dh-param (null)
ssl.ec-curve (null)
performance.write-behind off
performance.read-ahead on
performance.readdir-ahead on
performance.io-cache on
performance.quick-read on
performance.open-behind on
performance.stat-prefetch on
performance.client-io-threads on
performance.nfs.write-behind on
performance.nfs.read-ahead off
performance.nfs.io-cache off
performance.nfs.quick-read off
performance.nfs.stat-prefetch off
performance.nfs.io-threads off
performance.force-readdirp true
features.uss off
features.snapshot-directory .snaps
features.show-snapshot-directory off
network.compression off
network.compression.window-size -15
network.compression.mem-level 8
network.compression.min-size 0
network.compression.compression-level -1
network.compression.debug false
features.limit-usage (null)
features.quota-timeout 0
features.default-soft-limit 80%
features.soft-timeout 60
features.hard-timeout 5
features.alert-time 86400
features.quota-deem-statfs off
geo-replication.indexing off
geo-replication.indexing off
geo-replication.ignore-pid-check off
geo-replication.ignore-pid-check off
features.quota off
features.inode-quota off
features.bitrot disable
debug.trace off
debug.log-history no
debug.log-file no
debug.exclude-ops (null)
debug.include-ops (null)
debug.error-gen off
debug.error-failure (null)
debug.error-number (null)
debug.random-failure off
debug.error-fops (null)
nfs.enable-ino32 no
nfs.mem-factor 15
nfs.export-dirs on
nfs.export-volumes on
nfs.addr-namelookup off
nfs.dynamic-volumes off
nfs.register-with-portmap on
nfs.outstanding-rpc-limit 16
nfs.port 2049
nfs.rpc-auth-unix on
nfs.rpc-auth-null on
nfs.rpc-auth-allow all
nfs.rpc-auth-reject none
nfs.ports-insecure off
nfs.trusted-sync off
nfs.trusted-write off
nfs.volume-access read-write
nfs.export-dir
nfs.disable on
nfs.nlm on
nfs.acl on
nfs.mount-udp off
nfs.mount-rmtab /var/lib/glusterd/nfs/rmtab
nfs.rpc-statd /sbin/rpc.statd
nfs.server-aux-gids off
nfs.drc off
nfs.drc-size 0x20000
nfs.read-size (1 * 1048576ULL)
nfs.write-size (1 * 1048576ULL)
nfs.readdir-size (1 * 1048576ULL)
nfs.rdirplus on
nfs.exports-auth-enable (null)
nfs.auth-refresh-interval-sec (null)
nfs.auth-cache-ttl-sec (null)
features.read-only off
features.worm off
features.worm-file-level off
features.default-retention-period 120
features.retention-mode relax
features.auto-commit-period 180
storage.linux-aio off
storage.batch-fsync-mode reverse-fsync
storage.batch-fsync-delay-usec 0
storage.owner-uid -1
storage.owner-gid -1
storage.node-uuid-pathinfo off
storage.health-check-interval 30
storage.build-pgfid off
storage.bd-aio off
config.transport tcp,rdma
cluster.server-quorum-type off
cluster.server-quorum-ratio 0
changelog.changelog off
changelog.changelog-dir (null)
changelog.encoding ascii
changelog.rollover-time 15
changelog.fsync-interval 5
changelog.changelog-barrier-timeout 120
changelog.capture-del-path off
features.barrier disable
features.barrier-timeout 120
features.trash off
features.trash-dir .trashcan
features.trash-eliminate-path (null)
features.trash-max-filesize 5MB
features.trash-internal-op off
cluster.enable-shared-storage disable
cluster.write-freq-threshold 0
cluster.read-freq-threshold 0
cluster.tier-pause off
cluster.tier-promote-frequency 120
cluster.tier-demote-frequency 3600
cluster.watermark-hi 90
cluster.watermark-low 75
cluster.tier-mode cache
cluster.tier-max-promote-file-size 0
cluster.tier-max-mb 4000
cluster.tier-max-files 10000
features.ctr-enabled off
features.record-counters off
features.ctr-record-metadata-heat off
features.ctr_link_consistency off
features.ctr_lookupheal_link_timeout 300
features.ctr_lookupheal_inode_timeout 300
features.ctr-sql-db-cachesize 1000
features.ctr-sql-db-wal-autocheckpoint 1000
locks.trace off
locks.mandatory-locking off
cluster.disperse-self-heal-daemon enable
cluster.quorum-reads no
client.bind-insecure (null)
ganesha.enable off
features.shard off
features.shard-block-size 4MB
features.scrub-throttle lazy
features.scrub-freq biweekly
features.scrub false
features.expiry-time 120
features.cache-invalidation off
features.cache-invalidation-timeout 60
features.leases off
features.lease-lock-recall-timeout 60
disperse.background-heals 8
disperse.heal-wait-qlength 128
cluster.heal-timeout 600
dht.force-readdirp on
disperse.read-policy round-robin
cluster.shd-max-threads 1
cluster.shd-wait-qlength 1024
cluster.locking-scheme full
cluster.granular-entry-heal no
-------------- next part --------------
Volume Name: scratch
Type: Stripe
Volume ID: 38f6c032-b6bf-4923-8751-5bd93650e2d0
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp,rdma
Bricks:
Brick1: server01ib:/data/brick1/scratch
Brick2: server02ib:/data/brick1/scratch
Options Reconfigured:
config.transport: tcp,rdma
performance.cache-size: 8GB
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
performance.readdir-ahead: on
cluster.lookup-optimize: on
client.event-threads: 8
performance.client-io-threads: on
performance.io-thread-count: 24
performance.write-behind-window-size: 8MB
server.event-threads: 32
performance.write-behind: off
performance.stat-prefetch: on
performance.resync-failed-syncs-after-fsync: on
server.outstanding-rpc-limit: 512
-------------- next part --------------
[2018-02-22 18:07:45.241970] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.242018] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.242681] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.242704] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.242728] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.242737] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.242753] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.242762] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.242786] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.242795] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243058] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.243071] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243090] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.243098] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243116] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.243124] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243141] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.243149] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243384] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1022
[2018-02-22 18:07:45.243396] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.243416] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f6462b85000, wc.byte_len = 45056, post->reused = 1
[2018-02-22 18:07:45.243692] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f6462b85000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:45.244143] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f6461e73000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.159515] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f6462b7d000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.171016] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f6462b75000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.171548] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f6462b77000, wc.byte_len = 131072, post->reused = 1
-------------- next part --------------
[2018-02-22 18:07:45.241029] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023
[2018-02-22 18:07:45.241095] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.241269] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023
[2018-02-22 18:07:45.241333] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.241375] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023
[2018-02-22 18:07:45.241394] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.241461] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023
[2018-02-22 18:07:45.241482] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.241512] W [rpcsvc.c:265:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 172.17.1.61:1023
[2018-02-22 18:07:45.241529] E [rpcsvc.c:560:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2018-02-22 18:07:45.241560] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f10e7351000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:45.242274] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f10e7349000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.158767] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f10e733d000, wc.byte_len = 131072, post->reused = 1
-------------- next part --------------
[2018-02-22 18:07:45.221277] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.221309] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.221320] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54987 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.221334] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.221365] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.221368] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:45.221576] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.210946 (xid=0x54974)
[2018-02-22 18:07:45.221596] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.224781] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54988 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.224902] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54989 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225011] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225074] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225630] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225697] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225753] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225801] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5498f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.225830] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54990 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.226338] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54991 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.226383] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54992 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.226452] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54993 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.226947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54994 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227452] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54995 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227491] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54996 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227524] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54997 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227568] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54998 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227615] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54999 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227654] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227738] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227801] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227863] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227920] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.227972] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x5499f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228767] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228803] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790331: READ => -1 gfid=3a86afe5-7392-49a5-b60a-e0c93b050c01 fd=0x2b2f23f023b4 (Transport endpoint is not connected)
[2018-02-22 18:07:45.228826] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228864] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790330: READ => -1 gfid=3a86afe5-7392-49a5-b60a-e0c93b050c01 fd=0x2b2f23f023b4 (Transport endpoint is not connected)
[2018-02-22 18:07:45.228872] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228919] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228966] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.228989] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.229009] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229050] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229102] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229122] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.220065 (xid=0x54986)
[2018-02-22 18:07:45.229151] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229193] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549a9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229408] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549aa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 35 times between [2018-02-22 18:07:45.221365] and [2018-02-22 18:07:45.229445]
[2018-02-22 18:07:45.229449] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.229488] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ab Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 37 times between [2018-02-22 18:07:45.221334] and [2018-02-22 18:07:45.229507]
[2018-02-22 18:07:45.229524] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.229511] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.229549] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54385 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.229561] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.229572] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ac Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 37 times between [2018-02-22 18:07:45.221309] and [2018-02-22 18:07:45.229571]
[2018-02-22 18:07:45.229587] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.229583] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.229620] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:45.229623] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)" repeated 38 times between [2018-02-22 18:07:45.221277] and [2018-02-22 18:07:45.229677]
[2018-02-22 18:07:45.229678] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.229702] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ad Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229713] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.229736] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.220053 (xid=0x5437b)
[2018-02-22 18:07:45.229753] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.229769] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.229800] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.229812] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ae Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.229866] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.220767 (xid=0x5437c)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.229623] and [2018-02-22 18:07:45.229838]
[2018-02-22 18:07:45.229883] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.229867] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.229903] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549af Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.229713] and [2018-02-22 18:07:45.229913]
[2018-02-22 18:07:45.229928] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.229978] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.229991] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 2 times between [2018-02-22 18:07:45.229800] and [2018-02-22 18:07:45.229990]
[2018-02-22 18:07:45.230004] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.222057 (xid=0x5437d)
[2018-02-22 18:07:45.230037] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230004] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230040] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230083] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230088] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230126] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230130] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.222694 (xid=0x5437e)
[2018-02-22 18:07:45.230129] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230149] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230163] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54386 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.230165] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230176] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.230004] and [2018-02-22 18:07:45.230183]
[2018-02-22 18:07:45.230193] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.230187] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.230088] and [2018-02-22 18:07:45.230195]
[2018-02-22 18:07:45.230220] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230214] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230236] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230242] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54387 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.230234] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230245] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230257] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230267] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.224869 (xid=0x5437f)
[2018-02-22 18:07:45.230273] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230264] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.230274] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230308] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230314] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230344] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 2 times between [2018-02-22 18:07:45.230257] and [2018-02-22 18:07:45.230367]
[2018-02-22 18:07:45.230368] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230377] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230389] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.225018 (xid=0x54380)
The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)" repeated 2 times between [2018-02-22 18:07:45.230314] and [2018-02-22 18:07:45.230420]
[2018-02-22 18:07:45.230423] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230422] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230424] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230454] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230466] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230476] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54388 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.230497] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230502] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.226409 (xid=0x54381)
[2018-02-22 18:07:45.230514] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54389 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 7 times between [2018-02-22 18:07:45.229883] and [2018-02-22 18:07:45.230485]
[2018-02-22 18:07:45.230492] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230513] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230536] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230535] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.230542] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230557] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.230565] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230570] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230588] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230572] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230600] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 5 times between [2018-02-22 18:07:45.230245] and [2018-02-22 18:07:45.230600]
[2018-02-22 18:07:45.230624] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549b9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed" repeated 2 times between [2018-02-22 18:07:45.230422] and [2018-02-22 18:07:45.230623]
[2018-02-22 18:07:45.230612] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230625] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230618] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230635] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230652] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.227685 (xid=0x54382)
[2018-02-22 18:07:45.230665] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.230659] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230660] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.230612] and [2018-02-22 18:07:45.230674]
The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)" repeated 2 times between [2018-02-22 18:07:45.230565] and [2018-02-22 18:07:45.230688]
[2018-02-22 18:07:45.230677] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230690] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230717] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ba Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230728] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230740] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230763] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230764] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.228029 (xid=0x54383)
[2018-02-22 18:07:45.230762] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230773] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230784] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230782] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230786] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230788] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.230804] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.230677] and [2018-02-22 18:07:45.230802]
[2018-02-22 18:07:45.230829] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230819] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.230845] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230864] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230867] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230852] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.230874] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230898] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.229052 (xid=0x54384)
[2018-02-22 18:07:45.230900] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230914] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230914] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230901] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.230660] and [2018-02-22 18:07:45.230923]
[2018-02-22 18:07:45.230928] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.230946] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549be Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230956] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.230961] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790385: READ => -1 gfid=3a86afe5-7392-49a5-b60a-e0c93b050c01 fd=0x2b2f23f023b4 (Transport endpoint is not connected)
[2018-02-22 18:07:45.230960] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230972] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.230986] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.230990] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.231005] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549bf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.230990] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.231008] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:45.231015] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231032] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:45.231042] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.231029] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231031] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231073] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231080] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231078] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:45.231085] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:45.231104] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231099] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231112] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231123] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231130] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231130] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231144] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231136] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231164] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231148] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231149] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231175] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231184] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231207] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5438f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231209] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231210] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231211] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231219] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231238] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231231] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231255] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231264] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54390 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231261] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231261] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231275] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231283] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231279] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231297] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231303] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231306] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231310] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231324] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231333] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54391 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231326] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231340] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231341] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231335] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231377] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231388] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231383] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231381] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t1_0001.fits (3a86afe5-7392-49a5-b60a-e0c93b050c01) [Transport endpoint is not connected]
[2018-02-22 18:07:45.231417] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231419] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231432] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231434] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t1_0001.fits (3a86afe5-7392-49a5-b60a-e0c93b050c01) [Transport endpoint is not connected]
[2018-02-22 18:07:45.231446] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231452] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231463] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-scratch-dht: Found anomalies in /download/work/kstovall/57910/drx_57910_B0656+14_b2t1_0001.fits (gfid = 3a86afe5-7392-49a5-b60a-e0c93b050c01). Holes=1 overlaps=0
[2018-02-22 18:07:45.231456] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231471] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231488] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: 3a86afe5-7392-49a5-b60a-e0c93b050c01: failed to resolve (Transport endpoint is not connected)
[2018-02-22 18:07:45.231490] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231486] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231501] E [fuse-bridge.c:796:fuse_getattr_resume] 0-glusterfs-fuse: 790393: GETATTR 47480857159808 (3a86afe5-7392-49a5-b60a-e0c93b050c01) resolution failed
[2018-02-22 18:07:45.231494] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231509] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231516] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54392 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231515] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231520] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231540] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231538] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231542] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231564] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231570] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54393 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231580] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231587] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231594] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231604] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231617] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231610] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231626] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231638] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231638] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231664] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54394 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231653] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231667] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231688] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231694] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231696] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231714] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231715] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231721] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231747] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231751] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231765] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549c8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231789] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231800] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231801] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231816] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231823] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54395 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231824] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ca Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231822] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231833] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231828] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231850] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231843] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231865] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231863] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231873] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.231879] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231885] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231899] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231917] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.231919] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.231922] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.231928] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54396 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231925] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.231954] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.231959] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.231949] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231967] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.231985] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232000] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54398 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.231999] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232002] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232021] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232039] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232048] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232050] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232060] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54397 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.232057] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232064] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232075] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232073] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232092] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232098] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232100] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232104] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232114] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ce Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232121] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232136] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54399 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.232138] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232159] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232156] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232154] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232183] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232194] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549cf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232203] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232214] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232216] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232226] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232226] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232241] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232249] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232262] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232267] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232268] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232279] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.232276] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232302] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232303] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232314] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232320] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232329] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232328] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232335] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232344] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.232334] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232356] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232364] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232400] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232404] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232448] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232456] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232471] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232479] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232476] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232493] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.232496] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.232268] and [2018-02-22 18:07:45.232525]
[2018-02-22 18:07:45.232532] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232552] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.232564] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.232571] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.232364] and [2018-02-22 18:07:45.232579]
[2018-02-22 18:07:45.232592] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.232592] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232632] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232654] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232668] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232680] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.232708] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.232703] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.232706] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.232726] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.232757] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.233456] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.233454] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.233478] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.233485] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.233494] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.233483] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.233498] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233513] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233540] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.233532] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.233580] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.233592] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.233602] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5439f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.233614] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233630] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.233670] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.233692] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.233690] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.233706] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233739] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.233756] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.233755] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.233766] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233756] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.233795] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.233833] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.233835] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.233860] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.233871] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233892] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.233900] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.233908] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.233926] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.233936] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233937] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.233939] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.233960] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.233967] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.233978] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.233996] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234011] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234023] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234016] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234035] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234065] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234077] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.234079] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.234096] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549d9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.234100] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234106] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234119] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.234115] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234136] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234146] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234154] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234157] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234174] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234184] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234197] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234220] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234218] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234231] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.234227] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234247] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.234256] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549da Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.234262] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop" repeated 3 times between [2018-02-22 18:07:45.234065] and [2018-02-22 18:07:45.234264]
[2018-02-22 18:07:45.234283] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.234511] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234513] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234530] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234540] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234556] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234567] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234572] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234583] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234593] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234613] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234622] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234629] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234638] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234646] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234670] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234677] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234692] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234707] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543a9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234716] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234725] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234738] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234746] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543aa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234751] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234754] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234784] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234828] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234834] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234842] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234859] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ab Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234867] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234879] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234880] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234881] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234907] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ac Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234917] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234929] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234923] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234954] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ad Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.234953] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.234965] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.234988] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.234996] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.234998] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235015] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ae Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235024] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235037] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235059] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235075] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235083] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543af Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235094] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235119] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235182] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235197] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235196] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235205] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235231] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235258] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235260] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235273] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235280] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235297] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235321] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235336] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235344] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235352] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235362] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235399] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235413] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235412] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235423] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235436] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235444] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235460] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235465] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235486] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235495] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.235501] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.235514] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.235529] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543b5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.235537] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.235558] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.236105] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:45.236245] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234482 (xid=0x549dc)
[2018-02-22 18:07:45.236264] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.236558] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234491 (xid=0x549de)
[2018-02-22 18:07:45.236672] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234655 (xid=0x549df)
[2018-02-22 18:07:45.236899] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234699 (xid=0x549e0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2018-02-22 18:07:45.236264] and [2018-02-22 18:07:45.236914]
[2018-02-22 18:07:45.236931] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237010] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234804 (xid=0x549e1)
[2018-02-22 18:07:45.237025] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237045] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237061] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237069] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ee Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237076] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237135] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234822 (xid=0x549e2)
[2018-02-22 18:07:45.237357] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(PORTMAP) op(PORTBYBRICK(1)) called at 2018-02-22 18:07:45.234954 (xid=0x549e3)
[2018-02-22 18:07:45.237150] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237431] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237430] W [MSGID: 114032] [client-handshake.c:1508:client_query_portmap_cbk] 2-scratch-client-0: received RPC status error, try again later [Transport endpoint is not connected]
[2018-02-22 18:07:45.237450] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237458] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ef Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237466] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.234283] and [2018-02-22 18:07:45.237488]
[2018-02-22 18:07:45.237510] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237529] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237541] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237541] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.234979 (xid=0x549e4)
[2018-02-22 18:07:45.237549] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237555] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237566] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237554] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237585] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237584] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237594] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237623] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237657] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237658] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237675] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237673] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235016 (xid=0x549e5)
[2018-02-22 18:07:45.237690] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237696] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237698] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237710] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237721] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237736] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237748] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237763] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237771] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237765] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237784] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237799] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235054 (xid=0x549e6)
[2018-02-22 18:07:45.237797] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237813] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237827] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237814] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237835] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237842] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237862] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.237886] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237888] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237905] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237913] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237926] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.237934] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235159 (xid=0x549e7)
[2018-02-22 18:07:45.237942] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.237941] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237953] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.237950] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.237965] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.237862] and [2018-02-22 18:07:45.237976]
[2018-02-22 18:07:45.237986] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238005] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238012] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.237950] and [2018-02-22 18:07:45.238020]
[2018-02-22 18:07:45.238034] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238038] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.238033] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238070] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549f9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238068] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238079] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235589 (xid=0x549e8)
[2018-02-22 18:07:45.238080] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238097] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238100] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238110] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238122] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238111] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238129] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238147] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238174] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238183] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238182] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238195] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238207] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238208] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238220] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238214] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235705 (xid=0x549e9)
[2018-02-22 18:07:45.238229] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.238195] and [2018-02-22 18:07:45.238243]
[2018-02-22 18:07:45.238247] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238253] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238266] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238278] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238287] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238297] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238302] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238303] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238327] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549fe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238337] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238357] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235740 (xid=0x549ea)
[2018-02-22 18:07:45.238381] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238392] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238411] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238418] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x549ff Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238424] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:45.238297] and [2018-02-22 18:07:45.238446]
[2018-02-22 18:07:45.238469] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235777 (xid=0x549eb)
[2018-02-22 18:07:45.238465] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238485] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238491] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238507] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a00 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238516] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238530] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238536] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238531] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238556] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a01 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238579] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235812 (xid=0x549ec)
[2018-02-22 18:07:45.238566] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238583] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238579] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238595] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238610] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238621] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a02 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238629] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238643] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238651] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238644] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238672] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a03 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238682] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238692] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238696] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238709] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235840 (xid=0x549ed)
[2018-02-22 18:07:45.238711] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238729] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a05 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238739] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238752] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238752] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238761] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238788] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a04 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238798] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238812] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238823] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238824] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238837] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a06 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238847] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238862] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238869] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238865] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238885] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a07 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238893] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238903] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238907] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238922] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238935] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238943] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.238956] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.238964] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.238958] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.238977] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a09 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.238990] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239004] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239007] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239024] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239038] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a08 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239047] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239060] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239069] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239062] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239081] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239090] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239100] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239105] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239119] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239132] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239140] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239153] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239160] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239155] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239174] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239183] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239190] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239207] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239208] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239221] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239233] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239243] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239244] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239254] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239259] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a0e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239268] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.239279] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239270] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.239286] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239296] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ce Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.239293] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239307] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a10 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239305] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239309] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:45.239328] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239333] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.239345] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239330] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239361] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a11 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239383] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2018-02-22 18:07:45.239233] and [2018-02-22 18:07:45.239384]
[2018-02-22 18:07:45.239410] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239401] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239428] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a12 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239432] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235590 (xid=0x543b8)
[2018-02-22 18:07:45.239449] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239453] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239457] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239459] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239483] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a13 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239484] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239500] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239506] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239523] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239528] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239541] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a14 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239551] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239557] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235614 (xid=0x543b9)
[2018-02-22 18:07:45.239573] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239573] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239574] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239589] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239607] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a17 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239616] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239625] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.239633] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239638] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.239644] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239650] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239657] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543cf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.239666] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a16 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239676] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239682] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235622 (xid=0x543ba)
[2018-02-22 18:07:45.239685] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239696] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239704] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.239712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239733] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239748] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239758] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a15 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239769] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239789] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239799] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235647 (xid=0x543bb)
[2018-02-22 18:07:45.239798] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.239803] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239826] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.239824] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.239833] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a18 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239797] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239827] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239835] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239856] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.239851] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239881] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239884] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239899] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a19 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239898] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239911] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239921] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239925] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.239932] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.239936] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.239947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.239947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.239950] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239954] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235673 (xid=0x543bc)
[2018-02-22 18:07:45.239956] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239969] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239969] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.239988] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.239994] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.239998] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240006] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240016] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240026] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240036] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240035] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.240043] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240039] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240058] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240062] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240073] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240077] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240082] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235687 (xid=0x543bd)
[2018-02-22 18:07:45.240100] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240099] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240084] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240115] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240098] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240127] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240136] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240149] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240164] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240166] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240173] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240184] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240193] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240203] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240210] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a1f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240211] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240220] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240222] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235708 (xid=0x543be)
[2018-02-22 18:07:45.240237] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240239] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240243] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240257] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240267] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a20 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240275] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240286] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240291] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240297] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240311] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a21 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240316] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.240321] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240335] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240339] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240342] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235728 (xid=0x543bf)
[2018-02-22 18:07:45.240353] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240368] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240368] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240357] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240406] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.240360] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240424] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240426] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240428] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240438] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a22 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240435] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240449] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240452] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240460] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240471] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240473] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.240484] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a23 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240490] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240487] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235746 (xid=0x543c0)
[2018-02-22 18:07:45.240482] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240507] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240513] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240490] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240517] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240493] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240533] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240541] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240548] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240558] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240561] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240567] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240580] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240589] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240592] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a24 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240602] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed" repeated 2 times between [2018-02-22 18:07:45.240473] and [2018-02-22 18:07:45.240601]
[2018-02-22 18:07:45.240616] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235760 (xid=0x543c1)
[2018-02-22 18:07:45.240611] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240611] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240622] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240633] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240656] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240661] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240668] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a25 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240676] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240677] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240689] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240702] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240710] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a26 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240719] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240719] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240732] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240746] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240757] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a27 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240761] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240764] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.235783 (xid=0x543c2)
[2018-02-22 18:07:45.240768] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240777] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240777] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240797] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240815] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.240798] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240828] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.240831] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a28 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240835] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240841] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240854] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240857] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.240870] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240880] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a29 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240874] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240888] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240899] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240913] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240910] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.236943 (xid=0x543c3)
[2018-02-22 18:07:45.240911] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240920] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240919] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240924] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.240950] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.240956] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.240965] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.240986] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.240978] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.240999] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241006] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543d9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241001] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241015] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241033] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241038] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(PORTMAP) op(PORTBYBRICK(1)) called at 2018-02-22 18:07:45.237021 (xid=0x543c4)
[2018-02-22 18:07:45.241034] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241048] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241055] W [MSGID: 114032] [client-handshake.c:1508:client_query_portmap_cbk] 2-scratch-client-1: received RPC status error, try again later [Transport endpoint is not connected]
[2018-02-22 18:07:45.241043] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241068] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543da Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241079] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241080] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241102] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241106] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241113] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241108] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241111] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241130] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543db Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241142] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241149] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.237611 (xid=0x543c5)
[2018-02-22 18:07:45.241149] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241150] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241166] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241173] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241172] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241174] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241187] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241189] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241221] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241245] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241259] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241263] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241273] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241282] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543dc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241282] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241291] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241288] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241295] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241293] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.237634 (xid=0x543c6)
[2018-02-22 18:07:45.241313] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241315] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241321] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241330] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a2f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241335] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543dd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241334] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241335] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241351] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241357] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241344] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241348] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241378] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241389] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a30 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241389] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241408] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241405] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241421] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241430] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a31 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241434] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241439] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241442] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241442] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241474] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543de Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241483] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.237740 (xid=0x543c7)
[2018-02-22 18:07:45.241477] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241499] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241485] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241500] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241516] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241528] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543df Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241537] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241539] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241552] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241555] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241568] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241579] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241587] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241595] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241598] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241611] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241612] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241618] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a32 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241628] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241621] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.237917 (xid=0x543c8)
[2018-02-22 18:07:45.241616] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241637] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241648] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241650] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241649] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241636] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241672] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241674] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241680] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790398: READ => -1 gfid=cb243c85-02fe-4e37-b03f-80f9187e72df fd=0x2b2f23f02210 (Transport endpoint is not connected)
[2018-02-22 18:07:45.241690] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a33 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241685] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241704] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241708] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241706] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241722] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241726] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241741] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed" repeated 2 times between [2018-02-22 18:07:45.241612] and [2018-02-22 18:07:45.241740]
[2018-02-22 18:07:45.241741] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241753] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.241754] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241767] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241786] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a34 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241784] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241791] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241784] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241795] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241810] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241820] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241824] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
The message "W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)" repeated 2 times between [2018-02-22 18:07:45.241726] and [2018-02-22 18:07:45.241840]
[2018-02-22 18:07:45.241846] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241855] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241848] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(READ(12)) called at 2018-02-22 18:07:45.238059 (xid=0x543c9)
[2018-02-22 18:07:45.241842] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241826] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241881] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241885] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a35 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241886] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241890] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241894] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241910] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.241908] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241927] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790399: READ => -1 gfid=cb243c85-02fe-4e37-b03f-80f9187e72df fd=0x2b2f23f02210 (Transport endpoint is not connected)
[2018-02-22 18:07:45.241907] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.241940] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.241947] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a36 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.241945] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.241962] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.241962] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.241980] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.241974] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.241991] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.241998] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242004] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242008] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a37 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242008] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242000] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242023] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242018] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242040] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242049] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242054] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242063] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242067] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242065] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242084] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238080 (xid=0x543ca)
[2018-02-22 18:07:45.242095] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242086] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242101] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242107] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242109] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242125] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242130] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242132] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242137] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242141] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a38 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242156] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543e9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242162] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242164] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242176] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242191] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242199] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a39 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242201] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242204] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242223] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238146 (xid=0x543cb)
[2018-02-22 18:07:45.242220] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242249] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242246] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242252] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242277] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242288] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ea Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242299] E [iobuf.c:940:iobref_unref] (-->/usr/lib64/glusterfs/3.8.4/xlator/protocol/client.so(+0x2c813) [0x2b2ef151d813] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/stripe.so(+0x1ea60) [0x2b2ef1768a60] -->/lib64/libglusterfs.so.0(iobref_unref+0xd4) [0x2b2ee589c634] ) 2-iobuf: invalid argument: iobref [Invalid argument]
[2018-02-22 18:07:45.242312] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242313] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242332] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242330] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 790424: READ => -1 gfid=cb243c85-02fe-4e37-b03f-80f9187e72df fd=0x2b2f23f02210 (Transport endpoint is not connected)
[2018-02-22 18:07:45.242341] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242354] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543eb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242356] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242366] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.242246] and [2018-02-22 18:07:45.242363]
[2018-02-22 18:07:45.242384] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242369] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242403] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242408] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238325 (xid=0x543cc)
[2018-02-22 18:07:45.242408] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242436] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242435] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242431] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242436] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242460] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242461] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242476] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242468] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242489] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242489] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242493] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ec Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242497] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242526] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242528] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242511] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242530] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242530] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242524] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242552] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242555] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:45.238346 (xid=0x543cd)
[2018-02-22 18:07:45.242567] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242578] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ed Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242571] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242579] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242594] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242607] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242605] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242638] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ee Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.242579] and [2018-02-22 18:07:45.242647]
[2018-02-22 18:07:45.242658] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242680] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ef Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed" repeated 2 times between [2018-02-22 18:07:45.242567] and [2018-02-22 18:07:45.242679]
[2018-02-22 18:07:45.242689] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242701] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242702] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242718] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop" repeated 3 times between [2018-02-22 18:07:45.242607] and [2018-02-22 18:07:45.242733]
[2018-02-22 18:07:45.242748] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242740] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242762] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242761] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242762] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242771] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t2_0001.fits (cb243c85-02fe-4e37-b03f-80f9187e72df) [Transport endpoint is not connected]
[2018-02-22 18:07:45.242782] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.242689] and [2018-02-22 18:07:45.242777]
[2018-02-22 18:07:45.242797] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242795] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242801] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b2t2_0001.fits (cb243c85-02fe-4e37-b03f-80f9187e72df) [Transport endpoint is not connected]
[2018-02-22 18:07:45.242795] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242817] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242831] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-scratch-dht: Found anomalies in /download/work/kstovall/57910/drx_57910_B0656+14_b2t2_0001.fits (gfid = cb243c85-02fe-4e37-b03f-80f9187e72df). Holes=1 overlaps=0
[2018-02-22 18:07:45.242836] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242845] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242845] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242853] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242857] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: cb243c85-02fe-4e37-b03f-80f9187e72df: failed to resolve (Transport endpoint is not connected)
[2018-02-22 18:07:45.242870] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a3f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242845] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242875] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.242879] E [fuse-bridge.c:796:fuse_getattr_resume] 0-glusterfs-fuse: 790426: GETATTR 47480857159432 (cb243c85-02fe-4e37-b03f-80f9187e72df) resolution failed
[2018-02-22 18:07:45.242879] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242906] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242913] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.242917] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242917] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242927] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.242934] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a40 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242937] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.242952] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.242969] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.242979] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.242979] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.242989] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a41 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.242998] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243012] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243018] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243035] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243047] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a42 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243050] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243050] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243060] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243068] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243070] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243084] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a43 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243087] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243077] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243082] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243089] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243106] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243128] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243132] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243140] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243141] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243153] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a44 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243136] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243159] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243168] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243171] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243177] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243188] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243203] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243203] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243215] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243207] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243226] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243236] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243247] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243256] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243258] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243265] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243257] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243287] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a45 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243289] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243294] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243296] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243321] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243318] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243327] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243346] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243355] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a46 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243356] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243346] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243364] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243388] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243383] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243388] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243405] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243419] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a47 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243428] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243442] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243447] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243444] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243466] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a48 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243475] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243487] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243503] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243507] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243520] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a49 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243528] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243535] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243540] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243537] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243559] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243569] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243582] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243574] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243588] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243604] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243598] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243616] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543f9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243621] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243624] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243631] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243646] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243649] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243648] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243656] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243679] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243688] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243688] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243698] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243700] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243707] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243717] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243729] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243733] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243735] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243739] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243759] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243764] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243766] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243773] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243785] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243786] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243791] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243797] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243799] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243806] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243816] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243832] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243838] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243833] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243852] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243862] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243872] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243881] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243889] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243889] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243897] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243916] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243918] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.243928] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.243934] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.243940] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.243952] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543fd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.243958] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a4f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.243962] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243967] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.243973] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.243985] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.243992] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244002] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x543ff Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244003] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244010] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244023] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244027] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244043] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244057] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54400 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244065] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244075] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244076] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244092] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54401 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244092] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244100] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244112] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244118] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244136] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244149] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54402 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244158] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244167] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244168] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244183] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54403 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244185] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244196] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244204] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244223] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244231] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244228] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244245] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244250] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54404 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244254] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a50 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244264] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244265] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244271] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244279] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244273] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244274] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244293] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54405 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244297] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244302] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244310] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a51 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244314] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244320] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244322] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244325] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244341] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54406 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244344] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244351] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244363] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244364] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244383] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244391] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54407 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244388] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244403] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244413] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244425] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244432] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54408 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244430] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244445] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244443] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244460] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244473] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a52 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244476] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244483] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244496] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244517] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244522] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244533] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a53 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244542] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244552] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244561] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244568] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a54 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244579] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244579] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244590] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244603] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244610] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a55 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244620] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244621] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244633] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244644] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244651] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a56 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244660] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244663] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244674] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244676] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244692] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244699] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54409 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244708] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244724] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244734] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244741] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244745] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244750] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244762] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244775] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244774] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244788] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244784] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244797] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244806] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244824] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244832] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244837] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244843] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244858] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244865] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244866] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244883] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244886] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.244899] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244903] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.244915] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.244918] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244933] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a57 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:45.244832] and [2018-02-22 18:07:45.244923]
[2018-02-22 18:07:45.244949] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.244951] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244941] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244967] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a58 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.244965] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.244977] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.244976] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.244991] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.244993] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245009] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a59 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245020] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245017] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245017] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.245042] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.245046] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245051] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5440f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.245059] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245068] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245078] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245077] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245087] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245097] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.245099] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245101] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245126] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245129] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245134] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245156] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245169] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245177] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245180] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245186] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245196] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245201] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245208] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245212] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.245216] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245225] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.245232] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54410 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.245230] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245248] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245254] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.245248] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245272] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.245281] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54411 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.245288] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.245289] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245298] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:45.245314] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54412 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:45.245313] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:45.245322] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245339] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.245367] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:45.245389] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245404] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245422] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245432] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245443] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245454] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245461] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a5f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245468] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245471] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245499] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245507] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245507] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245522] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a60 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245529] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245567] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.245592] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:45.245601] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:45.245608] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a61 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:45.245614] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:45.245654] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:45.246260] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0)
[2018-02-22 18:07:45.247605] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-1: changing port to 49154 (from 0)
[2018-02-22 18:07:45.254977] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:45.255615] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-0: Connected to scratch-client-0, attached to remote volume '/data/brick1/scratch'.
[2018-02-22 18:07:45.255630] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-22 18:07:45.255641] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-0: 4 fds open - Delaying child_up until they are re-opened
[2018-02-22 18:07:45.255830] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:45.256099] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-02-22 18:07:45.256197] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-0: Server lk version = 1
[2018-02-22 18:07:45.256226] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-1: Connected to scratch-client-1, attached to remote volume '/data/brick1/scratch'.
[2018-02-22 18:07:45.256238] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-22 18:07:45.256247] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-1: 4 fds open - Delaying child_up until they are re-opened
[2018-02-22 18:07:45.256661] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-02-22 18:07:45.256739] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-1: Server lk version = 1
[2018-02-22 18:07:46.186045] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:46.186183] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184758 (xid=0x54a6e)
[2018-02-22 18:07:46.186197] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:46.186286] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184793 (xid=0x54a6f)
[2018-02-22 18:07:46.186379] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184809 (xid=0x54a70)
[2018-02-22 18:07:46.186472] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184824 (xid=0x54a71)
[2018-02-22 18:07:46.186557] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184837 (xid=0x54a72)
[2018-02-22 18:07:46.186641] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184851 (xid=0x54a73)
[2018-02-22 18:07:46.186718] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184881 (xid=0x54a74)
[2018-02-22 18:07:46.186794] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184919 (xid=0x54a75)
[2018-02-22 18:07:46.186879] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184933 (xid=0x54a76)
[2018-02-22 18:07:46.186963] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184948 (xid=0x54a77)
[2018-02-22 18:07:46.187044] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184971 (xid=0x54a78)
[2018-02-22 18:07:46.187128] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184992 (xid=0x54a79)
[2018-02-22 18:07:46.187206] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185049 (xid=0x54a7a)
[2018-02-22 18:07:46.187282] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185090 (xid=0x54a7b)
[2018-02-22 18:07:46.187367] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185110 (xid=0x54a7c)
[2018-02-22 18:07:46.187454] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185124 (xid=0x54a7d)
[2018-02-22 18:07:46.187547] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185137 (xid=0x54a7e)
[2018-02-22 18:07:46.187635] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185164 (xid=0x54a7f)
[2018-02-22 18:07:46.187712] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185186 (xid=0x54a80)
[2018-02-22 18:07:46.187799] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185308 (xid=0x54a81)
[2018-02-22 18:07:46.187886] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185351 (xid=0x54a82)
[2018-02-22 18:07:46.187973] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185391 (xid=0x54a83)
[2018-02-22 18:07:46.188050] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185410 (xid=0x54a84)
[2018-02-22 18:07:46.188126] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185447 (xid=0x54a85)
[2018-02-22 18:07:46.188203] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185462 (xid=0x54a86)
[2018-02-22 18:07:46.188281] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185485 (xid=0x54a87)
[2018-02-22 18:07:46.188369] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185498 (xid=0x54a88)
[2018-02-22 18:07:46.188470] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185514 (xid=0x54a89)
[2018-02-22 18:07:46.188757] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184866 (xid=0x5441f)
[2018-02-22 18:07:46.188678] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:46.188773] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:46.188866] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184890 (xid=0x54420)
[2018-02-22 18:07:46.188960] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184903 (xid=0x54421)
[2018-02-22 18:07:46.189053] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184958 (xid=0x54422)
[2018-02-22 18:07:46.189149] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.184985 (xid=0x54423)
[2018-02-22 18:07:46.189243] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185005 (xid=0x54424)
[2018-02-22 18:07:46.189338] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185019 (xid=0x54425)
[2018-02-22 18:07:46.189446] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185032 (xid=0x54426)
[2018-02-22 18:07:46.189543] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185058 (xid=0x54427)
[2018-02-22 18:07:46.189638] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185072 (xid=0x54428)
[2018-02-22 18:07:46.189735] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185103 (xid=0x54429)
[2018-02-22 18:07:46.189834] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185150 (xid=0x5442a)
[2018-02-22 18:07:46.189930] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185173 (xid=0x5442b)
[2018-02-22 18:07:46.190028] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185198 (xid=0x5442c)
[2018-02-22 18:07:46.190126] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185211 (xid=0x5442d)
[2018-02-22 18:07:46.190224] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185224 (xid=0x5442e)
[2018-02-22 18:07:46.190323] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185236 (xid=0x5442f)
[2018-02-22 18:07:46.190427] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185249 (xid=0x54430)
[2018-02-22 18:07:46.190529] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185265 (xid=0x54431)
[2018-02-22 18:07:46.190629] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185281 (xid=0x54432)
[2018-02-22 18:07:46.190729] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185295 (xid=0x54433)
[2018-02-22 18:07:46.190830] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185322 (xid=0x54434)
[2018-02-22 18:07:46.190931] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185334 (xid=0x54435)
[2018-02-22 18:07:46.191033] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185366 (xid=0x54436)
[2018-02-22 18:07:46.191135] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185422 (xid=0x54437)
[2018-02-22 18:07:46.191237] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185437 (xid=0x54438)
[2018-02-22 18:07:46.191342] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185472 (xid=0x54439)
[2018-02-22 18:07:46.191457] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:46.185524 (xid=0x5443a)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 27 times between [2018-02-22 18:07:46.188773] and [2018-02-22 18:07:46.191472]
[2018-02-22 18:07:47.148084] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.148118] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.148126] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 28 times between [2018-02-22 18:07:46.186197] and [2018-02-22 18:07:47.148136]
[2018-02-22 18:07:47.148142] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148200] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.148218] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148217] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.148229] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148259] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:49154)
[2018-02-22 18:07:47.148272] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.148271] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:49154) failed
[2018-02-22 18:07:47.148281] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 790447: FLUSH() ERR => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:47.148279] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148310] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148314] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.148346] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.148355] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.148362] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148369] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148410] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148435] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.148436] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.148450] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.148457] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148476] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.148505] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.148513] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.148520] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.148527] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148538] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.148560] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.148568] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.148575] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148596] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148617] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148626] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.148666] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.148678] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148702] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148761] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.148765] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.148807] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a8f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148803] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.148816] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.148846] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.148849] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148858] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148872] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.148888] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.148920] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.148924] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.148922] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.148954] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5443f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.148961] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.148964] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.148973] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a90 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.148994] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149000] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149008] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.149029] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149045] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149058] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54440 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149077] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149091] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149126] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.149132] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149141] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.149158] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a91 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.149162] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149177] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54441 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149168] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149185] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149185] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.149228] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149275] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54442 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149287] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149325] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149343] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149345] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149379] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54443 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149390] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149420] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149422] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149450] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149479] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54444 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149504] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149572] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149606] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149607] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149627] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54445 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149635] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149668] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149694] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149713] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149723] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54446 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149753] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149797] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149811] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54447 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149810] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149820] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149857] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149891] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149891] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149904] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54448 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149911] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.149935] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.149952] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.149966] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.149973] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54449 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.149985] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150013] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150057] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150067] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150066] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150078] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150104] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150136] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150138] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150162] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150174] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150193] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150217] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150225] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150219] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150232] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150246] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150259] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150258] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150266] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150262] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150293] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150319] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150321] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150335] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150348] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150369] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150403] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.150410] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150423] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.150433] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a92 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.150430] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150447] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150454] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5444f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150467] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150470] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.150484] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150510] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150520] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150527] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54451 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150535] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150558] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150585] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150598] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150604] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54453 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150613] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150640] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150683] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150697] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150707] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54455 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150726] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150738] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150752] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150759] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54454 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150759] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150768] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150794] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.150820] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.150823] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150832] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a93 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.150836] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.150854] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54456 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.150846] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150868] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.150866] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150890] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.150916] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.150925] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.150932] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a94 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.150940] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.150940] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.150979] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.150980] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.150998] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.151012] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a95 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.151010] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54450 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151007] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151025] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151065] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151078] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.151109] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151120] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151131] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151146] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54452 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151159] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151176] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151196] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151200] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151207] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54457 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151222] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151236] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151251] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151261] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54458 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151262] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151273] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151306] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151311] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.151328] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a96 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.151327] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.151343] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151350] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151384] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.151384] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151396] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54459 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151406] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151414] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151423] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151429] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151434] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151436] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151465] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151471] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151482] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151491] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151504] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151523] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151539] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151539] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151566] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151576] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151598] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151609] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151617] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151635] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151653] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151668] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151686] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151688] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151697] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151710] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151731] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151745] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151753] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54461 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151753] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151779] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151804] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151818] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151819] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151844] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54462 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151857] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.151899] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.151930] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.151948] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.151958] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5445f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.151970] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152001] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152033] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152042] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152053] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54465 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152062] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152089] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152092] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152107] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152116] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54463 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152127] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152140] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152153] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152164] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54464 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152168] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152173] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152207] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152233] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152243] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152250] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54467 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152251] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0)
[2018-02-22 18:07:47.152258] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152270] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152296] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152304] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152310] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54469 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152318] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152330] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152332] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152358] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152375] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54466 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152388] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152422] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152443] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152460] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152469] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54460 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152481] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152508] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152550] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152572] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152582] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc3000, wc.byte_len = 0, post->reused = 2
[2018-02-22 18:07:47.152588] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54468 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152636] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152667] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152681] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152709] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152729] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152734] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc5000, wc.byte_len = 0, post->reused = 3
[2018-02-22 18:07:47.152751] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152791] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152796] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152817] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152829] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152847] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152868] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.152909] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.152922] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.152930] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.152942] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.152948] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc3000, wc.byte_len = 0, post->reused = 3
[2018-02-22 18:07:47.152970] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.153025] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.153060] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.153083] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5446d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.153104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.153142] W [MSGID: 103070] [rdma.c:4283:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x2b2f1dfc3000, wc.byte_len = 0, post->reused = 4
[2018-02-22 18:07:47.153142] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.153646] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.153648] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.153682] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54a9f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.153698] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.153722] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.153886] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.153901] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.153908] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.153918] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.153936] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.153978] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.153993] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154004] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154015] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154031] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154087] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154104] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154116] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154128] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154144] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154170] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154188] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154195] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154196] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152215 (xid=0x54a99)
[2018-02-22 18:07:47.154206] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154229] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154230] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154239] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154270] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154282] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154292] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154317] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154320] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154344] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154369] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154382] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154410] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154507] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154506] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152542 (xid=0x54a9a)
[2018-02-22 18:07:47.154518] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154532] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154535] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154542] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154561] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154580] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154596] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154606] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154617] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154632] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154646] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154650] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154653] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154662] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154696] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154701] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154701] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152568 (xid=0x54a9b)
[2018-02-22 18:07:47.154714] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154730] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aa9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154758] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154743] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154756] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154792] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154801] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aaa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154813] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154837] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154857] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154858] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154879] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aab Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154893] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154905] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.154917] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.154926] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aac Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.154929] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.154918] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152591 (xid=0x54a9c)
[2018-02-22 18:07:47.154940] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154951] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.154980] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155029] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155041] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155049] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aad Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155059] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155091] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155087] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.152742 (xid=0x54a9d)
[2018-02-22 18:07:47.155110] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155131] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155147] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155157] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aae Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155168] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155206] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155240] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155252] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155262] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aaf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155271] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155284] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_cleanup_and_start+0x1a)[0x2b2ee5b4c5da] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153129 (xid=0x54a9e)
[2018-02-22 18:07:47.155312] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155318] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155368] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155382] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155389] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155397] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155425] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155452] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155461] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155468] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155474] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155495] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155526] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155535] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155542] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155553] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155576] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155585] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155593] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:47.155600] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155610] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155621] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155641] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155661] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155666] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155677] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155686] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155707] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155719] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155721] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155727] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155736] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155738] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153148 (xid=0x5446c)
[2018-02-22 18:07:47.155760] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155763] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155766] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155798] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155813] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155827] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155848] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155865] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155866] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155893] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155903] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155913] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153170 (xid=0x5446f)
[2018-02-22 18:07:47.155918] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.155938] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.155950] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.155961] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.155967] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.155992] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156036] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156073] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156086] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156094] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ab9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156100] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2018-02-22 18:07:47.153187 (xid=0x54470)
[2018-02-22 18:07:47.156128] W [MSGID: 114032] [client-handshake.c:1625:client_dump_version_cbk] 2-scratch-client-1: received RPC status error [Transport endpoint is not connected]
[2018-02-22 18:07:47.156129] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.156140] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156158] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.156169] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.156178] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156195] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.156197] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156225] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156238] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aba Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156251] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156272] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156275] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153204 (xid=0x54471)
[2018-02-22 18:07:47.156287] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156309] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156298] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156319] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156303] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156358] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156403] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156414] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156413] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156421] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156454] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156485] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156485] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156501] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156509] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156504] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153220 (xid=0x54472)
[2018-02-22 18:07:47.156529] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156543] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156564] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156583] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156592] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156601] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156636] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156665] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156676] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156686] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54abf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156693] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156701] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153237 (xid=0x54473)
[2018-02-22 18:07:47.156723] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156728] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156766] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156774] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156780] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156787] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156790] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156816] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.156824] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.156832] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.156830] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153269 (xid=0x54474)
[2018-02-22 18:07:47.156854] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156855] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156876] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.156865] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.156904] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.156916] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.156923] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156927] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156937] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.156936] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.156947] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.156957] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.156982] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.156994] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.156991] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153302 (xid=0x54475)
[2018-02-22 18:07:47.157009] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157010] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157025] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157034] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157042] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157043] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157060] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157064] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157085] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157087] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157101] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157101] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153334 (xid=0x54476)
[2018-02-22 18:07:47.157106] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.157107] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157118] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157124] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.157132] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:47.157034] and [2018-02-22 18:07:47.157127]
[2018-02-22 18:07:47.157142] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157138] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157158] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.157171] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.157179] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.157180] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5448f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.157179] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157195] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157192] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157201] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157208] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157221] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.157225] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153366 (xid=0x54477)
[2018-02-22 18:07:47.157233] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157248] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157245] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157259] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157266] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157273] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157285] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157295] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157299] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157301] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157312] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157325] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157333] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157338] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157340] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157354] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157357] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153393 (xid=0x54478)
[2018-02-22 18:07:47.157367] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157382] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157384] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157389] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ac9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157402] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157459] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157497] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.157498] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.157503] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157514] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54490 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.157519] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157521] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157526] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aca Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157532] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157532] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.157547] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153416 (xid=0x54479)
[2018-02-22 18:07:47.157562] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157565] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:47.157459] and [2018-02-22 18:07:47.157565]
[2018-02-22 18:07:47.157575] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157589] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157600] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157612] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157613] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157628] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157630] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157638] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157656] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.157670] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.157668] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153441 (xid=0x5447a)
[2018-02-22 18:07:47.157676] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54491 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.157685] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157690] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157700] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.157712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.157718] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.157725] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54492 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.157732] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157742] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157751] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157757] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:47.157600] and [2018-02-22 18:07:47.157766]
[2018-02-22 18:07:47.157767] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.157788] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157803] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153758 (xid=0x5447b)
[2018-02-22 18:07:47.157809] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157823] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ace Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157822] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157823] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157835] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157881] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157906] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157907] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157919] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54acf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157925] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157940] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153782 (xid=0x5447c)
[2018-02-22 18:07:47.157955] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.157957] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.157973] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.157987] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.157993] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.157999] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158031] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158067] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158067] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153910 (xid=0x5447d)
[2018-02-22 18:07:47.158077] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158084] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158082] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158090] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158108] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158129] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.158137] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.158144] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54493 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.158150] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158186] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.158183] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153928 (xid=0x5447e)
[2018-02-22 18:07:47.158198] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158209] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158218] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158218] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158225] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158264] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158288] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158289] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158305] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158312] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158309] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153945 (xid=0x5447f)
[2018-02-22 18:07:47.158325] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158456] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153964 (xid=0x54480)
[2018-02-22 18:07:47.158474] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158475] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158492] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158498] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158505] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:47.158264] and [2018-02-22 18:07:47.158515]
[2018-02-22 18:07:47.158529] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.158544] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.158550] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54494 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.158556] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158581] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.153990 (xid=0x54481)
[2018-02-22 18:07:47.158600] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158600] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.158649] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158658] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158665] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158672] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158704] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158708] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154202 (xid=0x54482)
[2018-02-22 18:07:47.158722] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158735] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158744] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158743] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158750] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158794] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158817] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158817] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158829] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158837] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158839] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154251 (xid=0x54483)
[2018-02-22 18:07:47.158854] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158876] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158919] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.158933] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.158939] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.158945] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.158979] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154278 (xid=0x54484)
[2018-02-22 18:07:47.158986] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.158997] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.159017] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.159033] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.159039] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ad9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.159045] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.159087] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.159103] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.159110] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154297 (xid=0x54485)
[2018-02-22 18:07:47.159118] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.159130] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54495 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.159127] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.159141] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.159178] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.159284] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154326 (xid=0x54486)
[2018-02-22 18:07:47.159298] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.159432] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154349 (xid=0x54487)
[2018-02-22 18:07:47.159569] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154507 (xid=0x54488)
[2018-02-22 18:07:47.159703] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154546 (xid=0x54489)
[2018-02-22 18:07:47.159837] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.154685 (xid=0x5448a)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 4 times between [2018-02-22 18:07:47.159298] and [2018-02-22 18:07:47.159852]
[2018-02-22 18:07:47.159919] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.159942] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.159948] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ada Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.160013] W [fuse-bridge.c:2312:fuse_writev_cbk] 0-glusterfs-fuse: 790458: WRITE => -1 gfid=91f97edd-97a3-4948-9d9d-469c2d174853 fd=0x2b2f23f02440 (Transport endpoint is not connected)
[2018-02-22 18:07:47.159959] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.160040] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.161163] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161183] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161195] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54496 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161205] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161225] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161258] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.161260] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.161280] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54adb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.161290] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161318] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.161360] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161395] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161407] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54497 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161418] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161448] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161473] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161492] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161501] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54498 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161510] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161523] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161566] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161580] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x54499 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161579] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161589] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161626] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161659] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161660] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161678] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449a Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161687] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161712] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161734] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161752] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161761] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449b Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161770] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161785] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161828] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.161841] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449c Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.161840] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.161850] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161882] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.161914] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.161925] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.161934] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54adc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.161943] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.161970] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162001] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162012] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162021] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54add Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162030] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162052] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162082] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.162083] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.162101] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449d Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.162109] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162139] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.162171] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162184] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162194] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ade Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162202] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162231] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162255] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162274] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162283] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54adf Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162292] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162296] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162334] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.162346] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.162355] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.162363] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162401] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.162434] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162447] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162446] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162456] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162497] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162530] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162531] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162549] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162558] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162589] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162614] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.162633] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.162642] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x5449f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.162650] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162682] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.162714] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162725] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162734] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162742] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162764] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162805] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.162816] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.162825] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.162833] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162866] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.162901] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.162913] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.162912] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.162922] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.162948] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.162976] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.162987] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.162996] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a1 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163006] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163040] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163073] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.163074] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.163093] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.163104] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163119] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.163147] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.163159] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.163167] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a2 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163176] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163212] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163234] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.163253] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.163262] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a3 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163270] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163296] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163337] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.163348] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.163356] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.163365] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163415] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.163451] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.163462] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.163471] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.163480] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163518] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.163550] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.163562] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163562] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.163571] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163616] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163649] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.163660] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.163669] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.163677] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163692] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.163720] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.163721] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.163738] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163747] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163787] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163819] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.163831] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.163841] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.163855] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.163897] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.163931] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.163942] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.163953] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54ae8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.163962] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.164005] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.164029] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.164048] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.164057] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.164066] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.164080] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.164189] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:47.164228] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.164245] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.164254] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.164263] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.164280] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.164343] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.164355] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.164365] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544a9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.164382] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.164427] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.164462] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-1: rdma is not connected to peer (172.17.2.255:24008)
[2018-02-22 18:07:47.164474] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.255:24008) failed
[2018-02-22 18:07:47.164487] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-1: failed to submit rpc-request (XID: 0x544aa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-1)
[2018-02-22 18:07:47.164496] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.164540] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-1: failed to send the fop
[2018-02-22 18:07:47.165116] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165133] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165143] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af4 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165155] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165162] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:47.165193] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.165235] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165248] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af5 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165247] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165258] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165299] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164108 (xid=0x54aea)
[2018-02-22 18:07:47.165311] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.165316] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165346] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165348] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165367] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af6 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165384] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165469] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.165522] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164130 (xid=0x54aeb)
[2018-02-22 18:07:47.165539] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165572] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165589] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af7 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165588] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165599] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165621] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.165668] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164175 (xid=0x54aec)
[2018-02-22 18:07:47.165682] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165746] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165748] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165770] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af8 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165802] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164210 (xid=0x54aed)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:47.165682] and [2018-02-22 18:07:47.165815]
[2018-02-22 18:07:47.165822] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.165876] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.165889] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54af9 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.165888] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.165899] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.165926] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS Handshake) op(SETVOLUME(1)) called at 2018-02-22 18:07:47.164238 (xid=0x54aee)
[2018-02-22 18:07:47.165941] W [MSGID: 114032] [client-handshake.c:1088:client_setvolume_cbk] 2-scratch-client-0: received RPC status error [Transport endpoint is not connected]
[2018-02-22 18:07:47.165953] I [MSGID: 114051] [client-handshake.c:1259:client_setvolume_cbk] 2-scratch-client-0: sending CHILD_CONNECTING event
[2018-02-22 18:07:47.165963] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.166001] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.166002] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.166020] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afa Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.166029] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166037] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164320 (xid=0x54aef)
[2018-02-22 18:07:47.166082] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.166163] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.166185] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164582 (xid=0x54af0)
[2018-02-22 18:07:47.166187] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.166212] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afb Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2018-02-22 18:07:47.166029] and [2018-02-22 18:07:47.166224]
[2018-02-22 18:07:47.166282] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164646 (xid=0x54af1)
[2018-02-22 18:07:47.166269] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.166300] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166404] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.166422] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afc Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.166421] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.166433] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166456] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.166508] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164697 (xid=0x54af2)
[2018-02-22 18:07:47.166518] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:47.166531] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:47.166525] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166552] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afd Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.166565] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166653] W [fuse-bridge.c:2312:fuse_writev_cbk] 0-glusterfs-fuse: 790460: WRITE => -1 gfid=91f97edd-97a3-4948-9d9d-469c2d174853 fd=0x2b2f23f02440 (Transport endpoint is not connected)
[2018-02-22 18:07:47.166688] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164725 (xid=0x54af3)
[2018-02-22 18:07:47.166704] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.166947] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:47.167042] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b3t1_0001.fits (dace8bc0-f4d7-48b2-98ab-a0f768b3be8c) [Transport endpoint is not connected]
[2018-02-22 18:07:47.167069] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164600 (xid=0x544ac)
[2018-02-22 18:07:47.167085] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167085] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.167099] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/drx_57910_B0656+14_b3t1_0001.fits (dace8bc0-f4d7-48b2-98ab-a0f768b3be8c) [Transport endpoint is not connected]
[2018-02-22 18:07:47.167128] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54afe Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.167126] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.167145] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167152] I [MSGID: 109063] [dht-layout.c:713:dht_layout_normalize] 2-scratch-dht: Found anomalies in /download/work/kstovall/57910/drx_57910_B0656+14_b3t1_0001.fits (gfid = dace8bc0-f4d7-48b2-98ab-a0f768b3be8c). Holes=1 overlaps=0
[2018-02-22 18:07:47.167191] W [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse: dace8bc0-f4d7-48b2-98ab-a0f768b3be8c: failed to resolve (Transport endpoint is not connected)
[2018-02-22 18:07:47.167205] E [fuse-bridge.c:796:fuse_getattr_resume] 0-glusterfs-fuse: 790462: GETATTR 47480857159620 (dace8bc0-f4d7-48b2-98ab-a0f768b3be8c) resolution failed
[2018-02-22 18:07:47.167242] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164622 (xid=0x544ad)
The message "W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop" repeated 2 times between [2018-02-22 18:07:47.166456] and [2018-02-22 18:07:47.167174]
[2018-02-22 18:07:47.167257] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167304] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.167319] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.167326] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54aff Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.167334] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167356] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.167463] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167478] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164669 (xid=0x544ae)
[2018-02-22 18:07:47.167501] W [MSGID: 114031] [client-rpc-fops.c:1495:client3_3_fstat_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167502] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167554] W [fuse-bridge.c:767:fuse_attr_cbk] 0-glusterfs-fuse: 790467: FSTAT() /download/work/kstovall/57910/drx_57910_B0656+14_b3t2_0001.fits => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:47.167560] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.167575] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.167582] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b00 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.167590] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167609] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.167656] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(PORTMAP) op(PORTBYBRICK(1)) called at 2018-02-22 18:07:47.164738 (xid=0x544af)
[2018-02-22 18:07:47.167669] W [MSGID: 114032] [client-handshake.c:1508:client_query_portmap_cbk] 2-scratch-client-1: received RPC status error, try again later [Transport endpoint is not connected]
[2018-02-22 18:07:47.167761] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.164755 (xid=0x544b0)
[2018-02-22 18:07:47.167774] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167861] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:24008)
[2018-02-22 18:07:47.167873] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:24008) failed
[2018-02-22 18:07:47.167880] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b01 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:47.167890] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.167909] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:47.167910] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165522 (xid=0x544b1)
[2018-02-22 18:07:47.167924] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.168078] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165551 (xid=0x544b2)
[2018-02-22 18:07:47.168222] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165676 (xid=0x544b3)
[2018-02-22 18:07:47.168423] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.165710 (xid=0x544b4)
[2018-02-22 18:07:47.168568] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166138 (xid=0x544b5)
[2018-02-22 18:07:47.168712] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166343 (xid=0x544b6)
[2018-02-22 18:07:47.168856] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166378 (xid=0x544b7)
[2018-02-22 18:07:47.169003] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:47.166492 (xid=0x544b8)
[2018-02-22 18:07:47.172418] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 7 times between [2018-02-22 18:07:47.167924] and [2018-02-22 18:07:47.169018]
[2018-02-22 18:07:47.176708] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:47.177249] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-0: Connected to scratch-client-0, attached to remote volume '/data/brick1/scratch'.
[2018-02-22 18:07:47.177261] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-22 18:07:47.177270] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-0: 4 fds open - Delaying child_up until they are re-opened
[2018-02-22 18:07:47.177839] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-02-22 18:07:47.177894] W [MSGID: 114031] [client-rpc-fops.c:799:client3_3_statfs_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:47.177935] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-0: Server lk version = 1
[2018-02-22 18:07:48.123815] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-02-22 18:07:48.548708] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-1: changing port to 49154 (from 0)
[2018-02-22 18:07:48.124421] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-02-22 18:07:48.553517] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:48.553937] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-1: Connected to scratch-client-1, attached to remote volume '/data/brick1/scratch'.
[2018-02-22 18:07:48.553946] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-22 18:07:48.553953] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-1: 4 fds open - Delaying child_up until they are re-opened
[2018-02-22 18:07:48.554300] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-02-22 18:07:48.554458] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-1: Server lk version = 1
[2018-02-22 18:07:49.139511] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.139545] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.139561] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b2e Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.139578] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.139584] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-0: disconnected from scratch-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:49.139616] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.139744] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2018-02-22 18:07:49.137922 (xid=0x54b13)
[2018-02-22 18:07:49.139762] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.139816] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.139836] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.139847] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b2f Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.139859] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138017 (xid=0x54b14)
[2018-02-22 18:07:49.139861] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.139883] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.139896] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.139960] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.139976] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.139981] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138034 (xid=0x54b15)
[2018-02-22 18:07:49.139986] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b30 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.140001] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140009] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140016] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.140093] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.140094] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138210 (xid=0x54b16)
[2018-02-22 18:07:49.140112] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.140119] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140124] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b31 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.140146] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140164] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.140219] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.140219] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138307 (xid=0x54b17)
[2018-02-22 18:07:49.140236] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.140243] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140248] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b32 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.140270] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140277] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.140343] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138332 (xid=0x54b18)
[2018-02-22 18:07:49.140361] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.140361] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140405] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.140418] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b33 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.140429] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140458] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.140491] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138404 (xid=0x54b19)
[2018-02-22 18:07:49.140505] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140609] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138432 (xid=0x54b1a)
[2018-02-22 18:07:49.140622] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140643] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.140664] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.140671] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b34 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.140678] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140690] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.140711] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138454 (xid=0x54b1b)
[2018-02-22 18:07:49.140726] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.140825] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138475 (xid=0x54b1c)
[2018-02-22 18:07:49.140924] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138506 (xid=0x54b1d)
[2018-02-22 18:07:49.141023] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138533 (xid=0x54b1e)
[2018-02-22 18:07:49.141122] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138560 (xid=0x54b1f)
[2018-02-22 18:07:49.141211] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138607 (xid=0x54b20)
[2018-02-22 18:07:49.141311] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138631 (xid=0x54b21)
[2018-02-22 18:07:49.141443] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138654 (xid=0x54b22)
[2018-02-22 18:07:49.141546] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138679 (xid=0x54b23)
[2018-02-22 18:07:49.141646] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138725 (xid=0x54b24)
[2018-02-22 18:07:49.141735] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138747 (xid=0x54b25)
[2018-02-22 18:07:49.141835] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138773 (xid=0x54b26)
[2018-02-22 18:07:49.141934] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138845 (xid=0x54b27)
[2018-02-22 18:07:49.142023] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138891 (xid=0x54b28)
[2018-02-22 18:07:49.142112] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138961 (xid=0x54b29)
[2018-02-22 18:07:49.142211] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138986 (xid=0x54b2a)
[2018-02-22 18:07:49.142312] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139037 (xid=0x54b2b)
[2018-02-22 18:07:49.142433] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139086 (xid=0x54b2c)
[2018-02-22 18:07:49.142523] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-0: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139164 (xid=0x54b2d)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]" repeated 18 times between [2018-02-22 18:07:49.140726] and [2018-02-22 18:07:49.142536]
[2018-02-22 18:07:49.142797] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 2-scratch-client-1: disconnected from scratch-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-02-22 18:07:49.142922] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2018-02-22 18:07:49.137938 (xid=0x544c6)
[2018-02-22 18:07:49.142939] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.142965] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.142984] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143128] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138060 (xid=0x544c7)
[2018-02-22 18:07:49.143144] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.143145] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143175] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143201] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143219] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143244] W [fuse-bridge.c:471:fuse_entry_cbk] 0-glusterfs-fuse: 790481: LOOKUP() /download => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:49.143247] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138083 (xid=0x544c8)
[2018-02-22 18:07:49.143261] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.143359] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138105 (xid=0x544c9)
[2018-02-22 18:07:49.143482] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138127 (xid=0x544ca)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 2 times between [2018-02-22 18:07:49.143261] and [2018-02-22 18:07:49.143503]
[2018-02-22 18:07:49.143515] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143574] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143605] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143639] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (7b022439-c4c4-4267-b11b-17b4661620fa) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143638] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138147 (xid=0x544cb)
[2018-02-22 18:07:49.143670] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.143768] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143801] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143820] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138169 (xid=0x544cc)
[2018-02-22 18:07:49.143840] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-0: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143841] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.143875] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2018-02-22 18:07:49.143912] W [fuse-bridge.c:471:fuse_entry_cbk] 0-glusterfs-fuse: 790485: LOOKUP() /download => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:49.143999] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138194 (xid=0x544cd)
[2018-02-22 18:07:49.144019] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.144169] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138231 (xid=0x544ce)
[2018-02-22 18:07:49.144295] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138253 (xid=0x544cf)
[2018-02-22 18:07:49.144404] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138273 (xid=0x544d0)
[2018-02-22 18:07:49.144541] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138294 (xid=0x544d1)
[2018-02-22 18:07:49.144695] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138348 (xid=0x544d2)
[2018-02-22 18:07:49.144847] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138582 (xid=0x544d3)
[2018-02-22 18:07:49.145003] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138707 (xid=0x544d4)
[2018-02-22 18:07:49.145155] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138796 (xid=0x544d5)
[2018-02-22 18:07:49.145311] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138819 (xid=0x544d6)
[2018-02-22 18:07:49.145422] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138863 (xid=0x544d7)
[2018-02-22 18:07:49.145567] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138910 (xid=0x544d8)
[2018-02-22 18:07:49.145721] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.138936 (xid=0x544d9)
[2018-02-22 18:07:49.145871] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139010 (xid=0x544da)
[2018-02-22 18:07:49.146021] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139057 (xid=0x544db)
[2018-02-22 18:07:49.146173] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139105 (xid=0x544dc)
[2018-02-22 18:07:49.146326] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139130 (xid=0x544dd)
[2018-02-22 18:07:49.146488] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.139190 (xid=0x544de)
[2018-02-22 18:07:49.146629] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140064 (xid=0x544df)
[2018-02-22 18:07:49.146783] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140327 (xid=0x544e0)
[2018-02-22 18:07:49.146935] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140501 (xid=0x544e1)
[2018-02-22 18:07:49.147086] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140522 (xid=0x544e2)
[2018-02-22 18:07:49.147236] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140545 (xid=0x544e3)
[2018-02-22 18:07:49.147393] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140564 (xid=0x544e4)
[2018-02-22 18:07:49.147536] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140582 (xid=0x544e5)
[2018-02-22 18:07:49.147641] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140601 (xid=0x544e6)
[2018-02-22 18:07:49.147790] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x2b2ee5874752] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x2b2ee5b4a8ce] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x2b2ee5b4a9de] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x2b2ee5b4c150] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x2b2ee5b4cc10] ))))) 2-scratch-client-1: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2018-02-22 18:07:49.140623 (xid=0x544e7)
[2018-02-22 18:07:49.147846] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 790480: FLUSH() ERR => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:49.152660] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 2-scratch-client-0: changing port to 49154 (from 0)
The message "W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]" repeated 26 times between [2018-02-22 18:07:49.144019] and [2018-02-22 18:07:49.147810]
[2018-02-22 18:07:49.156308] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 2-scratch-client-0: rdma is not connected to peer (172.17.2.254:49154)
[2018-02-22 18:07:49.156348] W [MSGID: 103037] [rdma.c:3016:gf_rdma_submit_request] 2-rpc-transport/rdma: sending request to peer (172.17.2.254:49154) failed
[2018-02-22 18:07:49.156358] W [rpc-clnt.c:1694:rpc_clnt_submit] 2-scratch-client-0: failed to submit rpc-request (XID: 0x54b37 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (scratch-client-0)
[2018-02-22 18:07:49.156377] W [MSGID: 114031] [client-rpc-fops.c:855:client3_3_writev_cbk] 2-scratch-client-0: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.156418] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 790496: FLUSH() ERR => -1 (Transport endpoint is not connected)
[2018-02-22 18:07:49.156449] W [MSGID: 114029] [client-rpc-fops.c:4384:client3_3_writev] 2-scratch-client-0: failed to send the fop
[2018-02-22 18:07:49.157895] I [MSGID: 114057] [client-handshake.c:1439:select_server_supported_programs] 2-scratch-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-02-22 18:07:49.158352] I [MSGID: 114046] [client-handshake.c:1215:client_setvolume_cbk] 2-scratch-client-0: Connected to scratch-client-0, attached to remote volume '/data/brick1/scratch'.
[2018-02-22 18:07:49.158366] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 2-scratch-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2018-02-22 18:07:49.158708] I [MSGID: 114042] [client-handshake.c:1046:client_post_handshake] 2-scratch-client-0: 2 fds open - Delaying child_up until they are re-opened
[2018-02-22 18:07:49.159196] I [MSGID: 114041] [client-handshake.c:677:client_child_up_reopen_done] 2-scratch-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-02-22 18:07:49.159242] W [MSGID: 114031] [client-rpc-fops.c:799:client3_3_statfs_cbk] 2-scratch-client-1: remote operation failed [Transport endpoint is not connected]
[2018-02-22 18:07:49.159319] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 2-scratch-client-0: Server lk version = 1
[2018-02-22 18:07:49.181472] W [MSGID: 114031] [client-rpc-fops.c:2938:client3_3_lookup_cbk] 2-scratch-client-1: remote operation failed. Path: /download/work/kstovall/57910/jp.pbs (e9f57eb0-e8a2-40dc-b56f-59eea34941a6) [Transport endpoint is not connected]
-------------- next part --------------
[2018-02-22 18:07:45.226382] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7fd5ba679000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:45.236646] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-0
[2018-02-22 18:07:45.236796] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:45.236854] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits
[2018-02-22 18:07:45.236910] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:45.236966] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log
[2018-02-22 18:07:45.237007] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-0
[2018-02-22 18:07:45.262237] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-1 (version: 3.8.4)
[2018-02-22 18:07:45.416142] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-1-22-0
[2018-02-22 18:07:45.416218] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OSZICAR
[2018-02-22 18:07:45.416254] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OUTCAR
[2018-02-22 18:07:45.416306] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-1-22-0
[2018-02-22 18:07:46.191329] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7fd5ba799000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.192099] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba8e3000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.192498] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba401000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.192876] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba7c5000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.195455] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-1
[2018-02-22 18:07:46.195537] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:46.195589] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:46.195634] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits
[2018-02-22 18:07:46.195675] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log
[2018-02-22 18:07:46.195742] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-1
[2018-02-22 18:07:48.559889] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-2 (version: 3.8.4)
[2018-02-22 18:07:49.144466] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7fd5bbf7b000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:49.145265] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7fd5ba53b000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:49.149583] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-2
[2018-02-22 18:07:49.149670] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:49.149726] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:49.149825] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-1-2-2
-------------- next part --------------
[2018-02-22 18:07:45.226574] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32cabd1000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:45.228378] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-0
[2018-02-22 18:07:45.228601] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits
[2018-02-22 18:07:45.228681] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:45.228737] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:45.228793] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log
[2018-02-22 18:07:45.230202] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1021)
[2018-02-22 18:07:45.230273] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1021) failed
[2018-02-22 18:07:45.230292] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54974, Program: GlusterFS 3.3, ProgVers: 330, Proc: 12) to rpc-transport (rdma.scratch-server)
[2018-02-22 18:07:45.230386] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x10e82) [0x7f32dc180e82] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x1cac6) [0x7f32d7dd4ac6] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed
[2018-02-22 18:07:45.230463] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-0
[2018-02-22 18:07:45.261717] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-1 (version: 3.8.4)
[2018-02-22 18:07:45.416150] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-0-22-0
[2018-02-22 18:07:45.416244] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OSZICAR
[2018-02-22 18:07:45.416315] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /slzhou/slzhou/CeO2-331/NEB1/neb-new/PBE-d2/09/SCF/OUTCAR
[2018-02-22 18:07:45.416373] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler276-32676-2018/02/22-16:07:03:805703-scratch-client-0-22-0
[2018-02-22 18:07:46.191486] W [MSGID: 103004] [rdma.c:2327:__gf_rdma_register_local_mr_for_rdma] 0-rpc-transport/rdma: registering memory for IBV_ACCESS_LOCAL_WRITE failed [Invalid argument]
[2018-02-22 18:07:46.191549] W [MSGID: 103004] [rdma.c:3701:gf_rdma_do_reads] 0-rpc-transport/rdma: registering local memory for rdma read failed
[2018-02-22 18:07:46.191570] W [MSGID: 103061] [rdma.c:3969:gf_rdma_recv_request] 0-rpc-transport/rdma: rdma read from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.191600] W [MSGID: 103063] [rdma.c:4084:gf_rdma_process_recv] 0-rpc-transport/rdma: receiving a request from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.191611] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32caced000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.192513] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca41d000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.192880] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca9c7000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.193399] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca7d3000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:46.193948] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-1
[2018-02-22 18:07:46.194011] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023)
[2018-02-22 18:07:46.194085] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:46.194086] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.194118] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:46.194131] W [MSGID: 103026] [rdma.c:3670:gf_rdma_do_reads] 0-rpc-transport/rdma: transport not connected to peer (172.17.1.61:1023), not doing rdma reads
[2018-02-22 18:07:46.194191] W [MSGID: 103061] [rdma.c:3969:gf_rdma_recv_request] 0-rpc-transport/rdma: rdma read from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.194191] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine2.log
[2018-02-22 18:07:46.194214] W [MSGID: 103063] [rdma.c:4084:gf_rdma_process_recv] 0-rpc-transport/rdma: receiving a request from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.194252] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023)
[2018-02-22 18:07:46.194264] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed
[2018-02-22 18:07:46.194342] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54a72, Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (rdma.scratch-server)
[2018-02-22 18:07:46.194288] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54a6f, Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (rdma.scratch-server)
[2018-02-22 18:07:46.194404] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x112bb) [0x7f32dc1812bb] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x1c651) [0x7f32d7dd4651] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed
[2018-02-22 18:07:46.194452] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x112bb) [0x7f32dc1812bb] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x1c651) [0x7f32d7dd4651] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed
[2018-02-22 18:07:46.194562] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-1
[2018-02-22 18:07:47.170678] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32cabaf000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.170781] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-2 (version: 3.8.4)
[2018-02-22 18:07:47.171160] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 5, wc.vendor_err = 249, post->buf = 0x7f32ca9eb000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:47.171941] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-2
[2018-02-22 18:07:47.171968] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023)
[2018-02-22 18:07:47.172145] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed
[2018-02-22 18:07:47.172168] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54aee, Program: GlusterFS Handshake, ProgVers: 2, Proc: 1) to rpc-transport (rdma.scratch-server)
[2018-02-22 18:07:47.172164] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-glusterfs: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-2
[2018-02-22 18:07:47.172214] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cf25) [0x7f32d7de4f25] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cec3) [0x7f32d7de4ec3] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed
[2018-02-22 18:07:47.172248] E [server.c:207:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cf25) [0x7f32d7de4f25] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2cec3) [0x7f32d7de4ec3] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x882e) [0x7f32d7dc082e] ) 0-: Reply submission failed
[2018-02-22 18:07:47.183288] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-3 (version: 3.8.4)
[2018-02-22 18:07:49.144790] W [MSGID: 103070] [rdma.c:4282:gf_rdma_handle_failed_send_completion] 0-rpc-transport/rdma: send work request on `mlx4_0' returned error wc.status = 1, wc.vendor_err = 105, post->buf = 0x7f32ca3b7000, wc.byte_len = 131072, post->reused = 1
[2018-02-22 18:07:49.147498] I [MSGID: 115036] [server.c:548:server_rpc_notify] 0-scratch-server: disconnecting connection from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-3
[2018-02-22 18:07:49.147530] W [MSGID: 103026] [rdma.c:3670:gf_rdma_do_reads] 0-rpc-transport/rdma: transport not connected to peer (172.17.1.61:1023), not doing rdma reads
[2018-02-22 18:07:49.147592] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b3_0001.fits
[2018-02-22 18:07:49.147596] W [MSGID: 103061] [rdma.c:3969:gf_rdma_recv_request] 0-rpc-transport/rdma: rdma read from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:49.147646] W [MSGID: 103063] [rdma.c:4084:gf_rdma_process_recv] 0-rpc-transport/rdma: receiving a request from peer (172.17.1.61:1023) failed
[2018-02-22 18:07:49.147659] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/combine1.log
[2018-02-22 18:07:49.147655] W [MSGID: 103026] [rdma.c:2843:gf_rdma_writev] 0-rdma.scratch-server: rdma is not connected to peer (172.17.1.61:1023)
[2018-02-22 18:07:49.147797] W [MSGID: 103037] [rdma.c:3053:gf_rdma_submit_reply] 0-rpc-transport/rdma: sending request to peer (172.17.1.61:1023) failed
[2018-02-22 18:07:49.147824] E [rpcsvc.c:1354:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x54b13, Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (rdma.scratch-server)
[2018-02-22 18:07:49.148988] E [server.c:200:server_submit_reply] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/debug/io-stats.so(+0x12857) [0x7f32dc182857] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x2b427) [0x7f32d7de3427] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.15/xlator/protocol/server.so(+0x87d5) [0x7f32d7dc07d5] ) 0-: Reply submission failed
[2018-02-22 18:07:49.149072] I [MSGID: 101055] [client_t.c:415:gf_client_unref] 0-scratch-server: Shutting down connection wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-3
[2018-02-22 18:07:49.164480] I [MSGID: 115029] [server-handshake.c:692:server_setvolume] 0-scratch-server: accepted client from wheeler061-17519-2018/02/22-18:04:06:449987-scratch-client-0-2-4 (version: 3.8.4)
[2018-02-22 18:07:46.194027] I [MSGID: 115013] [server-helpers.c:293:do_fd_cleanup] 0-scratch-server: fd cleanup on /download/work/kstovall/57910/drx_57910_B0656+14_b2_0001.fits
More information about the Gluster-users
mailing list