[Gluster-users] GlusterFS 3.1.2 and switch scheduler problem

[Gvozden Neskovic]

Hi All,

I'm trying to configure GlusterFS 3.1.2 with cluster/switch xlator. My goal
is to
store data on local disks, and minimize network traffic. I was planing to
have a
directory for every node in GlusterFS root name space, and use switch
scheduler.

I am getting random errors on file creating. Underlying file system is XFS.
Here is the volume file and log excerpt.

Any suggestions?

Thank you.

volume posix
 type storage/posix
 option directory /local/gluster_volumes/test-volume
end-volume

volume locks
  type features/locks
  subvolumes posix
end-volume

volume brick
 type performance/io-threads
 subvolumes locks
end-volume

volume server
 type protocol/server
 option transport-type rdma
 option auth.addr.brick.allow *
 subvolumes brick
end-volume

volume node1
  type protocol/client
  option transport-type rdma
  option remote-host node1
  option remote-subvolume brick
end-volume

volume node2
  type protocol/client
  option transport-type rdma
  option remote-host node2
  option remote-subvolume brick
end-volume

volume cluster-switch
 type cluster/switch
 option lookup-unhashed on
 option pattern.switch.case /node1*:node1;*:node2
subvolumes node1 node2
end-volume



[2011-03-02 15:26:35.594999] T [server-resolve.c:127:resolve_loc_touchup]
server: return value inode_path 1
[2011-03-02 15:26:35.595033] D [dht-linkfile.c:138:dht_linkfile_create] :
dict is NULL, need to make sure gfid's are same
[2011-03-02 15:26:35.595143] T
[auth-glusterfs.c:185:auth_glusterfs_authenticate] rpc-service: Auth Info:
pid: 24584, uid: 572, gid: 513, owner: 24584
[2011-03-02 15:26:35.595167] T [server-resolve.c:127:resolve_loc_touchup]
server: return value inode_path 10
[2011-03-02 15:26:35.595276] D [dht-diskusage.c:69:dht_du_info_cbk]
cluster-switch: on subvolume 'node9-009': avail_percent is: 99.00 and
avail_space is: 1768715341824
[2011-03-02 15:26:35.595313] D [dht-diskusage.c:69:dht_du_info_cbk]
cluster-switch: on subvolume 'node9-008': avail_percent is: 99.00 and
avail_space is: 1766295830528
[2011-03-02 15:26:35.595362] D [posix.c:265:posix_lstat_with_gfid] posix:
failed to get gfid
[2011-03-02 15:26:35.595381] E [posix.c:1127:posix_mknod] posix: mknod on
/file.test failed: No data available
[2011-03-02 15:26:35.595394] T [server3_1-fops.c:511:server_mknod_cbk]
server: 61: MKNOD /file.test ==> -1 (No data available)
[2011-03-02 15:26:35.595471] W [fuse-bridge.c:1514:fuse_create_cbk]
glusterfs-fuse: 80: /file.test => -1 (No data available)
[2011-03-02 15:26:35.595532] T [fuse-bridge.c:279:fuse_lookup]
glusterfs-fuse: 81: LOOKUP /file.test
[2011-03-02 15:26:35.595989] T
[auth-glusterfs.c:185:auth_glusterfs_authenticate] rpc-service: Auth Info:
pid: 24584, uid: 572, gid: 513, owner: 24584
[2011-03-02 15:26:35.595998] T [rpcsvc.c:723:rpcsvc_program_actor]
rpc-service: Actor found: GlusterFS-3.1.0 - LOOKUP
[2011-03-02 15:26:35.596016] T [server-resolve.c:127:resolve_loc_touchup]
server: return value inode_path 10
[2011-03-02 15:26:35.596094] T [rpc-clnt.c:631:rpc_clnt_reply_init]
rpc-clnt: recieved rpc message (RPC XID: 0x109x Program: GlusterFS 3.1,
ProgVers: 310, Proc: 27) from rpc-transport (node9-009)
[2011-03-02 15:26:35.596122] T [rpcsvc.c:1516:rpcsvc_submit_generic]
rpc-service: Tx message: 216
[2011-03-02 15:26:35.596152] T [rpcsvc.c:1151:rpcsvc_record_build_header]
rpc-service: Reply fraglen 240, payload: 216, rpc hdr: 24
[2011-03-02 15:26:35.596167] T [rpcsvc.c:1555:rpcsvc_submit_generic]
rpc-service: submitted reply for rpc-message (XID: 0x62x, Program:
GlusterFS-3.1.0, ProgVers: 310, Proc: 27) to rpc-transport (rdma.server)
[2011-03-02 15:26:35.596181] T [rpc-clnt.c:631:rpc_clnt_reply_init]
rpc-clnt: recieved rpc message (RPC XID: 0x62x Program: GlusterFS 3.1,
ProgVers: 310, Proc: 27) from rpc-transport (node9-008)
[2011-03-02 15:26:35.596205] D [dht-common.c:566:dht_lookup_everywhere_cbk]
cluster-switch: found on node9-008 linkfile /file.test (-> '')
[2011-03-02 15:26:35.596216] D [dht-common.c:605:dht_lookup_everywhere_cbk]
cluster-switch: deleting stale linkfile /file.test on node9-008
Gvozden Neskovic
neskovic at gmail.com