[Gluster-devel] mainline--2.5--patch-267: cp -ax runaway

[Krishna Srinivas]

Hi Gerry,

We were trying to see how glusterfs behaves when "cp -ax"
is started and while cp is happening, kill all the servers.
The log messages which were produced were same as the ones you
have given, So are you sure that glusterfsd processes were not dead
when cp was happening? ( Connection refused would indicate
that glusterfsd is not listening at that port)

We are checking how the log messages can be avoided
when this particular situation is encountered.

Thanks
Krishna

On 7/4/07, Gerry Reno <greno at verizon.net> wrote:
>
> Actions:
>
> [root at grp-01-20-01 mnt]# cd glusterfs2
> [root at grp-01-20-01 glusterfs2]# ls
> dir1  file2  file3  file333  local
> [root at grp-01-20-01 glusterfs2]# rm -rf local
> [root at grp-01-20-01 glusterfs2]#
> [root at grp-01-20-01 glusterfs2]#
> [root at grp-01-20-01 glusterfs2]#
> [root at grp-01-20-01 glusterfs2]# find /mnt/glusterfs*
> /mnt/glusterfs0
> /mnt/glusterfs0/dir1
> /mnt/glusterfs0/dir1/file0
> /mnt/glusterfs0/file3
> /mnt/glusterfs0/file333
> /mnt/glusterfs0/file2
> /mnt/glusterfs1
> /mnt/glusterfs1/dir1
> /mnt/glusterfs1/dir1/file0
> /mnt/glusterfs1/file3
> /mnt/glusterfs1/file333
> /mnt/glusterfs1/file2
> /mnt/glusterfs2
> /mnt/glusterfs2/dir1
> /mnt/glusterfs2/dir1/file0
> /mnt/glusterfs2/file3
> /mnt/glusterfs2/file333
> /mnt/glusterfs2/file2
> /mnt/glusterfs3
> /mnt/glusterfs3/dir1
> /mnt/glusterfs3/dir1/file0
> /mnt/glusterfs3/file3
> /mnt/glusterfs3/file333
> /mnt/glusterfs3/file2
> [root at grp-01-20-01 glusterfs2]# cd ..
>
> [root at grp-01-20-01 mnt]# cp -ax /usr /mnt/glusterfs0
> cp: cannot create directory `/mnt/glusterfs0/usr': Device or resource busy
> [root at grp-01-20-01 mnt]#
>
> In the log I see this:
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
> 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk]
> client3: client lookup failed
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
> 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk]
> client3: client lookup failed
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
> 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk]
> client3: client lookup failed
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
> 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk]
> client3: client lookup failed
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
> 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3:
> non-blocking connect() returned: 111 (Connection refused)
>
> repeated tens of thousands of times
>
> The log is growing rapidly.  It is now 3G.
> Log attached.
>
>
> About ten minutes later I retried the same command without doing
> anything to the servers or clients and it is running:
> [root at grp-01-20-01 mnt]# cp -ax /usr /mnt/glusterfs0
>
>
> This behavior is very strange.
>
>
>
> Gerry
>
>
>