[Gluster-users] Weird file and file truncation problems

[Ling Ho]

I have a file on a brick with weird permission mode. I thought the "T" 
only appears on zero length pointer files.

-r--r-s--T+ 2 psdatmgr ps-data 98780901596 Jan 18 15:06 
e141-r0001-s02-c01.xtc

lsof show it is being held open and read/write by the glusterfs process 
for the brick.

glusterfs 11336      root   55u      REG               8,48 
98780901596       4133 /brick3/cxi/cxi43212/xtc/e141-r0001-s02-c01.xtc


The file was written back in January, and I don't believe I have any 
client process that have opened the file with write mode.

And the glusterfsd process is consuming ~600 CPU% (out of 8 cores with 
hyperthreading).

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11336 root      20   0 3603m 130m 2512 S 579.1  0.3   8340:01 glusterfsd


I have started a rebalance job earlier yesterday, but later stopped it. 
Does this has something to do with rebalancing?

I would like to restart glusterd on this machine, but is there a way to 
tell if any of the files on this server is opened? I ran glusterfs 
volume top, but can't tell if the files shown are currently open file. I 
don't see this e141-r0001-s02-c01.xtc file from the top command.


A possibly related, and more troubling problem I am having is with file 
truncations.

In the past 6 months running 3.2.5 and 3.2.6, we have seen cases of file 
truncations after unplanned power outage to our storage devices. I have 
suspected write cache on our raid controllers since in those cases it 
took us 2-3 days before we could get power restored and bring the disks 
back up. The battery on the raid controller would not have lasted that long.

Unfortunately, in the last few days, after proper machine shutdowns and 
reboots, we discovered file truncations again. We run a script that take 
down the file size of our data files once a day, and the next day we 
found the file size of some of the files have been reduced. Some of the 
smaller files became zero length. These are files we wrote once and 
never write again, only open for reading.

The troubling thing is, these files aren't freshly written when the 
machines were rebooted. They were 2-3 days old. And yesterday, I found 
one file from the same batch of files as the above mentioned 
e141-r0001-s02-c01.xtc, also written in January became truncated.

Yesterday, I looked at about 10 truncated files from the same brick, 
examined them using xfs_bmap. They all appeared to be using a single 
extend. So, I look at the original untruncated files from the source, 
figured out the correct length, and just built new files based on the 
location returned by xfs_bmap, and the correct length from the original 
files. (something like this:   dd if=/dev/sdh of=/root/newfile bs=512 
skip=39696059392 count=83889655). Turns out, what I was able to extract 
was the same as the original file. So, the data was indeed written to 
disk, and not merely stored in cache somewhere. However, the file size 
had mysteriously changed.

I know this could be a XFS problem. I would appreciate if I can get some 
suggestions on how to reproduce it, and what to look for. I have a test 
machine with similar but smaller amount of disk space, but I have not 
been able to reproduce the problem there.

I have upgraded to 3.3.0. The latest round of file truncation happened 
after I have upgraded, after I stop gluster volume, rebooted the machine 
cleanly.

I am running RHEL6.2 kernel version 2.6.32-220.17.1.el6.x86_64 .

Thanks,
...
ling









Thanks,
...
ling