[Gluster-devel] Problem with TLA ver > 887
Anand Avati
avati at zresearch.com
Tue Feb 10 13:45:16 UTC 2009
Mickey,
Can you check if the latest tla code has resolved the issues you faced?
Thanks,
Avati
On Sun, Feb 8, 2009 at 11:19 PM, Mickey Mazarick <mic at digitaltadpole.com> wrote:
> Heh our tests are kind of an unholy mess... but here's the part I think is
> useful:
> We use a startup script that will iterate through vol files and mount the
> first available file on the list. We have a bunch of vol files that test a
> few different server configurations. After mountpoints are prepared we have
> other scripts that start virtual machine on the various mounts.
>
> In other words I have a directory called "/glustermounts/" and in that
> directory I have the files:
> main.vol main.vol.ib main.vol.tcp stripe.vol.ha stripe.vol.tcp
>
> after running "/etc/init.d/glustersystem start" I will have the following
> mount points:
> /system (our default mount, we actually store the vol files here)
> /mnt/main
> /mnt/stripe
>
> The output shows me if any vol file failed to mount and it automatically
> attempts the next one (ex" "mounting main.vol failed, trying main.vol.ib").
> We simply arrange vol files from most features to least. We have a separate
> script which starts up a virtual machine on each test mount. This is the
> actual "test" we use as it creates symbolic links, uses mmaps etc but it's
> pretty specific to us. This closely mirrors how we use it in production.
>
> I've included out startup script and I would suggest you simply run
> something similar to your production on a few mounts in the same way we
> have. I may share this with the entire group although there are probably
> better init scripts out there. This one does kill all processes attached to
> a mount point which is useful. Let me know if you have any questions!
>
> Thanks!
>
> -Mickey Mazarick
>
>
>
> Geoff Kassel wrote:
>
> Hi,
> As a fellow GlusterFS user, I was just wondering if you could point me to
> the regression tests you're using for GlusterFS?
>
> I've looked high and low for the unit tests that the GlusterFS devs are
> meants to be using (ala http://www.gluster.org/docs/index.php/GlusterFS_QA)
> so that I can do my own testing, but I've not been able to find them.
>
> If it's tests you've developed in-house, would you be interested in
> releasing them to the wider community?
>
> Kind regards,
>
> Geoff Kassel.
>
> On Thu, 5 Feb 2009, Mickey Mazarick wrote:
>
>
> I haven't done any full regression testing to see where the problem is
> but the later TLA versions are causeing out storage servers to spike to
> 100% cpu usage and the clients never see any files. Our initial tests
> are with ibverbs/HA but no performance translators.
>
> Thanks!
> -Mickey Mazarick
>
>
> --
>
> #!/bin/sh
> # Startup script for gluster Mount system
> volFiles="/glustermounts/"
> defaultcheckFile="customers"
> speclist="/etc/glusterfs-system.vol.ibverbs /etc/glusterfs-system.vol.ha
> /etc/glusterfs-system.vol.ibverbs /etc/glusterfs-system.vol.tcp"
> start() {
> specfile=${1}
> if [ "$#" -gt 1 ]; then
> mountpt=${2}
> else
> mountpt=`echo ${specfile} |sed "s#\.vol.*\\\$##" |sed
> "s#/.*/##"`
> mountpt="/mnt/${mountpt}"
> fi
> logfile=`echo ${specfile} |sed "s#\.vol.*\\\$##" |sed "s#/.*/##"`
> logfile="/var/${logfile}.log"
> pidfile=`echo ${specfile} |sed "s#\.vol.*\\\$##" |sed "s#/.*/##"`
> pidfile="/var/run/${pidfile}.pid"
> echo "mounting specfile:${specfile} at:${mountpt} with pid
> at:${pidfile}"
> currentpids=`pidof glusterfs`
> currentpids="0 ${currentpids}"
> mountct=`mount |grep ${mountpt} |grep -c glusterfs`
> if [ -f $pidfile ]; then
> currentpid=`cat ${pidfile}`
> pidct=`echo "${currentpids}" |grep -c ${currentpid}`
> if [ "${pidct}" -eq 0 ]; then
> rm -rf ${pidfile}
> echo "removing pid file: ${pidfile}"
> fi
> if [ "${mountct}" -lt 1 ]; then
> echo "Gluster System mount:${mountpt} died.
> Remounting."
> stop ${mountpt} ${pidfile}
> fi
> else
> rm -rf ${pidfile}
> if [ "${mountct}" -gt 0 ]; then
> myupid=`ps -ef |grep /system |grep gluster |sed
> "s#root\s*##" |sed "s#\s.*##"`
> if [ "${myupid}" -gt 0 ]; then
> echo "${myupid}" > ${pidfile}
> else
> echo "Gluster System mounted at:${mountpt} but
> with no pid. Remounting."
> stop ${mountpt} ${pidfile}
> fi
> fi
> fi
>
> if [ -e $pidfile ]; then
> echo "Gluster System Mount:${mountpt} is running with spec:
> ${specfile}"
> #echo "Gluster System Mount:${mountpt} is running."
> return 0
> else
> #rm -rf /var/glustersystemclient.log
> modprobe fuse
> sleep 1.5
> #rm -rf /var/glustersystemclient.log
> mkdir ${mountpt}
> rm -rf $pidfile
> cmd="/usr/local/sbin/glusterfs -p $pidfile -l ${logfile} -L ERROR -f
> ${specfile} --disable-direct-io-mode ${mountpt}"
> echo "${cmd}"
> ${cmd}
> #/usr/local/sbin/glusterfs -p $pidfile -l ${logfile}
> --volume-specfile=${specfile} --disable-direct-io-mode ${mountpt}
> #/usr/local/sbin/glusterfs -p $pidfile -l
> /var/glustersystemclient.log -f $specfile --direct-io-mode=DISABLE /system
> fi
> return 1
> }
>
> checkStart() {
> mountdir=$1
> checkfile="total"
> if [ "$#" -gt 1 ]; then
> checkfile=$2
> fi
> lspid=0
> sleep 1
> counter=0
> countermax=15
>
> ls -l ${mountdir} &
> while [ "${lspid}" != "" ]
> do
> echo "waiting for gluster to come up... ${counter}"
> sleep 1
> lspid=`/sbin/pidof ls`
> let counter++
> if [ "${counter}" -eq "${countermax}" ]
> then
> lspid=""
> fi
> done
> if [ "${counter}" -lt "${countermax}" ]; then
> errorct=`ls ${mountdir} 2>&1 |grep -c "not connected"`
> if [ "${errorct}" -eq 1 ]; then
> counter=`echo ${countermax}`
> else
> glcount=`ls -l ${mountdir} |grep ${checkfile} -c`
> if [ "${glcount}" -lt 1 ]; then
> counter=`echo ${countermax}`
> fi
> fi
> fi
>
> if [ "${counter}" -eq "${countermax}" ]
> then
> echo "gluster FAILED to mount:${mountdir} with spec: ${specfile}"
> lspid=`/sbin/pidof ls`
> kill $lspid
> lspid=10
> return 0
> else
> echo "Gluster System Mount:${mountdir} is running with spec:
> ${specfile}"
> #echo "gluster sucessfully mounted:${mountdir} with spec:
> ${specfile}"
> return 1
> fi
> }
>
> StartSpeclist() {
> specfilelist="${1}"
> echo "Attempting to mount first of: (${specfilelist})"
> for file in $specfilelist
> do
> specfile="${file}"
> if [ "$#" -gt 1 ]; then
> checkfile=${2}
> else
> checkfile="total"
> fi
> if [ "$#" -gt 2 ]; then
> mountpt=${3}
> else
> mountpt=`echo ${specfile} |sed "s#\.vol.*\\\$##" |sed
> "s#/.*/##"`
> mountpt="/mnt/${mountpt}"
> fi
>
> start ${specfile} ${mountpt}
> if [ "$?" -eq "1" ]; then
> checkStart ${mountpt} ${checkfile}
> if [ "$?" -eq "0" ]; then
> stop ${specfile} ${mountpt}
> else
> return 1
> fi
> else
> return 1
> fi
> done
> return 0
> }
>
> stop() {
> specfile1=${1}
> if [ "$#" -gt 1 ]; then
> mountpt=${2}
> else
> mountpt=`echo ${specfile1} |sed "s#\.vol.*\\\$##" |sed
> "s#/.*/##"`
> mountpt="/mnt/${mountpt}"
> fi
> pidfile=`echo ${specfile1} |sed "s#\.vol.*\\\$##" |sed "s#/.*/##"`
> pidfile="/var/run/${pidfile}.pid"
> #runningpids=`lsof |grep ${mountpt} |sed "s#..........##" |sed "s#
> .*##"`
> # for pid in `lsof |grep ${mountpt} |sed "s#\w*\s*##" |sed "s# .*##"`
> for pid in `lsof |grep ${mountpt} |sed
> "s#..........\(......\).*#\1#"`
> do
> kill -9 $pid
> done
> #fuser -km /system
> echo "Stopping mount:${mountpt} spec:${specfile1}"
> umount -f ${mountpt}
> currentpid=`cat ${pidfile}`
> kill $currentpid
> rm -rf $pidfile
> }
>
> stopmp(){
> mountpt=${1}
> spec=`ps -ef |grep gluster |grep ${mountpt} |grep specfile |sed
> "s#.*specfile=\(.*\)/s*.*#\1#"| sed "s# .*##"`
> stop "${spec}" "${mountpt}"
> }
>
> startAll() {
> StartSpeclist "${speclist}" "glustermounts" /system
> if [ "$?" -eq "0" ]; then
> echo "ERROR STARTING"
> else
> #for i in `ls -b ${volFiles}*.vol |sed s/.glustermounts.//`;
> for i in `ls -b ${volFiles}*.vol |sed s#${volFiles}##`;
> do
> list=`ls -C ${volFiles}${i}*`
> mountpt=`echo $i |sed s/\.vol//`
> StartSpeclist "${list}" "${defaultcheckFile}" /mnt/${mountpt}
> done
>
> fi
>
> }
>
> stopAll() {
> mountlist=`mount |grep glusterfs |sed "s#glusterfs on \(.*\)
> type.*#\1#"`
> for mountpt in ${mountlist};
> do
> stopmp ${mountpt}
> done
> kill `pidof glusterfs`
> }
>
> case "$1" in
> start)
> if [ "$#" -gt 1 ]; then
>
> StartSpeclist "${2}" ${3} ${4}
>
> else
> startAll
> fi
> ;;
>
> stop)
> if [ "$#" -gt 1 ]; then
> stopmp ${2}
> else
> stopAll
> stopAll
> fi
>
> ;;
>
> status)
> status
> ;;
> restart)
> stop
> start
> ;;
> condrestart)
> stop
> start
> ;;
>
> *)
> echo $"Usage: $0 {start|stop|restart|condrestart|status}"
> exit 1
>
> esac
>
> exit $RETVAL
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
More information about the Gluster-devel
mailing list