[Gluster-users] Quota trouble
Atin Mukherjee
amukherj at redhat.com
Wed Apr 22 03:58:41 UTC 2015
On 04/21/2015 08:09 PM, kenji kondo wrote:
> Hello Vijay, Atin and Avra,
> Thanks a lot for your advises.
>
> Because users are using, currently I cannot stop our server but I'm
> planning to restart the suspicious host gluster13.
>
> I use the glusterd version 3.6.1 for all servers and 3.6.0.29 for clients.
> The OS is CentOS 6.6.
> In /var/lib/glusterd/glusterd.info, “operating-version=1” is found for all
> server hosts.
>
> I don’t know how should I check the .cmd_log_history,
.cmd_log_history is a log file (hidden) which resides in
/var/log/glusterfs/ . This log file contains the cli commands which were
executed on the very same node. What I wanted to know from this file
across all the nodes is whether at a given point of time simultaneous
commands were executed, if that's the case it might end up in a stale
lock situation. The upcoming 3.7 release and 3.6.2 has the fix for it.
~Atin
> but I tried to find “volume status” in the log as following
> $ grep ‘volume status’ .cmd_log_history
> Basically I could find “volume status : SUCESS” for almost all node
> excepted with gluster13. In .cmd_log_history in gluster13, I found “volume
> status testvol : FAILED : Locking failed on gluster13. Please check log
> file for details.”
>
> Best regards,
> Kondo
>
>
>
> 2015-04-21 18:27 GMT+09:00 Atin Mukherjee <amukherj at redhat.com>:
>
>>
>>
>> On 04/21/2015 02:47 PM, Avra Sengupta wrote:
>>> In the logs I see, glusterd_lock() being used. This api is called only
>>> in older versions of gluster or if you have a cluster version is less
>>> then 30600. So along with the version of glusterfs used, could you also
>>> let us know what is the cluster version. You can check it as
>>> "operating-version" in /var/lib/glusterd/glusterd.info file.
>> Additionally please check whether concurrent volume operations were
>> triggered by checking .cmd_log_history across all the nodes, if so, this
>> could result into stale locks.
>>
>> ~Atin
>>>
>>> Regards,
>>> Avra
>>>
>>> On 04/21/2015 02:34 PM, Avra Sengupta wrote:
>>>> Hi Kondo,
>>>>
>>>> Can u also mention the version of gluster you are using.
>>>>
>>>> +Adding gluster-users
>>>>
>>>> Regards,
>>>> Avra
>>>> On 04/21/2015 02:27 PM, Avra Sengupta wrote:
>>>>> Hi Kondo,
>>>>>
>>>>> I went through the gluster13 logs you had sent. Seems like something
>>>>> on that machine is holding the lock and is not releasing it. There
>>>>> are ways in which the system might end up in this scenario. I will
>>>>> try and explain the same with an example.
>>>>>
>>>>> Let's say I have gluster 11, gluster12, and gluster 13 in my cluster.
>>>>> I initiate a command from gluster11. Now the first thing that command
>>>>> does is, it holds a lock on all the nodes in the cluster on behalf of
>>>>> gluster11. Once the command does what's intended, it's last act
>>>>> before ending is to unlock all the nodes in the cluster. Now, only
>>>>> the node that has issued the lock, can issue the unlock.
>>>>>
>>>>> In your case what has happened is some command after successfully
>>>>> acquired the lock on gluster13. Now the node which initiated the
>>>>> command, went down or glusterd on that node went down before it could
>>>>> complete the command and it never got to send the unlock to gluster13.
>>>>>
>>>>> There's a workaround to it. You can restart glusterd on gluster13 and
>>>>> it should work fine.
>>>>>
>>>>> Regards,
>>>>> Avra
>>>>>
>>>>> On 04/20/2015 06:55 PM, kenji kondo wrote:
>>>>>> Hello Vijay,
>>>>>> Maybe this is very rare case. But is there any idea?
>>>>>>
>>>>>> Thanks,
>>>>>> Kondo
>>>>>>
>>>>>> 2015-04-15 9:47 GMT+09:00 Vijaikumar M <vmallika at redhat.com
>>>>>> <mailto:vmallika at redhat.com>>:
>>>>>>
>>>>>> Adding Avra...
>>>>>>
>>>>>> Thanks,
>>>>>> Vijay
>>>>>>
>>>>>>
>>>>>> -------- Forwarded Message --------
>>>>>> Subject: Re: [Gluster-users] Quota trouble
>>>>>> Date: Wed, 15 Apr 2015 00:27:26 +0900
>>>>>> From: kenji kondo <kkay.jp at gmail.com>
>>>>>> <mailto:kkay.jp at gmail.com>
>>>>>> To: Vijaikumar M <vmallika at redhat.com>
>>>>>> <mailto:vmallika at redhat.com>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Vijay,
>>>>>>
>>>>>> Thanks for your comments.
>>>>>>
>>>>>>
>>>>>> The lock error occurs at one server it's called "gluster13".
>>>>>>
>>>>>> In the gluster13, I tried to create new volume and start quota.
>>>>>> But it failed as below,
>>>>>>
>>>>>>
>>>>>> In both host gluster10 and gluster13, ran below
>>>>>>
>>>>>> $ sudo mkdir /export11/testbrick1
>>>>>>
>>>>>> $ sudo mkdir /export11/testbrick2
>>>>>>
>>>>>> In gluster13, ran below
>>>>>>
>>>>>> $ sudo /usr/sbin/gluster volume create testvol2
>>>>>> gluster13:/export11/testbrick1 gluster13:/export11/testbrick2
>>>>>>
>>>>>> volume create: testvol2: failed: Locking failed on gluster13.
>>>>>> Please check log file for details.
>>>>>>
>>>>>> $ sudo /usr/sbin/gluster volume create testvol2
>>>>>> gluster10:/export11/testbrick1 gluster10:/export11/testbrick2
>>>>>>
>>>>>> volume create: testvol2: failed: Locking failed on gluster13.
>>>>>> Please check log file for details.
>>>>>>
>>>>>> But I recived error messages above.
>>>>>>
>>>>>> On the other hand, in gluster10, it was success.
>>>>>>
>>>>>> Again, in gluster13, I tried to run quota, but it failed as below.
>>>>>>
>>>>>> $ sudo /usr/sbin/gluster volume quota testvol2 enable
>>>>>>
>>>>>> quota command failed : Locking failed on gluster13. Please check
>>>>>> log file for details.
>>>>>>
>>>>>>
>>>>>> Could you find attached?
>>>>>>
>>>>>> We can find error messages in the log of gluster13.
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Kondo
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2015-04-13 19:38 GMT+09:00 Vijaikumar M <vmallika at redhat.com
>>>>>> <mailto:vmallika at redhat.com>>:
>>>>>>
>>>>>> Hi Kondo,
>>>>>>
>>>>>> The lock error you mentioned is because, another operation
>>>>>> is still running on the volume and hence not able to acquire
>>>>>> the lock.
>>>>>> This is bug of not displaying proper error message, we are
>>>>>> working on fixing this issue.
>>>>>>
>>>>>> I was not able to find any clue on why quotad is not running.
>>>>>>
>>>>>> I wanted to check, if we can manually start quotad something
>>>>>> like below:
>>>>>>
>>>>>> # /usr/local/sbin/glusterfs -s localhost --volfile-id
>>>>>> gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l
>>>>>> /var/log/glusterfs/quotad.log -S
>>>>>> /var/run/gluster/myquotad.socket --xlator-option
>>>>>> *replicate*.data-self-heal=off --xlator-option
>>>>>> *replicate*.metadata-self-heal=off --xlator-option
>>>>>> *replicate*.entry-self-heal=off
>>>>>>
>>>>>> or
>>>>>>
>>>>>> create a new temporary volume, and enable quota on this
>>>>>> volume. (quotad will be same for all the volume which has
>>>>>> quota enabled)
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Vijay
>>>>>>
>>>>>>
>>>>>> On Sunday 12 April 2015 07:05 PM, kenji kondo wrote:
>>>>>>> Hi Vijay,
>>>>>>>
>>>>>>> Thank you for your suggestion. But I'm sorry, it's
>>>>>>> difficult to access from outside because my glusterfs
>>>>>>> system is closed.
>>>>>>> I will give up if there is no clue information in attached
>>>>>>> log.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Kondo
>>>>>>>
>>>>>>>
>>>>>>> 2015-04-09 15:40 GMT+09:00 Vijaikumar M
>>>>>>> <vmallika at redhat.com <mailto:vmallika at redhat.com>>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thursday 09 April 2015 11:58 AM, Vijaikumar M wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wednesday 08 April 2015 09:57 PM, kenji kondo wrote:
>>>>>>>>> Hi Vijay,
>>>>>>>>>
>>>>>>>>> I checked the all of the setting.
>>>>>>>>> The all are 'features.quota=on' when I set quota
>>>>>>>>> enable and the all are 'features.quota=off' when I
>>>>>>>>> set quota disable.
>>>>>>>>>
>>>>>>>>> But I could find new issue.
>>>>>>>>> When I checked a volume status for all server, in one
>>>>>>>>> of the servers I received the error message as below.
>>>>>>>>>
>>>>>>>>> $ sudo /usr/sbin/gluster volume status testvol
>>>>>>>>> Locking failed on gluster13. Please check log file
>>>>>>>>> for details.
>>>>>>>>>
>>>>>>>>> In etc-glusterfs-glusterd.vol.log of problem server,
>>>>>>>>> I found error messages as below.
>>>>>>>>> [2015-04-08 08:40:04.782644] I
>>>>>>>>> [mem-pool.c:545:mem_pool_destroy] 0-management:
>>>>>>>>> size=588 max=0 total=0
>>>>>>>>> [2015-04-08 08:40:04.782685] I
>>>>>>>>> [mem-pool.c:545:mem_pool_destroy] 0-management:
>>>>>>>>> size=124 max=0 total=0
>>>>>>>>> [2015-04-08 08:40:04.782848] W
>>>>>>>>> [socket.c:611:__socket_rwv] 0-management: readv on
>>>>>>>>> /var/run/14b05cd492843e6e288e290c2d63093c.socket
>>>>>>>>> failed (Invalid arguments)
>>>>>>>>> [2015-04-08 08:40:04.805407] I [MSGID: 106006]
>>>>>>>>> [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify]
>>>>>>>>> 0-management: nfs has disconnected from glusterd.
>>>>>>>>> [2015-04-08 08:43:02.439001] I
>>>>>>>>>
>>>>>>>>> [glusterd-handler.c:3803:__glusterd_handle_status_volume]
>>>>>>>>> 0-management: Received status volume req for volume
>>>>>>>>> testvol
>>>>>>>>> [2015-04-08 08:43:02.460581] E
>>>>>>>>> [glusterd-utils.c:148:glusterd_lock] 0-management:
>>>>>>>>> Unable to get lock for uuid:
>>>>>>>>> 03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by:
>>>>>>>>> 03a32bce-ec63-4dc3-a287-4901a55dd8c9
>>>>>>>>> [2015-04-08 08:43:02.460632] E
>>>>>>>>> [glusterd-op-sm.c:6584:glusterd_op_sm] 0-management:
>>>>>>>>> handler returned: -1
>>>>>>>>> [2015-04-08 08:43:02.460654] E
>>>>>>>>> [glusterd-syncop.c:105:gd_collate_errors] 0-: Locking
>>>>>>>>> failed on gluster13. Please check log file for details.
>>>>>>>>> [2015-04-08 08:43:02.461409] E
>>>>>>>>> [glusterd-syncop.c:1602:gd_sync_task_begin]
>>>>>>>>> 0-management: Locking Peers Failed.
>>>>>>>>> [2015-04-08 08:43:43.698168] I
>>>>>>>>>
>>>>>>>>> [glusterd-handler.c:3803:__glusterd_handle_status_volume]
>>>>>>>>> 0-management: Received status volume req for volume
>>>>>>>>> testvol
>>>>>>>>> [2015-04-08 08:43:43.698813] E
>>>>>>>>> [glusterd-utils.c:148:glusterd_lock] 0-management:
>>>>>>>>> Unable to get lock for uuid:
>>>>>>>>> 03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by:
>>>>>>>>> 03a32bce-ec63-4dc3-a287-4901a55dd8c9
>>>>>>>>> [2015-04-08 08:43:43.698898] E
>>>>>>>>> [glusterd-op-sm.c:6584:glusterd_op_sm] 0-management:
>>>>>>>>> handler returned: -1
>>>>>>>>> [2015-04-08 08:43:43.698994] E
>>>>>>>>> [glusterd-syncop.c:105:gd_collate_errors] 0-: Locking
>>>>>>>>> failed on gluster13. Please check log file for details.
>>>>>>>>> [2015-04-08 08:43:43.702126] E
>>>>>>>>> [glusterd-syncop.c:1602:gd_sync_task_begin]
>>>>>>>>> 0-management: Locking Peers Failed.
>>>>>>>>> [2015-04-08 08:44:01.277139] I
>>>>>>>>>
>>>>>>>>> [glusterd-handler.c:3803:__glusterd_handle_status_volume]
>>>>>>>>> 0-management: Received status volume req for volume
>>>>>>>>> testvol
>>>>>>>>> [2015-04-08 08:44:01.277560] E
>>>>>>>>> [glusterd-utils.c:148:glusterd_lock] 0-management:
>>>>>>>>> Unable to get lock for uuid:
>>>>>>>>> 03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by:
>>>>>>>>> 03a32bce-ec63-4dc3-a287-4901a55dd8c9
>>>>>>>>> [2015-04-08 08:44:01.277639] E
>>>>>>>>> [glusterd-op-sm.c:6584:glusterd_op_sm] 0-management:
>>>>>>>>> handler returned: -1
>>>>>>>>> [2015-04-08 08:44:01.277676] E
>>>>>>>>> [glusterd-syncop.c:105:gd_collate_errors] 0-: Locking
>>>>>>>>> failed on gluster13. Please check log file for details.
>>>>>>>>> [2015-04-08 08:44:01.281514] E
>>>>>>>>> [glusterd-syncop.c:1602:gd_sync_task_begin]
>>>>>>>>> 0-management: Locking Peers Failed.
>>>>>>>>> [2015-04-08 08:45:42.599796] I
>>>>>>>>>
>>>>>>>>> [glusterd-handler.c:3803:__glusterd_handle_status_volume]
>>>>>>>>> 0-management: Received status volume req for volume
>>>>>>>>> testvol
>>>>>>>>> [2015-04-08 08:45:42.600343] E
>>>>>>>>> [glusterd-utils.c:148:glusterd_lock] 0-management:
>>>>>>>>> Unable to get lock for uuid:
>>>>>>>>> 03a32bce-ec63-4dc3-a287-4901a55dd8c9, lock held by:
>>>>>>>>> 03a32bce-ec63-4dc3-a287-4901a55dd8c9
>>>>>>>>> [2015-04-08 08:45:42.600417] E
>>>>>>>>> [glusterd-op-sm.c:6584:glusterd_op_sm] 0-management:
>>>>>>>>> handler returned: -1
>>>>>>>>> [2015-04-08 08:45:42.600482] E
>>>>>>>>> [glusterd-syncop.c:105:gd_collate_errors] 0-: Locking
>>>>>>>>> failed on gluster13. Please check log file for details.
>>>>>>>>> [2015-04-08 08:45:42.601039] E
>>>>>>>>> [glusterd-syncop.c:1602:gd_sync_task_begin]
>>>>>>>>> 0-management: Locking Peers Failed.
>>>>>>>>>
>>>>>>>>> Does this situation relate to my quota problems?
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is a glusterd different issue. Can we get the
>>>>>>>> glusterd logs from gluster13?
>>>>>>>> Can get access to these machines, so that we can debug
>>>>>>>> live?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Vijay
>>>>>>>>
>>>>>>> Regarding quota issue, quota feature is enabled
>>>>>>> successfully. I am wondering why quotad is not started.
>>>>>>> If we get the access to the machine, it will be easier
>>>>>>> to debug the issue.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Vijay
>>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Kondo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2015-04-08 15:14 GMT+09:00 Vijaikumar M
>>>>>>>>> <vmallika at redhat.com <mailto:vmallika at redhat.com>>:
>>>>>>>>>
>>>>>>>>> Hi Kondo,
>>>>>>>>>
>>>>>>>>> I suspect, in one of the node quota feature is
>>>>>>>>> not set for some reason and hence quotad is not
>>>>>>>>> starting.
>>>>>>>>>
>>>>>>>>> On all the nodes can you check if below option is
>>>>>>>>> set to 'on'
>>>>>>>>>
>>>>>>>>> # grep quota /var/lib/glusterd/vols/<volname>/info
>>>>>>>>> features.quota=on
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Also can I get brick logs from all the nodes?
>>>>>>>>>
>>>>>>>>> Also can you create a temporary volume and enable
>>>>>>>>> the quota here and see if see quota works fine
>>>>>>>>> with this volume?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Vijay
>>>>>>>>>
>>>>>>>>> On Tuesday 07 April 2015 08:34 PM, kenji kondo
>>>>>>>>> wrote:
>>>>>>>>>> Hi Vijay,
>>>>>>>>>>
>>>>>>>>>> Could you find attached?
>>>>>>>>>> I got logs of server and client.
>>>>>>>>>> As same as before, I could not create a file
>>>>>>>>>> after quota usage-limit setting.
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> Kondo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2015-04-07 18:34 GMT+09:00 Vijaikumar M
>>>>>>>>>> <vmallika at redhat.com <mailto:vmallika at redhat.com
>>>> :
>>>>>>>>>>
>>>>>>>>>> Hi Konda,
>>>>>>>>>>
>>>>>>>>>> Can we get all the log files?
>>>>>>>>>>
>>>>>>>>>> # gluster volume quota <volname> disable
>>>>>>>>>> # gluster volume quota <volname> enable
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Now copy all the logs files.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Vijay
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tuesday 07 April 2015 12:39 PM, K.Kondo
>>>>>>>>>> wrote:
>>>>>>>>>>> Thank you very much ! Vijay
>>>>>>>>>>> I want to use a quota because each volume
>>>>>>>>>>> became too big.
>>>>>>>>>>>
>>>>>>>>>>> Best regard
>>>>>>>>>>> Kondo
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2015/04/07 15:18、Vijaikumar M
>>>>>>>>>>> <vmallika at redhat.com
>>>>>>>>>>> <mailto:vmallika at redhat.com>> のメッセージ:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Kondo,
>>>>>>>>>>>>
>>>>>>>>>>>> I couldn’t find clue from the logs. I will
>>>>>>>>>>>> discuss about this issue with my
>>>>>>>>>>>> colleagues today.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Vijay
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Monday 06 April 2015 10:56 PM, kenji
>>>>>>>>>>>> kondo wrote:
>>>>>>>>>>>>> Hello Vijay,
>>>>>>>>>>>>> Is there something idea for this?
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Kondo
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2015-03-31 22:46 GMT+09:00 kenji kondo
>>>>>>>>>>>>> <kkay.jp at gmail.com
>>>>>>>>>>>>> <mailto:kkay.jp at gmail.com>>:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm sorry for late reply.
>>>>>>>>>>>>> I could get the debug mode log as
>>>>>>>>>>>>> attached.
>>>>>>>>>>>>> In this test, unfortunately the quota
>>>>>>>>>>>>> did not work as same as before.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could you find the cause of my problem?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> Kondo
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2015-03-25 17:20 GMT+09:00 Vijaikumar
>>>>>>>>>>>>> M <vmallika at redhat.com
>>>>>>>>>>>>> <mailto:vmallika at redhat.com>>:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Kondo,
>>>>>>>>>>>>>
>>>>>>>>>>>>> For some reason quota enable was
>>>>>>>>>>>>> not successful. We may have
>>>>>>>>>>>>> re-try enabling quota.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tuesday 24 March 2015 07:08
>>>>>>>>>>>>> PM, kenji kondo wrote:
>>>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>>> Thanks for your checking.
>>>>>>>>>>>>>> Unfortunately, currently I can't
>>>>>>>>>>>>>> stop the service because many
>>>>>>>>>>>>>> users are using.
>>>>>>>>>>>>>> But, I want to know this cause
>>>>>>>>>>>>>> of this trouble, so I will plan
>>>>>>>>>>>>>> to stop. Please wait to get the
>>>>>>>>>>>>>> log.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>> Kondo
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2015-03-24 17:01 GMT+09:00
>>>>>>>>>>>>>> Vijaikumar M
>>>>>>>>>>>>>> <vmallika at redhat.com
>>>>>>>>>>>>>> <mailto:vmallika at redhat.com>>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Kondo,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I couldn't find much clue in
>>>>>>>>>>>>>> the glusterd logs, other
>>>>>>>>>>>>>> than the error message you
>>>>>>>>>>>>>> mentioned below.
>>>>>>>>>>>>>> Can you try disabling and
>>>>>>>>>>>>>> enabling the quota again and
>>>>>>>>>>>>>> see if this start quotad?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Try below command:
>>>>>>>>>>>>>> # gluster volume quota
>>>>>>>>>>>>>> <volname> disable
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wait for all quota process
>>>>>>>>>>>>>> to terminate
>>>>>>>>>>>>>> #ps -ef | quota
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # service glusterd stop
>>>>>>>>>>>>>> # glusterd -LDEBUG
>>>>>>>>>>>>>> # gluster volume quota
>>>>>>>>>>>>>> <volname> enable
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now verify if quotad is
>> running
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Monday 23 March 2015
>>>>>>>>>>>>>> 06:24 PM, kenji kondo wrote:
>>>>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>>>> As you pointed out, the
>>>>>>>>>>>>>>> quotad is not running in
>>>>>>>>>>>>>>> the all of server.
>>>>>>>>>>>>>>> I checked the volume status
>>>>>>>>>>>>>>> and got following log.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Quota Daemon on
>>>>>>>>>>>>>>> gluster25N/ANN/A
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So, I attached requested
>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 'etc-glusterfs-glusterd.vol.log'.
>>>>>>>>>>>>>>> The error messages can be
>>>>>>>>>>>>>>> found in the log.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [2015-03-19
>>>>>>>>>>>>>>> 11:51:07.457697] E
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [glusterd-quota.c:1467:glusterd_op_stage_quota]
>>>>>>>>>>>>>>> 0-management: Quota is
>>>>>>>>>>>>>>> disabled, please enable quota
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If you want more some
>>>>>>>>>>>>>>> information to solve this
>>>>>>>>>>>>>>> problems, please ask me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> Kondo
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2015-03-23 16:04 GMT+09:00
>>>>>>>>>>>>>>> Vijaikumar M
>>>>>>>>>>>>>>> <vmallika at redhat.com
>>>>>>>>>>>>>>> <mailto:vmallika at redhat.com
>>>> :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Kondo,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you please verify
>>>>>>>>>>>>>>> if quotad is running?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> root at rh1:~ *# gluster
>>>>>>>>>>>>>>> volume status*
>>>>>>>>>>>>>>> Status of volume: vol1
>>>>>>>>>>>>>>> Gluster process TCP
>>>>>>>>>>>>>>> Port RDMA Port Online
>>>>>>>>>>>>>>> Pid
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> ------------------------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Brick
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> rh1:/var/opt/gluster/bricks/b1/dir
>>>>>>>>>>>>>>> 49152 0 Y 1858
>>>>>>>>>>>>>>> NFS Server on localhost
>>>>>>>>>>>>>>> 2049 0 Y 1879
>>>>>>>>>>>>>>> *Quota Daemon on
>>>>>>>>>>>>>>> localhost N/A N/A
>>>>>>>>>>>>>>> Y 1914 **
>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>> Task Status of Volume
>> vol1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>> ------------------------------------------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There are no active
>>>>>>>>>>>>>>> volume tasks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> root at rh1:~ # *ps -ef |
>>>>>>>>>>>>>>> grep quotad*
>>>>>>>>>>>>>>> root 1914 1 0
>>>>>>>>>>>>>>> 12:29 ? 00:00:00
>>>>>>>>>>>>>>> /usr/local/sbin/glusterfs
>>>>>>>>>>>>>>> -s localhost
>>>>>>>>>>>>>>> --volfile-id
>>>>>>>>>>>>>>> gluster/quotad -p
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> /var/lib/glusterd/quotad/run/quotad.pid
>>>>>>>>>>>>>>> -l
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> */var/log/glusterfs/quotad.log*-S
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> /var/run/gluster/bb6ab82f70f555fd5c0e188fa4e09584.socket
>>>>>>>>>>>>>>> --xlator-option
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *replicate*.data-self-heal=off
>>>>>>>>>>>>>>> --xlator-option
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *replicate*.metadata-self-heal=off
>>>>>>>>>>>>>>> --xlator-option
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *replicate*.entry-self-heal=off
>>>>>>>>>>>>>>> root 1970 1511 0
>>>>>>>>>>>>>>> 12:31 pts/1 00:00:00
>>>>>>>>>>>>>>> grep quotad
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> root at rh1:~ # *gluster
>>>>>>>>>>>>>>> volume info*
>>>>>>>>>>>>>>> Volume Name: vol1
>>>>>>>>>>>>>>> Type: Distribute
>>>>>>>>>>>>>>> Volume ID:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> a55519ec-65d1-4741-9ad3-f94020fc9b21
>>>>>>>>>>>>>>> Status: Started
>>>>>>>>>>>>>>> Number of Bricks: 1
>>>>>>>>>>>>>>> Transport-type: tcp
>>>>>>>>>>>>>>> Bricks:
>>>>>>>>>>>>>>> Brick1:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> rh1:/var/opt/gluster/bricks/b1/dir
>>>>>>>>>>>>>>> Options Reconfigured:
>>>>>>>>>>>>>>> *features.quota: on**
>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If quotad is not
>>>>>>>>>>>>>>> running, can you please
>>>>>>>>>>>>>>> provide glusterd logs
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 'usr-local-etc-glusterfs-glusterd.vol.log'.
>>>>>>>>>>>>>>> I will check is there
>>>>>>>>>>>>>>> are any issues starting
>>>>>>>>>>>>>>> quotad.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Vihay
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Monday 23 March 2015
>>>>>>>>>>>>>>> 11:54 AM, K.Kondo wrote:
>>>>>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>>>>> I could not find
>>>>>>>>>>>>>>>> the"quotad.log" in
>>>>>>>>>>>>>>>> directory
>>>>>>>>>>>>>>>> /var/log/glusterfs in
>>>>>>>>>>>>>>>> both servers and
>>>>>>>>>>>>>>>> client. But other test
>>>>>>>>>>>>>>>> server has the log.
>>>>>>>>>>>>>>>> Do you know why there
>>>>>>>>>>>>>>>> is no the file?
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Kondo
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2015/03/23 13:41、
>>>>>>>>>>>>>>>> Vijaikumar M
>>>>>>>>>>>>>>>> <vmallika at redhat.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <mailto:vmallika at redhat.com>>
>>>>>>>>>>>>>>>> のメッセージ:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Kondo,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> log file 'quotad.log'
>>>>>>>>>>>>>>>>> is missing in the
>>>>>>>>>>>>>>>>> attachment.Can you
>>>>>>>>>>>>>>>>> provide this log file
>>>>>>>>>>>>>>>>> as well?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Monday 23 March
>>>>>>>>>>>>>>>>> 2015 09:50 AM, kenji
>>>>>>>>>>>>>>>>> kondo wrote:
>>>>>>>>>>>>>>>>>> Hi Vijay,
>>>>>>>>>>>>>>>>>> Could you find the
>>>>>>>>>>>>>>>>>> attached?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>> Kondo
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2015-03-23 12:53
>>>>>>>>>>>>>>>>>> GMT+09:00 Vijaikumar
>>>>>>>>>>>>>>>>>> M
>>>>>>>>>>>>>>>>>> <vmallika at redhat.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> <mailto:vmallika at redhat.com>>:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Kondo,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can you please
>>>>>>>>>>>>>>>>>> provide below
>>>>>>>>>>>>>>>>>> mentioned
>>>>>>>>>>>>>>>>>> gluterfs logs?
>>>>>>>>>>>>>>>>>> client logs
>>>>>>>>>>>>>>>>>> (name of this
>>>>>>>>>>>>>>>>>> log will be
>>>>>>>>>>>>>>>>>> prefixed with
>>>>>>>>>>>>>>>>>> mount-point
>>>>>>>>>>>>>>>>>> dirname)
>>>>>>>>>>>>>>>>>> brick logs
>>>>>>>>>>>>>>>>>> quotad logs
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Friday 20
>>>>>>>>>>>>>>>>>> March 2015 06:31
>>>>>>>>>>>>>>>>>> PM, kenji kondo
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> Hi, Vijay and
>>>>>>>>>>>>>>>>>>> Peter
>>>>>>>>>>>>>>>>>>> Thanks for your
>>>>>>>>>>>>>>>>>>> reply.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I create new
>>>>>>>>>>>>>>>>>>> volume
>>>>>>>>>>>>>>>>>>> "testvol" with
>>>>>>>>>>>>>>>>>>> two bricks and
>>>>>>>>>>>>>>>>>>> set quota to
>>>>>>>>>>>>>>>>>>> simplify this
>>>>>>>>>>>>>>>>>>> problem.
>>>>>>>>>>>>>>>>>>> I got the
>>>>>>>>>>>>>>>>>>> glusterfs log
>>>>>>>>>>>>>>>>>>> as following
>>>>>>>>>>>>>>>>>>> after try to
>>>>>>>>>>>>>>>>>>> create a
>>>>>>>>>>>>>>>>>>> directory and
>>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>> BTW, my
>>>>>>>>>>>>>>>>>>> glusterd was
>>>>>>>>>>>>>>>>>>> upgraded from
>>>>>>>>>>>>>>>>>>> older version,
>>>>>>>>>>>>>>>>>>> although I
>>>>>>>>>>>>>>>>>>> don't know
>>>>>>>>>>>>>>>>>>> related to it.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>> Kondo
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.931016]
>> I
>>>>>>>>>>>>>>>>>>> [MSGID: 100030]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [glusterfsd.c:1998:main]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-/usr/sbin/glusterfs:
>>>>>>>>>>>>>>>>>>> Started running
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> /usr/sbin/glusterfs
>>>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>> 3.6.0.29 (args:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> /usr/sbin/glusterfs
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --volfile-server=gluster10
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --volfile-id=testvol
>>>>>>>>>>>>>>>>>>> testvol)
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.944850]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [dht-shared.c:337:dht_init_regex]
>>>>>>>>>>>>>>>>>>> 0-testvol-dht:
>>>>>>>>>>>>>>>>>>> using regex
>>>>>>>>>>>>>>>>>>> rsync-hash-regex
>> =
>>>>>>>>>>>>>>>>>>> ^\.(.+)\.[^.]+$
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.946256]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client.c:2280:notify]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-0:
>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>> translators are
>>>>>>>>>>>>>>>>>>> ready,
>>>>>>>>>>>>>>>>>>> attempting
>>>>>>>>>>>>>>>>>>> connect on
>>>>>>>>>>>>>>>>>>> transport
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.950674]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client.c:2280:notify]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-1:
>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>> translators are
>>>>>>>>>>>>>>>>>>> ready,
>>>>>>>>>>>>>>>>>>> attempting
>>>>>>>>>>>>>>>>>>> connect on
>>>>>>>>>>>>>>>>>>> transport
>>>>>>>>>>>>>>>>>>> Final graph:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>> +------------------------------------------------------------------------------+
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1: volume
>>>>>>>>>>>>>>>>>>> testvol-client-0
>>>>>>>>>>>>>>>>>>> 2: type
>>>>>>>>>>>>>>>>>>> protocol/client
>>>>>>>>>>>>>>>>>>> 3: option
>>>>>>>>>>>>>>>>>>> ping-timeout 42
>>>>>>>>>>>>>>>>>>> 4: option
>>>>>>>>>>>>>>>>>>> remote-host
>>>>>>>>>>>>>>>>>>> gluster24
>>>>>>>>>>>>>>>>>>> 5: option
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> remote-subvolume /export25/brick
>>>>>>>>>>>>>>>>>>> 6: option
>>>>>>>>>>>>>>>>>>> transport-type
>>>>>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>>>> 7: option
>>>>>>>>>>>>>>>>>>> send-gids true
>>>>>>>>>>>>>>>>>>> 8: end-volume
>>>>>>>>>>>>>>>>>>> 9:
>>>>>>>>>>>>>>>>>>> 10: volume
>>>>>>>>>>>>>>>>>>> testvol-client-1
>>>>>>>>>>>>>>>>>>> 11: type
>>>>>>>>>>>>>>>>>>> protocol/client
>>>>>>>>>>>>>>>>>>> 12: option
>>>>>>>>>>>>>>>>>>> ping-timeout 42
>>>>>>>>>>>>>>>>>>> 13: option
>>>>>>>>>>>>>>>>>>> remote-host
>>>>>>>>>>>>>>>>>>> gluster25
>>>>>>>>>>>>>>>>>>> 14: option
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> remote-subvolume /export25/brick
>>>>>>>>>>>>>>>>>>> 15: option
>>>>>>>>>>>>>>>>>>> transport-type
>>>>>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>>>> 16: option
>>>>>>>>>>>>>>>>>>> send-gids true
>>>>>>>>>>>>>>>>>>> 17: end-volume
>>>>>>>>>>>>>>>>>>> 18:
>>>>>>>>>>>>>>>>>>> 19: volume
>>>>>>>>>>>>>>>>>>> testvol-dht
>>>>>>>>>>>>>>>>>>> 20: type
>>>>>>>>>>>>>>>>>>>
>> cluster/distribute
>>>>>>>>>>>>>>>>>>> 21: subvolumes
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> testvol-client-0 testvol-client-1
>>>>>>>>>>>>>>>>>>> 22: end-volume
>>>>>>>>>>>>>>>>>>> 23:
>>>>>>>>>>>>>>>>>>> 24: volume
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> testvol-write-behind
>>>>>>>>>>>>>>>>>>> 25: type
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> performance/write-behind
>>>>>>>>>>>>>>>>>>> 26: subvolumes
>>>>>>>>>>>>>>>>>>> testvol-dht
>>>>>>>>>>>>>>>>>>> 27: end-volume
>>>>>>>>>>>>>>>>>>> 28:
>>>>>>>>>>>>>>>>>>> 29: volume
>>>>>>>>>>>>>>>>>>>
>> testvol-read-ahead
>>>>>>>>>>>>>>>>>>> 30: type
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> performance/read-ahead
>>>>>>>>>>>>>>>>>>> 31: subvolumes
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> testvol-write-behind
>>>>>>>>>>>>>>>>>>> 32: end-volume
>>>>>>>>>>>>>>>>>>> 33:
>>>>>>>>>>>>>>>>>>> 34: volume
>>>>>>>>>>>>>>>>>>> testvol-io-cache
>>>>>>>>>>>>>>>>>>> 35: type
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> performance/io-cache
>>>>>>>>>>>>>>>>>>> 36: subvolumes
>>>>>>>>>>>>>>>>>>>
>> testvol-read-ahead
>>>>>>>>>>>>>>>>>>> 37: end-volume
>>>>>>>>>>>>>>>>>>> 38:
>>>>>>>>>>>>>>>>>>> 39: volume
>>>>>>>>>>>>>>>>>>>
>> testvol-quick-read
>>>>>>>>>>>>>>>>>>> 40: type
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> performance/quick-read
>>>>>>>>>>>>>>>>>>> 41: subvolumes
>>>>>>>>>>>>>>>>>>> testvol-io-cache
>>>>>>>>>>>>>>>>>>> 42: end-volume
>>>>>>>>>>>>>>>>>>> 43:
>>>>>>>>>>>>>>>>>>> 44: volume
>>>>>>>>>>>>>>>>>>> testvol-md-cache
>>>>>>>>>>>>>>>>>>> 45: type
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> performance/md-cache
>>>>>>>>>>>>>>>>>>> 46: subvolumes
>>>>>>>>>>>>>>>>>>>
>> testvol-quick-read
>>>>>>>>>>>>>>>>>>> 47: end-volume
>>>>>>>>>>>>>>>>>>> 48:
>>>>>>>>>>>>>>>>>>> 49: volume
>>>>>>>>>>>>>>>>>>> testvol
>>>>>>>>>>>>>>>>>>> 50: type
>>>>>>>>>>>>>>>>>>> debug/io-stats
>>>>>>>>>>>>>>>>>>> 51: option
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> latency-measurement
>>>>>>>>>>>>>>>>>>> off
>>>>>>>>>>>>>>>>>>> 52: option
>>>>>>>>>>>>>>>>>>> count-fop-hits
>> off
>>>>>>>>>>>>>>>>>>> 53: subvolumes
>>>>>>>>>>>>>>>>>>> testvol-md-cache
>>>>>>>>>>>>>>>>>>> 54: end-volume
>>>>>>>>>>>>>>>>>>> 55:
>>>>>>>>>>>>>>>>>>> 56: volume
>>>>>>>>>>>>>>>>>>> meta-autoload
>>>>>>>>>>>>>>>>>>> 57: type meta
>>>>>>>>>>>>>>>>>>> 58: subvolumes
>>>>>>>>>>>>>>>>>>> testvol
>>>>>>>>>>>>>>>>>>> 59: end-volume
>>>>>>>>>>>>>>>>>>> 60:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>> +------------------------------------------------------------------------------+
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.955337]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:1759:rpc_clnt_reconfig]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-0:
>>>>>>>>>>>>>>>>>>> changing port
>>>>>>>>>>>>>>>>>>> to 49155 (from 0)
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.957549]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:1759:rpc_clnt_reconfig]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-1:
>>>>>>>>>>>>>>>>>>> changing port
>>>>>>>>>>>>>>>>>>> to 49155 (from 0)
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.959889]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>> [client-handshake.c:1415:select_server_supported_programs]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-0:
>>>>>>>>>>>>>>>>>>> Using Program
>>>>>>>>>>>>>>>>>>> GlusterFS 3.3,
>>>>>>>>>>>>>>>>>>> Num (1298437),
>>>>>>>>>>>>>>>>>>> Version (330)
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.960090]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>> [client-handshake.c:1415:select_server_supported_programs]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-1:
>>>>>>>>>>>>>>>>>>> Using Program
>>>>>>>>>>>>>>>>>>> GlusterFS 3.3,
>>>>>>>>>>>>>>>>>>> Num (1298437),
>>>>>>>>>>>>>>>>>>> Version (330)
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.960376]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:1200:client_setvolume_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-0:
>>>>>>>>>>>>>>>>>>> Connected to
>>>>>>>>>>>>>>>>>>> testvol-client-0,
>>>>>>>>>>>>>>>>>>> attached to
>>>>>>>>>>>>>>>>>>> remote volume
>>>>>>>>>>>>>>>>>>>
>> '/export25/brick'.
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.960405]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:1212:client_setvolume_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-0:
>>>>>>>>>>>>>>>>>>> Server and
>>>>>>>>>>>>>>>>>>> Client
>>>>>>>>>>>>>>>>>>> lk-version
>>>>>>>>>>>>>>>>>>> numbers are not
>>>>>>>>>>>>>>>>>>> same, reopening
>>>>>>>>>>>>>>>>>>> the fds
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.960471]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:1200:client_setvolume_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-1:
>>>>>>>>>>>>>>>>>>> Connected to
>>>>>>>>>>>>>>>>>>> testvol-client-1,
>>>>>>>>>>>>>>>>>>> attached to
>>>>>>>>>>>>>>>>>>> remote volume
>>>>>>>>>>>>>>>>>>>
>> '/export25/brick'.
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.960478]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:1212:client_setvolume_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-1:
>>>>>>>>>>>>>>>>>>> Server and
>>>>>>>>>>>>>>>>>>> Client
>>>>>>>>>>>>>>>>>>> lk-version
>>>>>>>>>>>>>>>>>>> numbers are not
>>>>>>>>>>>>>>>>>>> same, reopening
>>>>>>>>>>>>>>>>>>> the fds
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.962288]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [fuse-bridge.c:5042:fuse_graph_setup]
>>>>>>>>>>>>>>>>>>> 0-fuse:
>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>> graph 0
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.962351]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:188:client_set_lk_version_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-1:
>>>>>>>>>>>>>>>>>>> Server lk
>>>>>>>>>>>>>>>>>>> version = 1
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.962362]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:188:client_set_lk_version_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-0:
>>>>>>>>>>>>>>>>>>> Server lk
>>>>>>>>>>>>>>>>>>> version = 1
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:42:52.962424]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [fuse-bridge.c:3971:fuse_init]
>>>>>>>>>>>>>>>>>>> 0-glusterfs-fuse:
>>>>>>>>>>>>>>>>>>> FUSE inited
>>>>>>>>>>>>>>>>>>> with protocol
>>>>>>>>>>>>>>>>>>> versions:
>>>>>>>>>>>>>>>>>>> glusterfs 7.22
>>>>>>>>>>>>>>>>>>> kernel 7.14
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:13.352234]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [glusterfsd-mgmt.c:56:mgmt_cbk_spec]
>>>>>>>>>>>>>>>>>>> 0-mgmt: Volume
>>>>>>>>>>>>>>>>>>> file changed
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.518667]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [dht-shared.c:337:dht_init_regex]
>>>>>>>>>>>>>>>>>>> 2-testvol-dht:
>>>>>>>>>>>>>>>>>>> using regex
>>>>>>>>>>>>>>>>>>> rsync-hash-regex
>> =
>>>>>>>>>>>>>>>>>>> ^\.(.+)\.[^.]+$
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.520034]
>> W
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [graph.c:344:_log_if_unknown_option]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-quota: option
>>>>>>>>>>>>>>>>>>> 'timeout' is
>>>>>>>>>>>>>>>>>>> not recognized
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.520091]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client.c:2280:notify]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-0:
>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>> translators are
>>>>>>>>>>>>>>>>>>> ready,
>>>>>>>>>>>>>>>>>>> attempting
>>>>>>>>>>>>>>>>>>> connect on
>>>>>>>>>>>>>>>>>>> transport
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.524546]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client.c:2280:notify]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-1:
>>>>>>>>>>>>>>>>>>> parent
>>>>>>>>>>>>>>>>>>> translators are
>>>>>>>>>>>>>>>>>>> ready,
>>>>>>>>>>>>>>>>>>> attempting
>>>>>>>>>>>>>>>>>>> connect on
>>>>>>>>>>>>>>>>>>> transport
>>>>>>>>>>>>>>>>>>> Final graph:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>> +------------------------------------------------------------------------------+
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1: volume
>>>>>>>>>>>>>>>>>>> testvol-client-0
>>>>>>>>>>>>>>>>>>> 2: type
>>>>>>>>>>>>>>>>>>> protocol/client
>>>>>>>>>>>>>>>>>>> 3: option
>>>>>>>>>>>>>>>>>>> ping-timeout 42
>>>>>>>>>>>>>>>>>>> 4: option
>>>>>>>>>>>>>>>>>>> remote-host
>>>>>>>>>>>>>>>>>>> gluster24
>>>>>>>>>>>>>>>>>>> 5: option
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> remote-subvolume /export25/brick
>>>>>>>>>>>>>>>>>>> 6: option
>>>>>>>>>>>>>>>>>>> transport-type
>>>>>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>>>> 7: option
>>>>>>>>>>>>>>>>>>> send-gids true
>>>>>>>>>>>>>>>>>>> 8: end-volume
>>>>>>>>>>>>>>>>>>> 9:
>>>>>>>>>>>>>>>>>>> 10: volume
>>>>>>>>>>>>>>>>>>> testvol-client-1
>>>>>>>>>>>>>>>>>>> 11: type
>>>>>>>>>>>>>>>>>>> protocol/client
>>>>>>>>>>>>>>>>>>> 12: option
>>>>>>>>>>>>>>>>>>> ping-timeout 42
>>>>>>>>>>>>>>>>>>> 13: option
>>>>>>>>>>>>>>>>>>> remote-host
>>>>>>>>>>>>>>>>>>> gluster25
>>>>>>>>>>>>>>>>>>> 14: option
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> remote-subvolume /export25/brick
>>>>>>>>>>>>>>>>>>> 15: option
>>>>>>>>>>>>>>>>>>> transport-type
>>>>>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>>>> 16: option
>>>>>>>>>>>>>>>>>>> send-gids true
>>>>>>>>>>>>>>>>>>> 17: end-volume
>>>>>>>>>>>>>>>>>>> 18:
>>>>>>>>>>>>>>>>>>> 19: volume
>>>>>>>>>>>>>>>>>>> testvol-dht
>>>>>>>>>>>>>>>>>>> 20: type
>>>>>>>>>>>>>>>>>>>
>> cluster/distribute
>>>>>>>>>>>>>>>>>>> 21: subvolumes
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> testvol-client-0 testvol-client-1
>>>>>>>>>>>>>>>>>>> 22: end-volume
>>>>>>>>>>>>>>>>>>> 23:
>>>>>>>>>>>>>>>>>>> 24: volume
>>>>>>>>>>>>>>>>>>> testvol-quota
>>>>>>>>>>>>>>>>>>> 25: type
>>>>>>>>>>>>>>>>>>> features/quota
>>>>>>>>>>>>>>>>>>> 26: option
>>>>>>>>>>>>>>>>>>> timeout 0
>>>>>>>>>>>>>>>>>>> 27: option
>>>>>>>>>>>>>>>>>>> deem-statfs off
>>>>>>>>>>>>>>>>>>> 28: subvolumes
>>>>>>>>>>>>>>>>>>> testvol-dht
>>>>>>>>>>>>>>>>>>> 29: end-volume
>>>>>>>>>>>>>>>>>>> 30:
>>>>>>>>>>>>>>>>>>> 31: volume
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> testvol-write-behind
>>>>>>>>>>>>>>>>>>> 32: type
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> performance/write-behind
>>>>>>>>>>>>>>>>>>> 33: subvolumes
>>>>>>>>>>>>>>>>>>> testvol-quota
>>>>>>>>>>>>>>>>>>> 34: end-volume
>>>>>>>>>>>>>>>>>>> 35:
>>>>>>>>>>>>>>>>>>> 36: volume
>>>>>>>>>>>>>>>>>>>
>> testvol-read-ahead
>>>>>>>>>>>>>>>>>>> 37: type
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> performance/read-ahead
>>>>>>>>>>>>>>>>>>> 38: subvolumes
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> testvol-write-behind
>>>>>>>>>>>>>>>>>>> 39: end-volume
>>>>>>>>>>>>>>>>>>> 40:
>>>>>>>>>>>>>>>>>>> 41: volume
>>>>>>>>>>>>>>>>>>> testvol-io-cache
>>>>>>>>>>>>>>>>>>> 42: type
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> performance/io-cache
>>>>>>>>>>>>>>>>>>> 43: subvolumes
>>>>>>>>>>>>>>>>>>>
>> testvol-read-ahead
>>>>>>>>>>>>>>>>>>> 44: end-volume
>>>>>>>>>>>>>>>>>>> 45:
>>>>>>>>>>>>>>>>>>> 46: volume
>>>>>>>>>>>>>>>>>>>
>> testvol-quick-read
>>>>>>>>>>>>>>>>>>> 47: type
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> performance/quick-read
>>>>>>>>>>>>>>>>>>> 48: subvolumes
>>>>>>>>>>>>>>>>>>> testvol-io-cache
>>>>>>>>>>>>>>>>>>> 49: end-volume
>>>>>>>>>>>>>>>>>>> 50:
>>>>>>>>>>>>>>>>>>> 51: volume
>>>>>>>>>>>>>>>>>>> testvol-md-cache
>>>>>>>>>>>>>>>>>>> 52: type
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> performance/md-cache
>>>>>>>>>>>>>>>>>>> 53: subvolumes
>>>>>>>>>>>>>>>>>>>
>> testvol-quick-read
>>>>>>>>>>>>>>>>>>> 54: end-volume
>>>>>>>>>>>>>>>>>>> 55:
>>>>>>>>>>>>>>>>>>> 56: volume
>>>>>>>>>>>>>>>>>>> testvol
>>>>>>>>>>>>>>>>>>> 57: type
>>>>>>>>>>>>>>>>>>> debug/io-stats
>>>>>>>>>>>>>>>>>>> 58: option
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> latency-measurement
>>>>>>>>>>>>>>>>>>> off
>>>>>>>>>>>>>>>>>>> 59: option
>>>>>>>>>>>>>>>>>>> count-fop-hits
>> off
>>>>>>>>>>>>>>>>>>> 60: subvolumes
>>>>>>>>>>>>>>>>>>> testvol-md-cache
>>>>>>>>>>>>>>>>>>> 61: end-volume
>>>>>>>>>>>>>>>>>>> 62:
>>>>>>>>>>>>>>>>>>> 63: volume
>>>>>>>>>>>>>>>>>>> meta-autoload
>>>>>>>>>>>>>>>>>>> 64: type meta
>>>>>>>>>>>>>>>>>>> 65: subvolumes
>>>>>>>>>>>>>>>>>>> testvol
>>>>>>>>>>>>>>>>>>> 66: end-volume
>>>>>>>>>>>>>>>>>>> 67:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>> +------------------------------------------------------------------------------+
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.530005]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:1759:rpc_clnt_reconfig]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-1:
>>>>>>>>>>>>>>>>>>> changing port
>>>>>>>>>>>>>>>>>>> to 49155 (from 0)
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.530047]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [rpc-clnt.c:1759:rpc_clnt_reconfig]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-0:
>>>>>>>>>>>>>>>>>>> changing port
>>>>>>>>>>>>>>>>>>> to 49155 (from 0)
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.539062]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>> [client-handshake.c:1415:select_server_supported_programs]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-1:
>>>>>>>>>>>>>>>>>>> Using Program
>>>>>>>>>>>>>>>>>>> GlusterFS 3.3,
>>>>>>>>>>>>>>>>>>> Num (1298437),
>>>>>>>>>>>>>>>>>>> Version (330)
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.539299]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>> [client-handshake.c:1415:select_server_supported_programs]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-0:
>>>>>>>>>>>>>>>>>>> Using Program
>>>>>>>>>>>>>>>>>>> GlusterFS 3.3,
>>>>>>>>>>>>>>>>>>> Num (1298437),
>>>>>>>>>>>>>>>>>>> Version (330)
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.539462]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:1200:client_setvolume_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-1:
>>>>>>>>>>>>>>>>>>> Connected to
>>>>>>>>>>>>>>>>>>> testvol-client-1,
>>>>>>>>>>>>>>>>>>> attached to
>>>>>>>>>>>>>>>>>>> remote volume
>>>>>>>>>>>>>>>>>>>
>> '/export25/brick'.
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.539485]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:1212:client_setvolume_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-1:
>>>>>>>>>>>>>>>>>>> Server and
>>>>>>>>>>>>>>>>>>> Client
>>>>>>>>>>>>>>>>>>> lk-version
>>>>>>>>>>>>>>>>>>> numbers are not
>>>>>>>>>>>>>>>>>>> same, reopening
>>>>>>>>>>>>>>>>>>> the fds
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.539729]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:1200:client_setvolume_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-0:
>>>>>>>>>>>>>>>>>>> Connected to
>>>>>>>>>>>>>>>>>>> testvol-client-0,
>>>>>>>>>>>>>>>>>>> attached to
>>>>>>>>>>>>>>>>>>> remote volume
>>>>>>>>>>>>>>>>>>>
>> '/export25/brick'.
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.539751]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:1212:client_setvolume_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-0:
>>>>>>>>>>>>>>>>>>> Server and
>>>>>>>>>>>>>>>>>>> Client
>>>>>>>>>>>>>>>>>>> lk-version
>>>>>>>>>>>>>>>>>>> numbers are not
>>>>>>>>>>>>>>>>>>> same, reopening
>>>>>>>>>>>>>>>>>>> the fds
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.542878]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [fuse-bridge.c:5042:fuse_graph_setup]
>>>>>>>>>>>>>>>>>>> 0-fuse:
>>>>>>>>>>>>>>>>>>> switched to
>>>>>>>>>>>>>>>>>>> graph 2
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.542959]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:188:client_set_lk_version_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-1:
>>>>>>>>>>>>>>>>>>> Server lk
>>>>>>>>>>>>>>>>>>> version = 1
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:47:15.542987]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-handshake.c:188:client_set_lk_version_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-0:
>>>>>>>>>>>>>>>>>>> Server lk
>>>>>>>>>>>>>>>>>>> version = 1
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:48:04.586291]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client.c:2289:notify]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-0:
>>>>>>>>>>>>>>>>>>> current graph
>>>>>>>>>>>>>>>>>>> is no longer
>>>>>>>>>>>>>>>>>>> active,
>>>>>>>>>>>>>>>>>>> destroying
>>>>>>>>>>>>>>>>>>> rpc_client
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:48:04.586360]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client.c:2289:notify]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-1:
>>>>>>>>>>>>>>>>>>> current graph
>>>>>>>>>>>>>>>>>>> is no longer
>>>>>>>>>>>>>>>>>>> active,
>>>>>>>>>>>>>>>>>>> destroying
>>>>>>>>>>>>>>>>>>> rpc_client
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:48:04.586378]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client.c:2215:client_rpc_notify]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-0:
>>>>>>>>>>>>>>>>>>> disconnected
>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>> testvol-client-0.
>>>>>>>>>>>>>>>>>>> Client process
>>>>>>>>>>>>>>>>>>> will keep
>>>>>>>>>>>>>>>>>>> trying to
>>>>>>>>>>>>>>>>>>> connect to
>>>>>>>>>>>>>>>>>>> glusterd until
>>>>>>>>>>>>>>>>>>> brick's port is
>>>>>>>>>>>>>>>>>>> available
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:48:04.586430]
>> I
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client.c:2215:client_rpc_notify]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 0-testvol-client-1:
>>>>>>>>>>>>>>>>>>> disconnected
>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>> testvol-client-1.
>>>>>>>>>>>>>>>>>>> Client process
>>>>>>>>>>>>>>>>>>> will keep
>>>>>>>>>>>>>>>>>>> trying to
>>>>>>>>>>>>>>>>>>> connect to
>>>>>>>>>>>>>>>>>>> glusterd until
>>>>>>>>>>>>>>>>>>> brick's port is
>>>>>>>>>>>>>>>>>>> available
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:48:04.589552]
>> W
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-rpc-fops.c:306:client3_3_mkdir_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-0:
>>>>>>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>> operation
>>>>>>>>>>>>>>>>>>> failed:
>>>>>>>>>>>>>>>>>>> Transport
>>>>>>>>>>>>>>>>>>> endpoint is not
>>>>>>>>>>>>>>>>>>> connected.
>>>>>>>>>>>>>>>>>>> Path: /test/a
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:48:04.589608]
>> W
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [fuse-bridge.c:481:fuse_entry_cbk]
>>>>>>>>>>>>>>>>>>> 0-glusterfs-fuse:
>>>>>>>>>>>>>>>>>>> 78: MKDIR()
>>>>>>>>>>>>>>>>>>> /test/a => -1
>>>>>>>>>>>>>>>>>>> (Transport
>>>>>>>>>>>>>>>>>>> endpoint is not
>>>>>>>>>>>>>>>>>>> connected)
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:48:11.073349]
>> W
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [client-rpc-fops.c:2212:client3_3_create_cbk]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2-testvol-client-1:
>>>>>>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>> operation
>>>>>>>>>>>>>>>>>>> failed:
>>>>>>>>>>>>>>>>>>> Transport
>>>>>>>>>>>>>>>>>>> endpoint is not
>>>>>>>>>>>>>>>>>>> connected.
>>>>>>>>>>>>>>>>>>> Path: /test/f
>>>>>>>>>>>>>>>>>>> [2015-03-20
>>>>>>>>>>>>>>>>>>> 03:48:11.073419]
>> W
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [fuse-bridge.c:1937:fuse_create_cbk]
>>>>>>>>>>>>>>>>>>> 0-glusterfs-fuse:
>>>>>>>>>>>>>>>>>>> 82: /test/f =>
>>>>>>>>>>>>>>>>>>> -1 (Transport
>>>>>>>>>>>>>>>>>>> endpoint is not
>>>>>>>>>>>>>>>>>>> connected)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2015-03-20
>>>>>>>>>>>>>>>>>>> 11:27 GMT+09:00
>>>>>>>>>>>>>>>>>>> Vijaikumar M
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> <vmallika at redhat.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> <mailto:vmallika at redhat.com>>:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Kondo,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Can you
>>>>>>>>>>>>>>>>>>> please
>>>>>>>>>>>>>>>>>>> provide all
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> glusterfs
>>>>>>>>>>>>>>>>>>> log files?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Vijay
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Friday
>>>>>>>>>>>>>>>>>>> 20 March
>>>>>>>>>>>>>>>>>>> 2015 07:33
>>>>>>>>>>>>>>>>>>> AM, K.Kondo
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>> experts
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I had a
>>>>>>>>>>>>>>>>>>>> trouble
>>>>>>>>>>>>>>>>>>>> about quota.
>>>>>>>>>>>>>>>>>>>> I set
>>>>>>>>>>>>>>>>>>>> quota to
>>>>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> distributed volume
>>>>>>>>>>>>>>>>>>>> "vol12" as
>>>>>>>>>>>>>>>>>>>> bellow.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> gluster>
>>>>>>>>>>>>>>>>>>>> volume
>>>>>>>>>>>>>>>>>>>> quota
>>>>>>>>>>>>>>>>>>>> vol12 enable
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> volume
>>>>>>>>>>>>>>>>>>>> quota :
>>>>>>>>>>>>>>>>>>>> success
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> gluster>
>>>>>>>>>>>>>>>>>>>> volume
>>>>>>>>>>>>>>>>>>>> quota
>>>>>>>>>>>>>>>>>>>> vol12
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> limit-usage /test
>>>>>>>>>>>>>>>>>>>> 10GB
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> volume
>>>>>>>>>>>>>>>>>>>> quota :
>>>>>>>>>>>>>>>>>>>> success
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> But I
>>>>>>>>>>>>>>>>>>>> couldn't
>>>>>>>>>>>>>>>>>>>> create a
>>>>>>>>>>>>>>>>>>>> file and
>>>>>>>>>>>>>>>>>>>> directory
>>>>>>>>>>>>>>>>>>>> with below
>>>>>>>>>>>>>>>>>>>> error
>>>>>>>>>>>>>>>>>>>> message.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In a
>>>>>>>>>>>>>>>>>>>> client host,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> $cd test
>>>>>>>>>>>>>>>>>>>> (mounted
>>>>>>>>>>>>>>>>>>>> using fuse)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> $mkdir a
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> mkdir:
>>>>>>>>>>>>>>>>>>>> cannot
>>>>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>>>> directory
>>>>>>>>>>>>>>>>>>>> `a':
>>>>>>>>>>>>>>>>>>>> Transport
>>>>>>>>>>>>>>>>>>>> endpoint
>>>>>>>>>>>>>>>>>>>> is not
>>>>>>>>>>>>>>>>>>>> connected
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>> Additionally,
>>>>>>>>>>>>>>>>>>>> I couldn't
>>>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>> quota
>>>>>>>>>>>>>>>>>>>> status
>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>> gluster
>>>>>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> gluster>
>>>>>>>>>>>>>>>>>>>> volume
>>>>>>>>>>>>>>>>>>>> quota
>>>>>>>>>>>>>>>>>>>> vol12 list
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Path
>>>>>>>>>>>>>>>>>>>> Hard-limit
>>>>>>>>>>>>>>>>>>>> Soft-limit
>>>>>>>>>>>>>>>>>>>> Used
>>>>>>>>>>>>>>>>>>>> Available
>>>>>>>>>>>>>>>>>>>> Soft-limit
>>>>>>>>>>>>>>>>>>>> exceeded?
>>>>>>>>>>>>>>>>>>>> Hard-limit
>>>>>>>>>>>>>>>>>>>> exceeded?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>> ---------------------------------------------------------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Here,
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>> stops, so
>>>>>>>>>>>>>>>>>>>> I have to
>>>>>>>>>>>>>>>>>>>> do Ctrl-C.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Gluster
>>>>>>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>>>>> 3.6.1 and
>>>>>>>>>>>>>>>>>>>> 3.6.0.29
>>>>>>>>>>>>>>>>>>>> for server
>>>>>>>>>>>>>>>>>>>> and client
>>>>>>>>>>>>>>>>>>>>
>> respectively.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Any idea
>>>>>>>>>>>>>>>>>>>> for this?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best
>> regards,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> K. Kondo
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>>>>>>>>>>> <mailto:Gluster-users at gluster.org>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>> --
>> ~Atin
>>
>
--
~Atin
More information about the Gluster-users
mailing list