[Gluster-users] Bareos backup from Gluster mount

Pranith Kumar Karampuri pkarampu at redhat.com
Thu Jul 30 08:30:26 UTC 2015 


On 07/30/2015 01:50 PM, Meghana Madhusudhan wrote:
> Hi Ryan,
>
> I am just trying understand the scenario better here.
> So all the files with that encountered "Permission denied" errors were "nfsnobody" files.
> Rest of the files with 602 permissions were backed up successfully from both the bricks.
> I see that there are no permissions for these "nobody" files. I'd expect "Permission denied"
> errors on such files. I am not too sure about how these files ended up with such permissions.
> Can you please attach nfs.log located at "/var/log/glusterfs" ?
Meghana,
       These files with no permissions are dht linkto files. Adding dht 
folks to see if they could help as well. As far as I remember the 
uid/gid of linkto files and normal files should match.

Pranith
>
>
> Meghana
>
> ----- Original Message -----
> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> To: "Ryan Clough" <ryan.clough at dsic.com>
> Cc: "gluster-users" <Gluster-users at gluster.org>, "Niels de Vos" <ndevos at redhat.com>, "Soumya Koduri" <skoduri at redhat.com>, "Jiffin Thottan" <jthottan at redhat.com>, "Meghana Madhusudhan" <mmadhusu at redhat.com>
> Sent: Thursday, July 30, 2015 1:07:53 PM
> Subject: Re: [Gluster-users] Bareos backup from Gluster mount
>
> Added folks who work on nfs on gluster. Let's see.
>
> Pranith
>
> On 07/30/2015 09:25 AM, Ryan Clough wrote:
>> Okay, I think this has to do with Gluster NFS even though I was not
>> accessing the Gluster volume via NFS.
>>
>> Directly on the bricks the files look like this:
>>
>> Hgluster01
>> -r--r--r--. 2       602 602 832M Apr  3 06:17 scan_89.tar.bak
>> ---------T  2 nfsnobody nfsnobody    0 Jul 13 11:42 scan_90.tar.bak
>> ---------T  2 nfsnobody nfsnobody    0 Jul 13 11:42 scan_91.tar.bak
>> ---------T  2 nfsnobody nfsnobody    0 Jul 13 11:42 scan_92.tar.bak
>> ---------T  2 nfsnobody nfsnobody    0 Jul 13 11:42 scan_94.tar.bak
>> ---------T  2 nfsnobody nfsnobody    0 Jul 13 11:42 scan_95.tar.bak
>> -r--r--r--. 2       602       602 839M Apr  3 11:39 scan_96.tar.bak
>> ---------T  2 nfsnobody nfsnobody    0 Jul 13 11:42 scan_98.tar.bak
>> ---------T  2 nfsnobody nfsnobody    0 Jul 13 11:42 scan_99.tar.bak
>>
>> Hgluster02
>> ---------T  2 nfsnobody nfsnobody           0 Jul 13 11:42 scan_89.tar.bak
>> -r--r--r--. 2       602       602   869939200 Apr  3 07:36 scan_90.tar.bak
>> -r--r--r--. 2       602       602   868331520 Apr  3 09:36 scan_91.tar.bak
>> -r--r--r--. 2       602       602   870092800 Apr  3 09:37 scan_92.tar.bak
>> -r--r--r--. 2       602       602   875448320 Apr  3 09:39 scan_93.tar.bak
>> -r--r--r--. 2       602       602   870656000 Apr  3 09:40 scan_94.tar.bak
>> -r--r--r--. 2       602       602   869396480 Apr  3 11:38 scan_95.tar.bak
>> -r--r--r--. 2       602       602   881858560 Apr  3 11:40 scan_97.tar.bak
>> -r--r--r--. 2       602       602   868188160 Apr  3 11:41 scan_98.tar.bak
>> -r--r--r--. 2       602       602   865382400 Apr  3 13:32 scan_99.tar.bak
>>
>> So I turned off NFS and from a client tried to move the files to see
>> if that would get rid of these weird nfsnobody files. Didn't work.
>> After moving the files to a new directory, the new directory still had
>> all of the nfsnobody files on the bricks.
>>
>> Next, I used rsync from the same client to copy all of the files to a
>> new directory and, lo and behold, the nfsnobody files were gone. I
>> tested a Bareos backup job and the data was read without issue from
>> both nodes. There was zero empty files.
>>
>> I guess I will blame Gluster NFS for this one? If you need any more
>> information from me, I would be happy to oblige.
>>
>> ___________________________________________
>> ¯\_(ツ)_/¯
>> Ryan Clough
>> Information Systems
>> Decision Sciences International Corporation
>> <http://www.decisionsciencescorp.com/><http://www.decisionsciencescorp.com/>
>>
>> On Wed, Jul 29, 2015 at 5:53 AM, Pranith Kumar Karampuri
>> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>>
>>      hi Ryan,
>>             What do you see in the logs of glusterfs mount and bricks?
>>      Do you think it is possible for you to attach those logs to this
>>      thread so that we can see what could be going on?
>>
>>      Pranith
>>
>>
>>      On 07/28/2015 02:32 AM, Ryan Clough wrote:
>>>      Hello,
>>>
>>>      I have cross-posted this question in the bareos-users mailing list.
>>>
>>>      Wondering if anyone has tried this because I am unable to backup
>>>      data that is mounted via Gluster Fuse or Gluster NFS. Basically,
>>>      I have the Gluster volume mounted on the Bareos Director which
>>>      also has the tape changer attached.
>>>
>>>      Here is some information about versions:
>>>      Bareos version 14.2.2
>>>      Gluster version 3.7.2
>>>      Scientific Linux version 6.6
>>>
>>>      Our Gluster volume consists of two nodes in distribute only. Here
>>>      is the configuration of our volume:
>>>      [root at hgluster02 ~]# gluster volume info
>>>
>>>      Volume Name: export_volume
>>>      Type: Distribute
>>>      Volume ID: c74cc970-31e2-4924-a244-4c70d958dadb
>>>      Status: Started
>>>      Number of Bricks: 2
>>>      Transport-type: tcp
>>>      Bricks:
>>>      Brick1: hgluster01:/gluster_data
>>>      Brick2: hgluster02:/gluster_data
>>>      Options Reconfigured:
>>>      performance.io-thread-count: 24
>>>      server.event-threads: 20
>>>      client.event-threads: 4
>>>      performance.readdir-ahead: on
>>>      features.inode-quota: on
>>>      features.quota: on
>>>      nfs.disable: off
>>>      auth.allow: 192.168.10.*,10.0.10.*,10.8.0.*,10.2.0.*,10.0.60.*
>>>      server.allow-insecure: on
>>>      server.root-squash: on
>>>      performance.read-ahead: on
>>>      features.quota-deem-statfs: on
>>>      diagnostics.brick-log-level: WARNING
>>>
>>>      When I try to backup a directory from Gluster Fuse or Gluster NFS
>>>      mount and I monitor the network communication I only see data
>>>      being pulled from the hgluster01 brick. When the job finishes
>>>      Bareos thinks that it completed without error but included in the
>>>      messages for the job are lots and lots of permission denied
>>>      errors like this:
>>>      15-Jul 02:03 ripper.red.dsic.com-fd JobId 613:      Cannot open
>>>      "/export/rclough/psdv-2014-archives-2/scan_111.tar.bak":
>>>      ERR=Permission denied.
>>>      15-Jul 02:03 ripper.red.dsic.com-fd JobId 613:      Cannot open
>>>      "/export/rclough/psdv-2014-archives-2/run_219.tar.bak":
>>>      ERR=Permission denied.
>>>      15-Jul 02:03 ripper.red.dsic.com-fd JobId 613:      Cannot open
>>>      "/export/rclough/psdv-2014-archives-2/scan_112.tar.bak":
>>>      ERR=Permission denied.
>>>      15-Jul 02:03 ripper.red.dsic.com-fd JobId 613:      Cannot open
>>>      "/export/rclough/psdv-2014-archives-2/run_220.tar.bak":
>>>      ERR=Permission denied.
>>>      15-Jul 02:03 ripper.red.dsic.com-fd JobId 613:      Cannot open
>>>      "/export/rclough/psdv-2014-archives-2/scan_114.tar.bak":
>>>      ERR=Permission denied.
>>>
>>>      At first I thought this might be a root-squash problem but, if I
>>>      try to read/copy a file using the root user from the Bareos
>>>      server that is trying to do the backup, I can read files just fine.
>>>
>>>      When the job finishes is reports that it finished "OK -- with
>>>      warnings" but, again the log for the job is filled with
>>>      "ERR=Permission denied" messages. In my opinion, this job did not
>>>      finish OK and should be Failed. Some of the files from the
>>>      HGluster02 brick are backed up but all of the ones with
>>>      permission errors do not. When I restore the job, all of the
>>>      files with permission errors are empty.
>>>
>>>      Has anyone successfully used Bareos to backup data from Gluster
>>>      mounts? This is an important use case for us because this is the
>>>      largest single volume that we have to prepare large amounts of
>>>      data to be archived.
>>>
>>>      Thank you for your time,
>>>      ___________________________________________
>>>      ¯\_(ツ)_/¯
>>>      Ryan Clough
>>>      Information Systems
>>>      Decision Sciences International Corporation
>>>      <http://www.decisionsciencescorp.com/><http://www.decisionsciencescorp.com/>
>>>
>>>      This email and its contents are confidential. If you are not the
>>>      intended recipient, please do not disclose or use the information
>>>      within this email or its attachments. If you have received this
>>>      email in error, please report the error to the sender by return
>>>      email and delete this communication from your records.
>>>
>>>
>>>      _______________________________________________
>>>      Gluster-users mailing list
>>>      Gluster-users at gluster.org  <mailto:Gluster-users at gluster.org>
>>>      http://www.gluster.org/mailman/listinfo/gluster-users
>>
>>
>> This email and its contents are confidential. If you are not the
>> intended recipient, please do not disclose or use the information
>> within this email or its attachments. If you have received this email
>> in error, please report the error to the sender by return email and
>> delete this communication from your records.

More information about the Gluster-users mailing list