[Gluster-users] [External] Re: A broken file that can not be deleted

[Dmitry Isakbayev]

Nithyya, Raghavendra, Davide

Thank you for your help.


*>how the permissions are displayed for the file on the servers?*
Trying to answer this question is what fixed the problem.  It looked just
fine on all 3 servers.  And it looks like running "ls" on the servers fixed
it on the clients.  I had to repeat it on all 3 servers.
It fixed file permissions on 2 clients and made the file show up on the 3rd
client.

Even though it fixed the file permissions and I could now view contents of
the file, the software was still having issues with renaming
".download_suspensions.memo.writing"  to  ".download_suspensions.memo"
When I tried to replace the file manually, I got
$ mv .download_suspensions.memo.writing .download_suspensions.memo
mv: ‘.download_suspensions.memo.writing’ and ‘.download_suspensions.memo’
are the same file

I ended up removing both files and having the software rebuild them.

>
*Wondering whether its a case of split brain.*Very possible.  All 3 servers
were rebooted.  It brought down linux cluster running on the same 3 servers
as well.



> On Thu, Jan 10, 2019 at 10:00 AM Raghavendra Gowdappa <rgowdapp at redhat.com>
> wrote:
>
>>
>>
>> On Wed, Jan 9, 2019 at 7:48 PM Dmitry Isakbayev <isakdim at gmail.com>
>> wrote:
>>
>>> I am seeing a broken file that exists on 2 out of 3 nodes.
>>>
>>
>> Wondering whether its a case of split brain.
>>
>>
>>> The application trying to use the file throws file permissions error.
>>> ls, rm, mv, touch all throw "Input/output error"
>>>
>>> $ ls -la
>>> ls: cannot access .download_suspensions.memo: Input/output error
>>> drwxrwxr-x. 2 ossadmin ossadmin  4096 Jan  9 08:06 .
>>> drwxrwxr-x. 5 ossadmin ossadmin  4096 Jan  3 11:36 ..
>>> -?????????? ? ?        ?            ?            ?
>>> .download_suspensions.memo
>>>
>>> $ rm ".download_suspensions.memo"
>>> rm: cannot remove ‘.download_suspensions.memo’: Input/output error
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> --
> Davide Obbi
> Senior System Administrator
>
> Booking.com B.V.
> Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
> Direct +31207031558
> [image: Booking.com] <https://www.booking.com/>
> Empowering people to experience the world since 1996
> 43 languages, 214+ offices worldwide, 141,000+ global destinations, 29
> million reported listings
> Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190110/92c46f4e/attachment.html>