[Gluster-users] mv lost some files ?
Raghavendra Gowdappa
rgowdapp at redhat.com
Wed Sep 5 02:50:45 UTC 2018
On Tue, Sep 4, 2018 at 5:28 PM, yu sun <sunyu1949 at gmail.com> wrote:
> Hi all:
>
> I have a replicated volume project2 with info:
> Volume Name: project2 Type: Distributed-Replicate Volume ID:
> 60175b8e-de0e-4409-81ae-7bb5eb5cacbf Status: Started Snapshot Count: 0
> Number of Bricks: 84 x 2 = 168 Transport-type: tcp Bricks: Brick1:
> node20:/data2/bricks/project2 Brick2: node21:/data2/bricks/project2 Brick3:
> node22:/data2/bricks/project2 Brick4: node23:/data2/bricks/project2 Brick5:
> node24:/data2/bricks/project2 Brick6: node25:/data2/bricks/project2 Brick7:
> node26:/data2/bricks/project2 Brick8: node27:/data2/bricks/project2 Brick9:
> node28:/data2/bricks/project2 Brick10: node29:/data2/bricks/project2
> Brick11: node30:/data2/bricks/project2 Brick12:
> node31:/data2/bricks/project2 Brick13: node32:/data2/bricks/project2
> Brick14: node33:/data2/bricks/project2 Brick15:
> node20:/data3/bricks/project2 Brick16: node21:/data3/bricks/project2
> Brick17: node22:/data3/bricks/project2 Brick18:
> node23:/data3/bricks/project2 Brick19: node24:/data3/bricks/project2
> Brick20: node25:/data3/bricks/project2 Brick21:
> node26:/data3/bricks/project2 Brick22: node27:/data3/bricks/project2
> Brick23: node28:/data3/bricks/project2 Brick24:
> node29:/data3/bricks/project2 Brick25: node30:/data3/bricks/project2
> Brick26: node31:/data3/bricks/project2 Brick27:
> node32:/data3/bricks/project2 Brick28: node33:/data3/bricks/project2
> Brick29: node20:/data4/bricks/project2 Brick30:
> node21:/data4/bricks/project2 Brick31: node22:/data4/bricks/project2
> Brick32: node23:/data4/bricks/project2 Brick33:
> node24:/data4/bricks/project2 Brick34: node25:/data4/bricks/project2
> Brick35: node26:/data4/bricks/project2 Brick36:
> node27:/data4/bricks/project2 Brick37: node28:/data4/bricks/project2
> Brick38: node29:/data4/bricks/project2 Brick39:
> node30:/data4/bricks/project2 Brick40: node31:/data4/bricks/project2
> Brick41: node32:/data4/bricks/project2 Brick42:
> node33:/data4/bricks/project2 Brick43: node20:/data5/bricks/project2
> Brick44: node21:/data5/bricks/project2 Brick45:
> node22:/data5/bricks/project2 Brick46: node23:/data5/bricks/project2
> Brick47: node24:/data5/bricks/project2 Brick48:
> node25:/data5/bricks/project2 Brick49: node26:/data5/bricks/project2
> Brick50: node27:/data5/bricks/project2 Brick51:
> node28:/data5/bricks/project2 Brick52: node29:/data5/bricks/project2
> Brick53: node30:/data5/bricks/project2 Brick54:
> node31:/data5/bricks/project2 Brick55: node32:/data5/bricks/project2
> Brick56: node33:/data5/bricks/project2 Brick57:
> node20:/data6/bricks/project2 Brick58: node21:/data6/bricks/project2
> Brick59: node22:/data6/bricks/project2 Brick60:
> node23:/data6/bricks/project2 Brick61: node24:/data6/bricks/project2
> Brick62: node25:/data6/bricks/project2 Brick63:
> node26:/data6/bricks/project2 Brick64: node27:/data6/bricks/project2
> Brick65: node28:/data6/bricks/project2 Brick66:
> node29:/data6/bricks/project2 Brick67: node30:/data6/bricks/project2
> Brick68: node31:/data6/bricks/project2 Brick69:
> node32:/data6/bricks/project2 Brick70: node33:/data6/bricks/project2
> Brick71: node20:/data7/bricks/project2 Brick72:
> node21:/data7/bricks/project2 Brick73: node22:/data7/bricks/project2
> Brick74: node23:/data7/bricks/project2 Brick75:
> node24:/data7/bricks/project2 Brick76: node25:/data7/bricks/project2
> Brick77: node26:/data7/bricks/project2 Brick78:
> node27:/data7/bricks/project2 Brick79: node28:/data7/bricks/project2
> Brick80: node29:/data7/bricks/project2 Brick81:
> node30:/data7/bricks/project2 Brick82: node31:/data7/bricks/project2
> Brick83: node32:/data7/bricks/project2 Brick84:
> node33:/data7/bricks/project2 Brick85: node20:/data8/bricks/project2
> Brick86: node21:/data8/bricks/project2 Brick87:
> node22:/data8/bricks/project2 Brick88: node23:/data8/bricks/project2
> Brick89: node24:/data8/bricks/project2 Brick90:
> node25:/data8/bricks/project2 Brick91: node26:/data8/bricks/project2
> Brick92: node27:/data8/bricks/project2 Brick93:
> node28:/data8/bricks/project2 Brick94: node29:/data8/bricks/project2
> Brick95: node30:/data8/bricks/project2 Brick96:
> node31:/data8/bricks/project2 Brick97: node32:/data8/bricks/project2
> Brick98: node33:/data8/bricks/project2 Brick99:
> node20:/data9/bricks/project2 Brick100: node21:/data9/bricks/project2
> Brick101: node22:/data9/bricks/project2 Brick102:
> node23:/data9/bricks/project2 Brick103: node24:/data9/bricks/project2
> Brick104: node25:/data9/bricks/project2 Brick105:
> node26:/data9/bricks/project2 Brick106: node27:/data9/bricks/project2
> Brick107: node28:/data9/bricks/project2 Brick108:
> node29:/data9/bricks/project2 Brick109: node30:/data9/bricks/project2
> Brick110: node31:/data9/bricks/project2 Brick111:
> node32:/data9/bricks/project2 Brick112: node33:/data9/bricks/project2
> Brick113: node20:/data10/bricks/project2 Brick114:
> node21:/data10/bricks/project2 Brick115: node22:/data10/bricks/project2
> Brick116: node23:/data10/bricks/project2 Brick117:
> node24:/data10/bricks/project2 Brick118: node25:/data10/bricks/project2
> Brick119: node26:/data10/bricks/project2 Brick120:
> node27:/data10/bricks/project2 Brick121: node28:/data10/bricks/project2
> Brick122: node29:/data10/bricks/project2 Brick123:
> node30:/data10/bricks/project2 Brick124: node31:/data10/bricks/project2
> Brick125: node32:/data10/bricks/project2 Brick126:
> node33:/data10/bricks/project2 Brick127: node20:/data11/bricks/project2
> Brick128: node21:/data11/bricks/project2 Brick129:
> node22:/data11/bricks/project2 Brick130: node23:/data11/bricks/project2
> Brick131: node24:/data11/bricks/project2 Brick132:
> node25:/data11/bricks/project2 Brick133: node26:/data11/bricks/project2
> Brick134: node27:/data11/bricks/project2 Brick135:
> node28:/data11/bricks/project2 Brick136: node29:/data11/bricks/project2
> Brick137: node30:/data11/bricks/project2 Brick138:
> node31:/data11/bricks/project2 Brick139: node32:/data11/bricks/project2
> Brick140: node33:/data11/bricks/project2 Brick141:
> node20:/data12/bricks/project2 Brick142: node21:/data12/bricks/project2
> Brick143: node22:/data12/bricks/project2 Brick144:
> node23:/data12/bricks/project2 Brick145: node24:/data12/bricks/project2
> Brick146: node25:/data12/bricks/project2 Brick147:
> node26:/data12/bricks/project2 Brick148: node27:/data12/bricks/project2
> Brick149: node28:/data12/bricks/project2 Brick150:
> node29:/data12/bricks/project2 Brick151: node30:/data12/bricks/project2
> Brick152: node31:/data12/bricks/project2 Brick153:
> node32:/data12/bricks/project2 Brick154: node33:/data12/bricks/project2
> Brick155: node20:/data13/bricks/project2 Brick156:
> node21:/data13/bricks/project2 Brick157: node22:/data13/bricks/project2
> Brick158: node23:/data13/bricks/project2 Brick159:
> node24:/data13/bricks/project2 Brick160: node25:/data13/bricks/project2
> Brick161: node26:/data13/bricks/project2 Brick162:
> node27:/data13/bricks/project2 Brick163: node28:/data13/bricks/project2
> Brick164: node29:/data13/bricks/project2 Brick165:
> node30:/data13/bricks/project2 Brick166: node31:/data13/bricks/project2
> Brick167: node32:/data13/bricks/project2 Brick168:
> node33:/data13/bricks/project2 Options Reconfigured:
> performance.force-readdirp: on performance.write-behind: off
> performance.stat-prefetch: on performance.client-io-threads: on
> nfs.disable: on transport.address-family: inet features.quota: on
> features.inode-quota: on features.quota-deem-statfs: on
> cluster.readdir-optimize: on cluster.lookup-optimize: on
> dht.force-readdirp: off client.event-threads: 10 server.event-threads: 10
> performance.readdir-ahead: on performance.io-cache: on
> performance.flush-behind: on performance.cache-size: 5GB
> performance.cache-max-file-size: 1MB performance.write-behind-window-size:
> 10MB performance.read-ahead: off network.remote-dio: enable
> performance.strict-o-direct: disable performance.io-thread-count: 25
>
>
> the volume looks ok, and I mount this volume on my client machine:
> mount -t glusterfs -o oom-score-adj=-999 -o direct-io-mode=disable -o
> use-readdirp=no node20:/project2 /mnt/project2
>
> I have a directory in /mnt/project2/, but when I mv the directory to other
> dirs, files in the dir lost while tree or ls the dir, some files missing,
> my operations is list as below:
>
Looks very similar to:
https://bugzilla.redhat.com/show_bug.cgi?id=1118762
https://bugzilla.redhat.com/show_bug.cgi?id=1337394
> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ mkdir test-dir
> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ tree
> .
> ├── face_landmarks
> │ └── alive
> │ └── logs_20180823_28
> │ ├── info_000000.out
> │ ├── info_000001.out
> │ ├── info_000002.out
> │ ├── info_000003.out
> │ ├── info_000004.out
> │ ├── info_000005.out
> │ ├── info_000006.out
> │ ├── info_000007.out
> │ ├── info_000008.out
> │ ├── info_000009.out
> │ ├── info_000010.out
> │ ├── info_000011.out
> │ ├── info_000012.out
> │ ├── info_000013.out
> │ ├── info_000014.out
> │ ├── info_000015.out
> │ ├── info_000016.out
> │ ├── info_000017.out
> │ ├── info_000018.out
> │ └── info_000019.out
> └── test-dir
>
> 4 directories, 20 files
> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ mv face_landmarks/
> test-dir/
> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ tree
> .
> └── test-dir
> └── face_landmarks
>
> 2 directories, 0 files
> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ cd
> test-dir/face_landmarks/
> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829/test-dir/face_landmarks$
> ls
> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829/test-dir/face_landmarks$
> cd ..
> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829/test-dir$ mv
> face_landmarks/ ..
> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829/test-dir$ cd ..
> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ tree
> .
> ├── face_landmarks
> │ └── alive
> └── test-dir
>
> 3 directories, 0 files
> root at ml-gpu-ser129.nmg01:/mnt/project2/371_37829$
>
> I think i make some mistakes with volume option, buti i am not sure, so
> how can i find the lost files? the files seems still int the directory,
> because i cant remove the directory and rm tell me "Not empty directory"
>
Its likely that src and dst of mv having same gfid and that's causing the
issues. Can you look into both src and dst paths on all bricks? Union of
contents of both directories should give all the files in the src directory
before mv. Once found you can,
* keep a backup of contents of src and dst on all bricks
* remove trusted.gfid xattr on src and dst from all bricks
* remove gfid handle (.glusterfs/<first two characters of gfid>/<second set
of two characters of gfid>/<gfid> on each brick)
* disable readdirplus in entire stack (maybe you can use a tmp mount for
this) [1]
* stat src and dst on a mount point with readdirplus disabled.
* Now you'll see two directories src and dst on mountpoint. You can copy
the contents of both into a new directory
[1] https://lists.gluster.org/pipermail/gluster-users/2017-March/030148.html
>
> Any suggestions is appreciated.
> Many Thanks
>
> Best regards
> Yu
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180905/55f121f0/attachment.html>
More information about the Gluster-users
mailing list