<div dir="ltr">Hi all:<div><br><div>I have a <span style="color:rgb(0,0,0);white-space:pre-wrap">replicated volume project2 with info:</span><div><font color="#000000"><span style="white-space:pre-wrap">Volume Name: project2
Type: Distributed-Replicate
Volume ID: 60175b8e-de0e-4409-81ae-7bb5eb5cacbf
Status: Started
Snapshot Count: 0
Number of Bricks: 84 x 2 = 168
Transport-type: tcp
Bricks:
Brick1: node20:/data2/bricks/project2
Brick2: node21:/data2/bricks/project2
Brick3: node22:/data2/bricks/project2
Brick4: node23:/data2/bricks/project2
Brick5: node24:/data2/bricks/project2
Brick6: node25:/data2/bricks/project2
Brick7: node26:/data2/bricks/project2
Brick8: node27:/data2/bricks/project2
Brick9: node28:/data2/bricks/project2
Brick10: node29:/data2/bricks/project2
Brick11: node30:/data2/bricks/project2
Brick12: node31:/data2/bricks/project2
Brick13: node32:/data2/bricks/project2
Brick14: node33:/data2/bricks/project2
Brick15: node20:/data3/bricks/project2
Brick16: node21:/data3/bricks/project2
Brick17: node22:/data3/bricks/project2
Brick18: node23:/data3/bricks/project2
Brick19: node24:/data3/bricks/project2
Brick20: node25:/data3/bricks/project2
Brick21: node26:/data3/bricks/project2
Brick22: node27:/data3/bricks/project2
Brick23: node28:/data3/bricks/project2
Brick24: node29:/data3/bricks/project2
Brick25: node30:/data3/bricks/project2
Brick26: node31:/data3/bricks/project2
Brick27: node32:/data3/bricks/project2
Brick28: node33:/data3/bricks/project2
Brick29: node20:/data4/bricks/project2
Brick30: node21:/data4/bricks/project2
Brick31: node22:/data4/bricks/project2
Brick32: node23:/data4/bricks/project2
Brick33: node24:/data4/bricks/project2
Brick34: node25:/data4/bricks/project2
Brick35: node26:/data4/bricks/project2
Brick36: node27:/data4/bricks/project2
Brick37: node28:/data4/bricks/project2
Brick38: node29:/data4/bricks/project2
Brick39: node30:/data4/bricks/project2
Brick40: node31:/data4/bricks/project2
Brick41: node32:/data4/bricks/project2
Brick42: node33:/data4/bricks/project2
Brick43: node20:/data5/bricks/project2
Brick44: node21:/data5/bricks/project2
Brick45: node22:/data5/bricks/project2
Brick46: node23:/data5/bricks/project2
Brick47: node24:/data5/bricks/project2
Brick48: node25:/data5/bricks/project2
Brick49: node26:/data5/bricks/project2
Brick50: node27:/data5/bricks/project2
Brick51: node28:/data5/bricks/project2
Brick52: node29:/data5/bricks/project2
Brick53: node30:/data5/bricks/project2
Brick54: node31:/data5/bricks/project2
Brick55: node32:/data5/bricks/project2
Brick56: node33:/data5/bricks/project2
Brick57: node20:/data6/bricks/project2
Brick58: node21:/data6/bricks/project2
Brick59: node22:/data6/bricks/project2
Brick60: node23:/data6/bricks/project2
Brick61: node24:/data6/bricks/project2
Brick62: node25:/data6/bricks/project2
Brick63: node26:/data6/bricks/project2
Brick64: node27:/data6/bricks/project2
Brick65: node28:/data6/bricks/project2
Brick66: node29:/data6/bricks/project2
Brick67: node30:/data6/bricks/project2
Brick68: node31:/data6/bricks/project2
Brick69: node32:/data6/bricks/project2
Brick70: node33:/data6/bricks/project2
Brick71: node20:/data7/bricks/project2
Brick72: node21:/data7/bricks/project2
Brick73: node22:/data7/bricks/project2
Brick74: node23:/data7/bricks/project2
Brick75: node24:/data7/bricks/project2
Brick76: node25:/data7/bricks/project2
Brick77: node26:/data7/bricks/project2
Brick78: node27:/data7/bricks/project2
Brick79: node28:/data7/bricks/project2
Brick80: node29:/data7/bricks/project2
Brick81: node30:/data7/bricks/project2
Brick82: node31:/data7/bricks/project2
Brick83: node32:/data7/bricks/project2
Brick84: node33:/data7/bricks/project2
Brick85: node20:/data8/bricks/project2
Brick86: node21:/data8/bricks/project2
Brick87: node22:/data8/bricks/project2
Brick88: node23:/data8/bricks/project2
Brick89: node24:/data8/bricks/project2
Brick90: node25:/data8/bricks/project2
Brick91: node26:/data8/bricks/project2
Brick92: node27:/data8/bricks/project2
Brick93: node28:/data8/bricks/project2
Brick94: node29:/data8/bricks/project2
Brick95: node30:/data8/bricks/project2
Brick96: node31:/data8/bricks/project2
Brick97: node32:/data8/bricks/project2
Brick98: node33:/data8/bricks/project2
Brick99: node20:/data9/bricks/project2
Brick100: node21:/data9/bricks/project2
Brick101: node22:/data9/bricks/project2
Brick102: node23:/data9/bricks/project2
Brick103: node24:/data9/bricks/project2
Brick104: node25:/data9/bricks/project2
Brick105: node26:/data9/bricks/project2
Brick106: node27:/data9/bricks/project2
Brick107: node28:/data9/bricks/project2
Brick108: node29:/data9/bricks/project2
Brick109: node30:/data9/bricks/project2
Brick110: node31:/data9/bricks/project2
Brick111: node32:/data9/bricks/project2
Brick112: node33:/data9/bricks/project2
Brick113: node20:/data10/bricks/project2
Brick114: node21:/data10/bricks/project2
Brick115: node22:/data10/bricks/project2
Brick116: node23:/data10/bricks/project2
Brick117: node24:/data10/bricks/project2
Brick118: node25:/data10/bricks/project2
Brick119: node26:/data10/bricks/project2
Brick120: node27:/data10/bricks/project2
Brick121: node28:/data10/bricks/project2
Brick122: node29:/data10/bricks/project2
Brick123: node30:/data10/bricks/project2
Brick124: node31:/data10/bricks/project2
Brick125: node32:/data10/bricks/project2
Brick126: node33:/data10/bricks/project2
Brick127: node20:/data11/bricks/project2
Brick128: node21:/data11/bricks/project2
Brick129: node22:/data11/bricks/project2
Brick130: node23:/data11/bricks/project2
Brick131: node24:/data11/bricks/project2
Brick132: node25:/data11/bricks/project2
Brick133: node26:/data11/bricks/project2
Brick134: node27:/data11/bricks/project2
Brick135: node28:/data11/bricks/project2
Brick136: node29:/data11/bricks/project2
Brick137: node30:/data11/bricks/project2
Brick138: node31:/data11/bricks/project2
Brick139: node32:/data11/bricks/project2
Brick140: node33:/data11/bricks/project2
Brick141: node20:/data12/bricks/project2
Brick142: node21:/data12/bricks/project2
Brick143: node22:/data12/bricks/project2
Brick144: node23:/data12/bricks/project2
Brick145: node24:/data12/bricks/project2
Brick146: node25:/data12/bricks/project2
Brick147: node26:/data12/bricks/project2
Brick148: node27:/data12/bricks/project2
Brick149: node28:/data12/bricks/project2
Brick150: node29:/data12/bricks/project2
Brick151: node30:/data12/bricks/project2
Brick152: node31:/data12/bricks/project2
Brick153: node32:/data12/bricks/project2
Brick154: node33:/data12/bricks/project2
Brick155: node20:/data13/bricks/project2
Brick156: node21:/data13/bricks/project2
Brick157: node22:/data13/bricks/project2
Brick158: node23:/data13/bricks/project2
Brick159: node24:/data13/bricks/project2
Brick160: node25:/data13/bricks/project2
Brick161: node26:/data13/bricks/project2
Brick162: node27:/data13/bricks/project2
Brick163: node28:/data13/bricks/project2
Brick164: node29:/data13/bricks/project2
Brick165: node30:/data13/bricks/project2
Brick166: node31:/data13/bricks/project2
Brick167: node32:/data13/bricks/project2
Brick168: node33:/data13/bricks/project2
Options Reconfigured:
performance.force-readdirp: on
performance.write-behind: off
performance.stat-prefetch: on
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
features.quota: on
features.inode-quota: on
features.quota-deem-statfs: on
cluster.readdir-optimize: on
cluster.lookup-optimize: on
dht.force-readdirp: off
client.event-threads: 10
server.event-threads: 10
performance.readdir-ahead: on
performance.io-cache: on
performance.flush-behind: on
performance.cache-size: 5GB
performance.cache-max-file-size: 1MB
performance.write-behind-window-size: 10MB
performance.read-ahead: off
network.remote-dio: enable
performance.strict-o-direct: disable
performance.io-thread-count: 25</span></font><br></div><div><font color="#000000"><span style="white-space:pre-wrap"><br></span></font></div><div><br></div><div>the volume looks ok, and I mount this volume on my client machine:</div><div>mount -t glusterfs -o oom-score-adj=-999 -o direct-io-mode=disable -o use-readdirp=no node20:/project2 /mnt/project2<br></div><div><br></div><div>I have a directory in /mnt/project2/, but when I mv the directory to other dirs, files in the dir lost while tree or ls the dir, some files missing, my operations is list as below:</div><div><br></div><div><div>root@ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ mkdir test-dir</div><div>root@ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ tree</div><div>.</div><div>├── face_landmarks</div><div>│ └── alive</div><div>│ └── logs_20180823_28</div><div>│ ├── info_000000.out</div><div>│ ├── info_000001.out</div><div>│ ├── info_000002.out</div><div>│ ├── info_000003.out</div><div>│ ├── info_000004.out</div><div>│ ├── info_000005.out</div><div>│ ├── info_000006.out</div><div>│ ├── info_000007.out</div><div>│ ├── info_000008.out</div><div>│ ├── info_000009.out</div><div>│ ├── info_000010.out</div><div>│ ├── info_000011.out</div><div>│ ├── info_000012.out</div><div>│ ├── info_000013.out</div><div>│ ├── info_000014.out</div><div>│ ├── info_000015.out</div><div>│ ├── info_000016.out</div><div>│ ├── info_000017.out</div><div>│ ├── info_000018.out</div><div>│ └── info_000019.out</div><div>└── test-dir</div><div><br></div><div>4 directories, 20 files</div><div>root@ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ mv face_landmarks/ test-dir/</div><div>root@ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ tree</div><div>.</div><div>└── test-dir</div><div> └── face_landmarks</div><div><br></div><div>2 directories, 0 files</div><div>root@ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ cd test-dir/face_landmarks/</div><div>root@ml-gpu-ser129.nmg01:/mnt/project2/371_37829/test-dir/face_landmarks$ ls</div><div>root@ml-gpu-ser129.nmg01:/mnt/project2/371_37829/test-dir/face_landmarks$ cd ..</div><div>root@ml-gpu-ser129.nmg01:/mnt/project2/371_37829/test-dir$ mv face_landmarks/ ..</div><div>root@ml-gpu-ser129.nmg01:/mnt/project2/371_37829/test-dir$ cd ..</div><div>root@ml-gpu-ser129.nmg01:/mnt/project2/371_37829$ tree</div><div>.</div><div>├── face_landmarks</div><div>│ └── alive</div><div>└── test-dir</div><div><br></div><div>3 directories, 0 files</div><div>root@ml-gpu-ser129.nmg01:/mnt/project2/371_37829$</div></div><div><br></div><div>I think i make some mistakes with volume option, buti i am not sure, so how can i find the lost files? the files seems still int the directory, because i cant remove the directory and rm tell me "Not empty directory"</div><div><br></div><div><br></div><div>Any suggestions is appreciated. </div><div>Many Thanks</div></div></div><div><br></div><div>Best regards</div><div>Yu</div></div>