[Gluster-users] Maildir issue.

[W K]

We brought up a test cluster to investigate GlusterFS.

Using the Quick Start instructions, we brought up a 2 server 1 brick 
replicating setup and mounted to it from a third box with the fuse mount 
(all ver 3.4.1)

# gluster volume info

Volume Name: mailtest
Type: Replicate
Volume ID: 9e412774-b8c9-4135-b7fb-bc0dd298d06a
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster2:/gluster1/mailtest/brick1
Brick2: gluster1:/gluster1/mailtest/brick1
Options Reconfigured:
server.root-squash: no

We then began loading the cluster with data (from a nearby Mailserver) 
by rsyncing Maildirs onto the mount. We chose maildirs because small 
files are supposed to be a worst case scenario in Gluster.

During our testing, everything worked great. Speed was acceptable (we 
are familiar with other Distributed File Systems, so we don't have 
unrealistic expectations).

We yanked cords, turned off machines and generally tortured the setup, 
all to good effect. Everything performs as advertised, though you have 
to do a LOT of googling to get some of the answers when 'recovering'. 
There appears to be a lot of 'secret' recipes to get things done and I 
think the doc site should link to JoeJulians blog, among others <grin>.

We then decided to see what email Maildir performance was like using a 
pop3 tester program.

So we quickly installed qmail which has the qmail-pop3d daemon.

The result was that the pop daemon cant see the email

# telnet localhost 110
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
+OK <20481.1385248027 at mailtest>
USER glusttest
+OK
PASS somepass
+OK
LIST
+OK
.

If we copy that SAME directory over to /users (not on the gluster 
mount), then the LIST command shows that email. So we know that the 
qmail-pop3d setup is working fine.

LIST
+OK
1 3120
2 11157
3 3267
etc.

So since we normally use POP/IMAP over NFS, we decided to try the 
gluster NFS mount.

That the NFS mount gave us an even stranger result. It doesn't even see 
the Maildir.

# telnet localhost 110
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
+OK <20488.1385248223 at mailtest>
USER glusttest
+OK
PASS somepass
-ERR this user has no $HOME/Maildir
Connection closed by foreign host.


We have verified that the NTP daemon and thus the time settings are correct.
There are NO errors corresponding to this activity in any of the log 
files on either the data servers or the mount (logs only refer to the 
mount/unmount ops).
Using the command line, we can manually manipulate files on the mount to 
our hearts content (as root or as an appropriate user) with no errors. 
mkdir, rm, touch, all work fine.

However, using the VmailMGR command line tools, we are unable to add new 
users when mounted under NFS

# su - glusttest

$ vadduser test2
Enter the user's new password:
Please type it again for verification:
vadduser: error adding the virtual user:
   Can't create the mail directory './users/test2'

then when its manually created qmail refused to deliver to that Maildir 
because it cant see the install
(in this case .qmail file)

1385249260.729221 delivery 146770: failure: 
Sorry,_no_mailbox_here_by_that_name._(#5.1.1)/


Under the FUSE mount, VmailMGR tools work fine and email is able to be 
delivered to the Maildir.

$ vadduser test3
Enter the user's new password:
Please type it again for verification:
vadduser: user 'test3' successfully added.

but  of course the pop3 daemon doesn't see the email that qmail process 
just was just delivered.

Finally, you can see from the info provided, that we deliberately 
disabled root_squash (since thats a new 3.4 feature) but that made no 
difference to any of the above results.

SELINUX is disabled on all 3 machines.

So....

What is going on here?

Its not a mission critical, as GlusterFS is probably inappropriate for a 
mailserver, but I really need to understand what is going on so I can 
recommend GlusterFS for other situations.

Is this a MailDir thing or is something else going on?


Sincerely,

-bill