[Gluster-devel] Faster hashing for DHT

[Jeff Darcy]

While looking at the DHT code, I noticed that it's using a 10-round
Davies-Meyer construction to generate the hashes used for file
placement.  A little surprised, by this, I ran it by a couple of friends
who are experts in both cryptography and distributed data storage.  The
consensus seems to be that the hash used for this purpose needs to be
collision resistant but not cryptographically strong.  One theorized
that the choice made in DHT is probably based on prior examples (e.g.
Freenet and Mojo Nation) where cryptographically strong hashes were
chosen, but that the requirements driving those decisions probably don't
apply to GlusterFS.  This is a non-trivial issue because these hashes
are used quite frequently and the current one is quite computationally
expensive.  I note that Hsieh's SuperFastHash is already implemented in
GlusterFS and is used for other purposes.  It's about 3x as fast as the
DM hash, and has better collision resistance as well.  MurmurHash
(http://murmurhash.googlepages.com/) is even faster and more collision
resistant.  For future releases, I suggest dropping the DM hash and
switching to one of these others.