[Gluster-devel] Faster hashing for DHT

[Martin Fick]

Hmm, if it were collision resistant, wouldn't that mean that you would need one server for each file you want to store?  I suspect you want many collisions, just a good even distribution of those collisions,

-Martin

--- On Tue, 1/5/10, Jeff Darcy <jdarcy at redhat.com> wrote:

> From: Jeff Darcy <jdarcy at redhat.com>
> Subject: [Gluster-devel] Faster hashing for DHT
> To: gluster-devel at nongnu.org
> Date: Tuesday, January 5, 2010, 12:12 PM
> While looking at the DHT code, I
> noticed that it's using a 10-round
> Davies-Meyer construction to generate the hashes used for
> file
> placement.  A little surprised, by this, I ran it by a
> couple of friends
> who are experts in both cryptography and distributed data
> storage.  The
> consensus seems to be that the hash used for this purpose
> needs to be
> collision resistant but not cryptographically strong. 
> One theorized
> that the choice made in DHT is probably based on prior
> examples (e.g.
> Freenet and Mojo Nation) where cryptographically strong
> hashes were
> chosen, but that the requirements driving those decisions
> probably don't
> apply to GlusterFS.  This is a non-trivial issue
> because these hashes
> are used quite frequently and the current one is quite
> computationally
> expensive.  I note that Hsieh's SuperFastHash is
> already implemented in
> GlusterFS and is used for other purposes.  It's about
> 3x as fast as the
> DM hash, and has better collision resistance as well. 
> MurmurHash
> (http://murmurhash.googlepages.com/) is even faster and
> more collision
> resistant.  For future releases, I suggest dropping
> the DM hash and
> switching to one of these others.
> 
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>