[Gluster-devel] [Gluster-infra] lists.gluster.org issues this weekend

[Michael Scherer]

Le mardi 19 septembre 2017 à 17:33 +0100, Michael Scherer a écrit :
> Le samedi 16 septembre 2017 à 20:48 +0530, Nigel Babu a écrit :
> > Hello folks,
> > 
> > We have discovered that for the last few weeks our mailman server
> > was
> > used
> > for a spam attack. The attacker would make use of the + feature
> > offered by
> > gmail and hotmail. If you send an email to example at hotmail.com,
> > example+foo at hotmail.com, example+bar at hotmail.com, it goes to the
> > same
> > inbox. We were constantly hit with requests to subscribe to a few
> > inboxes.
> > These requests overloaded our mail server so much that it gave up.
> > We
> > detected this failure because a postmortem email to
> > gluster-infra at gluster.org bounced. Any emails sent to our mailman
> > server
> > may have been on hold for the last 24 hours or so. They should be
> > processed
> > now as your email provider re-attempts.
> > 
> > For the moment, we've banned subscribing with an email address with
> > a
> > + in
> > the name. If you are already subscribed to the lists with a + in
> > your
> > email
> > address, you will continue to be able to use the lists.
> > 
> > We're looking at banning the spam IP addresses from being able to
> > hit
> > the
> > web interface at all. When we have a working alternative, we will
> > look at
> > removing the current ban of using + in address.
> 
> So we have a alternative in place, I pushed a blacklist using
> mod_security and a few DNS blacklist:
> https://github.com/gluster/gluster.org_ansible_configuration/commit/2
> f4
> c1b8feeae16e1d0b7d6073822a6786ed21dde
> 
> 
> 
> 
> > Apologies for the outage and a big shout out to Michael for taking
> > time out
> > of his weekend to debug and fix the issue.
> 
> Well, you can thanks the airport in Prague for being less interesting
> than a spammer attacking us.

So, it turned out to have a 2nd problem on the lists server.

Since the 2017 security incident, we have installed a remote syslog
server to store all logs. However, this logs server disk became full
(I still need to investigate why, but I think "lack of proper log
rotation" somewhere down the line).

In turn, the disk being full did cause slowdown on some syslog clients,
even if for now, we only seen issue on postfix for
supercolony.gluster.org. 

As a emergency measure, I did remove logs export for supercolony, and I
will be likely moving the logs server to the community cage and fix the
setup once I am back from PTO next week.
-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20170925/e7a7e319/attachment.sig>