[pLog-svn] website

Wed Oct 15 13:54:47 EDT 2008

On Wed, 2008-10-15 at 01:59 -0400, Jon Daley wrote:
> 	These aren't ddos attempts - the worst offender is Microsoft's MSN 
> search bot - I blacklisted him, since he sends hundreds of requests within 
> seconds.
>  	It also happens when there are lots of search engines all crawling 
> at the same time, so even those that do obey the crawl delay, when they 
> all come in at once, it adds up to a lot of requests.
> 
>  	Today, I think I finally (hopefully) got the right setting to help 
> stay alive during these overload situations, primarily by turning down the 
> MaxClient setting, but checking out the other settings within the prefork 
> module settings for apache.

Did this method work?  What did your team end up deciding on for
bringing the site back up?

Andy

> 
> On Wed, 15 Oct 2008, Mark Wu wrote:
> 
> > Another question ... Can you use
> >
> > netstat -ntp |awk '/tcp|udp/{gsub(/:.*/,"",$5);print $5}'|sort|uniq -c|sort
> >
> > to see the connections ...
> >
> > Maybe lifetype.net just ddos attack by some hackers ...
> >
> > mod_limitipconn is not a perfect solution, but it is the most useful way to
> > prevent the DDOS attack.
> >
> > Or, you can try ddos deflate http://deflate.medialayer.com/  ( I never test
> > it before)
> >
> > Mark
> >
> >
> >
> > -----Original Message-----
> > From: plog-svn-bounces at devel.lifetype.net
> > [mailto:plog-svn-bounces at devel.lifetype.net] On Behalf Of Jon Daley
> > Sent: Wednesday, October 15, 2008 12:47 PM
> > To: LifeType Developer List
> > Subject: Re: [pLog-svn] website
> >
> > 	I have played around with squid some - it is hard to get right on a
> > dynamic site.  I haven't found that the search engines care much about the
> > crawl delay directive??
> > 	Opinions on eaccelerator vs. apc? (Is that one of those
> > which-distribution-of-linux-do-you-like-best sort of questions?)
> >
> > 	deflate compresses html and text on the way out?  Do you notice an
> > increased CPU load?  For me, bandwidth is generally cheaper than CPU power.
> > 	The mod_expires is interesting - I do see browsers requesting stuff
> > more than I think they should, though for lifetype - the primary issue is
> > the actual php pages, by spammers and search engines.
> >
> >
> > On Wed, 15 Oct 2008, Mark Wu wrote:
> >> 1. APC for php for opcode cache
> >> 2. mod_expires, mod_deflate for apache for reduce the bandwidth 3.
> >> mod_cache for apache for cache the images/scripts in each web server
> >> to reduce NFS I/O (if you use multple server for load balance, it is
> >> very
> >> useful)
> >>
> >> And, maybe you can add an Nginx or Squid server for reverse proxy, it
> >> might help, too.
> >>
> >> If you want to lowering the indexing frequency of seach engine, maybe
> >> you can try
> >> http://en.wikipedia.org/wiki/Robots_Exclusion_Standard#Crawl-delay_dir
> >> ective
> >>
> >> Mark
> >>
> >> -----Original Message-----
> >> From: plog-svn-bounces at devel.lifetype.net
> >> [mailto:plog-svn-bounces at devel.lifetype.net] On Behalf Of Jon Daley
> >> Sent: Wednesday, October 15, 2008 11:26 AM
> >> To: LifeType Developer List
> >> Subject: Re: [pLog-svn] website
> >>
> >> 	Ok, thanks, it is probably worth another try then.  I tried it
> >> probably 8 or 12 months ago.
> >>
> >> On Tue, 14 Oct 2008, Andy Wright wrote:
> >>
> >>>> 	The last time I tried the accelerators, I had some trouble with the
> >>>> code not executing accurately.
> >>>
> >>> I had trouble with many scripts up until the last few minor versions.
> >>> Now, absolutely everything works absolutely wonderful with eaccelerator.
> >>>
> >>> At that time I was using a lighttpd module called htscanner which
> >>> allowed lighttpd to parse .htaccess files (even though it is not
> >>> supported as native on the web server) and send a PHP option to the
> >>> opcode cacher to disable it on a per directory or hierarchical
> >>> filesystem basis.
> >>>
> >>> It is a rather new technique to the PHP world, still, and it has come
> >>> a long way since the last time you have likely tried using it.
> >>>
> >>> Andy
> >>>
> >>>> On Tue, 14 Oct 2008, Andy Wright wrote:
> >>>>
> >>>>> Could you compress output using the PHP option for scripts, cache
> >>>>> static files on the webserver side + compression, fast-cgi for PHP?
> >>>>>
> >>>>> I have found eaccelerator does for me exactly what you are asking for.
> >>>>> It is an opcode cacher that I use with fast-cgi and lighttpd..  if
> >>>>> you are using fast-cgi, make sure to only spawn one process, and
> >>>>> any number of children.
> >>>>>
> >>>>> It will use PHP output in a compiled state to execute phpbb, send
> >>>>> it to your webserver which could compress the html output to the
> >>>>> client, and serve other static files not provided by the PHP
> >>>>> process in a compressed and cached state..
> >>>>>
> >>>>> might help...
> >>>>>
> >>>>> Andy
> >>>>>
> >>>>> http://myspew.com
> >>>>>
> >>>>> --
> >>>>>
> >>>>> On Tue, 2008-10-14 at 16:40 -0400, Jon Daley wrote:
> >>>>>> I wrote the below email this morning, but didn't send it due to
> >>>>>> being stressed about the server being down.
> >>>>>>
> >>>>>> I am now back in Pittsburgh, and the machine appears to be fine,
> >>>>>> as long as I leave the lifetype sites turned off.
> >>>>>>
> >>>>>> I wonder if there is a known vulnerability in phpbb or something
> >>>>>> that someone is exploiting?  I don't know how one would get a
> >>>>>> complete server crash without any logs or anything from a php
> >>>>>> process, remotely - maybe the site has been hacked?  I haven't
> >>>>>> looked
> >> through the files yet.
> >>>>>>
> >>>>>> If anyone else has time(Reto?), that would be great.
> >>>>>>
> >>>>>> ---------------------------------
> >>>>>>
> >>>>>>  	Search engines have taken down the server a couple times in the
> >>>>>> last week.  I am traveling today, and had to take lifetype.net
> >>>>>> down, as I can't have it killing everything else.
> >>>>>>  	I talked to an employee of Google, and I gather that the search
> >>>>>> engines are trying to be more aggressive in having the most
> >>>>>> up-to-date content, thus causing them to grab data continuously
> >> throughout the day.
> >>>>>> MSN is the worst offender, but some of the little unknown search
> >>>>>> engines are causing trouble too.
> >>>>>>  	We'll probably need to find some caching or something.  If
> >>>>>> anyone has time to look up some caching for phpbb3, that would be
> > great.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> pLog-svn mailing list
> >>>>> pLog-svn at devel.lifetype.net
> >>>>> http://limedaley.com/mailman/listinfo/plog-svn
> >>>>>
> >>>>
> >>>
> >>> _______________________________________________
> >>> pLog-svn mailing list
> >>> pLog-svn at devel.lifetype.net
> >>> http://limedaley.com/mailman/listinfo/plog-svn
> >>>
> >>
> >> --
> >> Jon Daley
> >> http://jon.limedaley.com
> >> ~~
> >> Before you criticize someone, you should walk a mile in their shoes.
> >> That way, when you criticize them,
> >>   you're a mile away and you have their shoes.
> >> _______________________________________________
> >> pLog-svn mailing list
> >> pLog-svn at devel.lifetype.net
> >> http://limedaley.com/mailman/listinfo/plog-svn
> >>
> >> _______________________________________________
> >> pLog-svn mailing list
> >> pLog-svn at devel.lifetype.net
> >> http://limedaley.com/mailman/listinfo/plog-svn
> >>
> >
> > --
> > Jon Daley
> > http://jon.limedaley.com
> > ~~
> > There is no "I" in Team, but it does contain a silent "scapegoat"
> > _______________________________________________
> > pLog-svn mailing list
> > pLog-svn at devel.lifetype.net
> > http://limedaley.com/mailman/listinfo/plog-svn
> >
> > _______________________________________________
> > pLog-svn mailing list
> > pLog-svn at devel.lifetype.net
> > http://limedaley.com/mailman/listinfo/plog-svn
> >
> 