[pLog-svn] Process size

Ayalon ayalon at blog.nl
Thu Dec 14 13:45:01 GMT 2006


Using subdomain enabled, the rest is in the screenshot.

-----------------------------------------------------------------------------------------------------------------------------------
Jon:

Also in your config there minor issues related to url's:

http://jon.limedaley.com/plog/category/pregnancy will show the categorie
http://jon.limedaley.com/plog/category/pregnancy/ is showing the main page

At my situation yahoo bot is for some reason crawling:

http://amsterdam.blog.nl/amsterdam/2006/12/07 (not a correct url, and shows 
mainpage)
http://amsterdam.blog.nl/amsterdam/2006/12/07/ (not a correct url and gets a 
mem error)

Obvious yahoo crawler is doing incorrect things, but this can't ofcourse get 
there serious errors...




----- Original Message ----- 
From: "Jon Daley" <plogworld at jon.limedaley.com>
To: <plog-svn at devel.lifetype.net>
Sent: Thursday, December 14, 2006 1:50 PM
Subject: Re: [pLog-svn] Process size


What are your custom URL settings set to?  I can't duplicate this
on my blog.

On Thu, 14 Dec 2006, Ayalon wrote:

> First i'm running freedbsd 6.0  with php 5.1.6 and apache 2.2.2
>
> The exact url, i make an exampe:
>
> i'm running a subdomain config with straight categories behind it.
>
> so: subdomain.domain.com/categorie
>
> This ofcourse excist, if i type a categorie behind it that doesn´t excist 
> it
> comes up with an error.
>
> Now i call a categorie but with a archive page look:
>
> subdomain.domain.com/categorie/year/month/date This works fine.
>
> But yahoo is searching it with a / behind the url:
>
> subdomain.domain.com/categorie/year/month/date/ The thing is that this 
> will
> work, but if you put a / behind the url it's not working anymore and is
> causing memory errors etc.
>
> for example:
>
> http://amsterdam.blog.nl/amsterdam/2006/12/07 works fine
>
> http://amsterdam.blog.nl/amsterdam/2006/12/07/ gets errors in the log and 
> out
> of memory issues, please don't click it :)
>
> Anybody an idea how to fix this?
>
>
>
>
>
> ----- Original Message ----- From: "Oscar Renalias" <oscar at renalias.net>
> To: <plog-svn at devel.lifetype.net>
> Sent: Thursday, December 14, 2006 9:51 AM
> Subject: Re: [pLog-svn] Process size
>
>
>> Can you provide the exact URLs? Or even better, the exact line(s) from 
>> the
>> logs.
>>
>> How about your version of PHP? Are you running PHP 4 or PHP 5?
>>
>> On 12/14/06, Ayalon <ayalon at blog.nl> wrote:
>>> Found it!
>>>
>>> Searched the logs for the yahoobot. It looks like yahoo is crawling a 
>>> lot
>>> of
>>> pages that don't exist.
>>>
>>> for example blogname.mydomain.com/categoryname/year/month/date
>>>
>>> While a normal yahoo request is passing by normally. This page doesn't
>>> excist in my enviroment and this will cause some kind of loop in the
>>> script
>>> with eventually causing a lot of mem in php, a lot of buffering in 
>>> apache
>>> etc. I looked in the php and apache forums and bug reports, but can't 
>>> find
>>> anything. Is there something wrong in the way not excisting url's etc 
>>> are
>>> being handled? Because i see in my logs normal request are also handle
>>> correct. Pages, and especially within subdirs are causing these 
>>> problems..
>>>
>>> If you need more info, i'm happy to give. Probably also devel had this
>>> problem last weekend...
>>>
>>>
>>> ----- Original Message -----
>>> From: "Ayalon" <ayalon at blog.nl>
>>> To: <plog-svn at devel.lifetype.net>
>>> Sent: Thursday, December 14, 2006 8:32 AM
>>> Subject: RE: [pLog-svn] Process size
>>>
>>>
>>> > Hi Oscar,
>>> >
>>> > Thanks for your email. But anyway, we're not allowing apache to grow 
>>> > so
>>> > big,
>>> > it's php with e mem limit of 32. If I try to run it with a lower value 
>>> >  >
>>> the
>>> > pages with comments are not shown (100+ comments). So do I have 
>>> > another
>>> > choice?
>>> >
>>> > Jon: I'll true to get the logs to show you what yahoo is hitting..
>>> > Jon2: Can you see anything on devel?
>>> >
>>> > -----Oorspronkelijk bericht-----
>>> > Van: plog-svn-bounces at devel.lifetype.net
>>> > [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
>>> > Verzonden: donderdag 14 december 2006 8:28
>>> > Aan: plog-svn at devel.lifetype.net
>>> > Onderwerp: Re: [pLog-svn] Process size
>>> >
>>> > In addition to all I said before, why do you allow up to 32mb to each
>>> > Apache
>>> > project? I think that's too much, nowadays 8-12mb should be a more
>>> > reasonable figure. If there's memory leaks somewhere in PHP (not in 
>>> > our
>>> > code, remember that there's no way to explicitely deallocate an object 
>>> >  >
>>> in
>>> > PHP code as far as I know), 32mb isn't exactly going to help...
>>> >
>>> > On 12/14/06, Oscar Renalias <oscar at renalias.net> wrote:
>>> >> No, we're not checking anything. From Lifetype's point of view, we
>>> >> don't really care about who is making the request.
>>> >>
>>> >> Could it be that the Yahoo blog is performing searches? You should be
>>> >> able to see what kind of requests the crawler is making by 
>>> >> referencing
>>> >> the timestamps you posted below from apache's error log file with the
>>> >> data you've got in the access log. The apache access log should
>>> >> contain the exact request, please find it and post it here, otherwise
>>> >> we're just guessing.
>>> >>
>>> >> On 14 Dec 2006, at 00:15, Ayalon wrote:
>>> >>
>>> >> > Nope, there's nobody who has so many post in the mainpage as I
>>> >> > configure everybody's blog.
>>> >> >
>>> >> > It's not just an issue with all bots, it's only with yahoo bot. 
>>> >> > When
>>> >> > searching the net I found more similair problems. No check is done
>>> >> > who is coming on the site or something??
>>> >> >
>>> >> > -----Oorspronkelijk bericht-----
>>> >> > Van: plog-svn-bounces at devel.lifetype.net
>>> >> > [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
>>> >> > Verzonden: woensdag 13 december 2006 23:10
>>> >> > Aan: plog-svn at devel.lifetype.net
>>> >> > Onderwerp: Re: [pLog-svn] Process size
>>> >> >
>>> >> > I guess it's the same issue we had in devel.lifetype.net.
>>> >> >
>>> >> > One thing you should check is whether you've got any user who has
>>> >> > configured his/her blog to display something like 80 or 100 posts 
>>> >> > in
>>> >> > the front page, as that can cause performance problems. There's
>>> >> > already a fix for that in LT 1.2, but in the meantime you will have
>>> >> > to keep an eye on it on your own.
>>> >> > Otherwise a crawler performs the exact same operations as a user 
>>> >> > via
>>> >> > a browser would do.
>>> >> >
>>> >> > On 13 Dec 2006, at 20:48, Ayalon wrote:
>>> >> >
>>> >> >> Ok, I found the problem, and I tell you it's really strange but >>
>>> >> true:
>>> >> >>
>>> >> >> The yahoo bot (inktomi bot) is hitting my site and then the
>>> >> >> Cache_lite.php is for some reason using to much memory:
>>> >> >>
>>> >> >> [Tue Dec 12 00:01:44 2006] [error] [client 74.6.85.156] PHP Fatal
>>> >> >> error:
>>> >> >> Allowed memory size of 33554432 bytes exhausted (tried to allocate
>>> >> >> 84 bytes)
>>> >> >> in /data/www/www.blog.nl/class/cache/Cache_Lite/Lite.php on line
>>> >> >> 352
>>> >> >>
>>> >> >> [Tue Dec 12 00:05:27 2006] [error] [client 74.6.86.205] PHP Fatal
>>> >> >> error:
>>> >> >> Allowed memory size of 33554432 bytes exhausted (tried to allocate
>>> >> >> 93 bytes)
>>> >> >> in /data/www/www.blog.nl/class/cache/Cache_Lite/Lite.php on line
>>> >> >> 352
>>> >> >>
>>> >> >>
>>> >> >> When a normal user is hitting the site, there's nothing at all.
>>> >> >> When the bot
>>> >> >> is hitting my site this is happening with a lot of the requests.
>>> >> >> Also the
>>> >> >> apache process is growing so big that at the end the process is
>>> >> >> using so much memory that it's starting to use the swap.
>>> >> >>
>>> >> >> What can cause this problem? Now I blocked yahoo bot via htaccess
>>> >> >> and the problem is not there anymore. It started to happen after
>>> >> >> the upgrade to the new lifetype platform. Are there some checks 
>>> >> >> for
>>> >> >> who is coming in?
>>> >> >> Do you
>>> >> >> need more data?
>>> >> >>
>>> >> >> Anyway it's strange and interesting, anybody an idea....??
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> -----Oorspronkelijk bericht-----
>>> >> >> Van: plog-svn-bounces at devel.lifetype.net
>>> >> >> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Jon Daley
>>> >> >> Verzonden: woensdag 13 december 2006 17:06
>>> >> >> Aan: plog-svn at devel.lifetype.net
>>> >> >> Onderwerp: RE: [pLog-svn] Process size
>>> >> >>
>>> >> >> On Wed, 13 Dec 2006, Ayalon wrote:
>>> >> >>> Where can I find the post of the rewrite?
>>> >> >> http://forums.lifetype.net/viewtopic.php?p=23240&highlight=htaccess
>>> >> >> +rewrite+
>>> >> >> error
>>> >> >>
>>> >> >>> Anyway my provider is telling me that it looks like lifetype has
>>> >> >>> memoryleaks in various aspects of the script, starting with >> 
>>> >> >>>  >>>
>>> caching.
>>> >> >>> Is
>>> >> >> that possible?
>>> >> >>      It is certainly possible.  I would expect to see more memory
>>> >> >> usage in my setup if that were the case.
>>> >> >>
>>> >> >>> Is it possible to disable everything related to cache lift? Just
>>> >> >>> to test some things...
>>> >> >>      Line 39 of class/cache/cachemanager.class.php.  Change
>>> >> >> $cacheEnable to false.  I think that should do the trick.
>>> >> >> _______________________________________________
>>> >> >> pLog-svn mailing list
>>> >> >> pLog-svn at devel.lifetype.net
>>> >> >> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>> >> >>
>>> >> >>
>>> >> >> _______________________________________________
>>> >> >> pLog-svn mailing list
>>> >> >> pLog-svn at devel.lifetype.net
>>> >> >> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>> >> >>
>>> >> >
>>> >> > _______________________________________________
>>> >> > pLog-svn mailing list
>>> >> > pLog-svn at devel.lifetype.net
>>> >> > http://devel.lifetype.net/mailman/listinfo/plog-svn
>>> >> >
>>> >> >
>>> >> > _______________________________________________
>>> >> > pLog-svn mailing list
>>> >> > pLog-svn at devel.lifetype.net
>>> >> > http://devel.lifetype.net/mailman/listinfo/plog-svn
>>> >> >
>>> >>
>>> >> _______________________________________________
>>> >> pLog-svn mailing list
>>> >> pLog-svn at devel.lifetype.net
>>> >> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>> >>
>>> > _______________________________________________
>>> > pLog-svn mailing list
>>> > pLog-svn at devel.lifetype.net
>>> > http://devel.lifetype.net/mailman/listinfo/plog-svn
>>> >
>>> >
>>> > _______________________________________________
>>> > pLog-svn mailing list
>>> > pLog-svn at devel.lifetype.net
>>> > http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>
>>>
>>> ______________________________________________________________________
>>> This email has been scanned by the MessageLabs Email Security System.
>>> For more information please visit http://www.messagelabs.com/email
>>> ______________________________________________________________________
>>> _______________________________________________
>>> pLog-svn mailing list
>>> pLog-svn at devel.lifetype.net
>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>
>> _______________________________________________
>> pLog-svn mailing list
>> pLog-svn at devel.lifetype.net
>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
> _______________________________________________
> pLog-svn mailing list
> pLog-svn at devel.lifetype.net
> http://devel.lifetype.net/mailman/listinfo/plog-svn
>

-- 
Jon Daley
http://jon.limedaley.com/

The difference between genius and stupidity
is that genius has its limits.
-- Anonymous


--------------------------------------------------------------------------------


> _______________________________________________
> pLog-svn mailing list
> pLog-svn at devel.lifetype.net
> http://devel.lifetype.net/mailman/listinfo/plog-svn 


______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________
-------------- next part --------------
A non-text attachment was scrubbed...
Name: urls.jpg
Type: image/jpeg
Size: 116211 bytes
Desc: not available
Url : http:// devel.lifetype.net/pipermail/plog-svn/attachments/20061214/f8d710c0/urls-0001.jpg


More information about the pLog-svn mailing list