[pLog-svn] Process size

Oscar Renalias oscar at renalias.net
Fri Mar 2 07:47:29 EST 2007 

Can you tell us how long your apache processes are running?
Specifically, look at the MaxRequestsPerChild parameter
(http://httpd.apache.org/docs/1.3/mod/core.html#maxrequestsperchild)
and  I recently checked a site where they had this value to over 1000,
and a number of childs of over 100. As a result they had Apache
processes that were over 1 week old and consuming over 100mb of
memory, while the new ones only used about 15-20mb.

You should set this parameter to a relatively low value. It's pretty
much a fact that PHP leaks memory per every request so by setting this
value to a lower number you will ensure that apache processes are
periodically killed and their memory freed.

There's also other Apache parameter you should look at,
MaxSpareServers, MinSpareServers, StartServers, etc.

On 3/2/07, Ayalon <ayalon at blog.nl> wrote:
> The following problem also excist in 1.2 with an out of the box
> installation:
>
>
> Hi All,
>
> Concerning this issue I made a full test enviroment availble for custom
> url's. Wildcards dns etc is there, default lifetype is there, the error is
> there;)..
>
> Who wants access, send me private the details:
>
> Ip of your connection
>
> The I'll send login details in return..
>
>
>
> -----Oorspronkelijk bericht-----
> Van: plog-svn-bounces at devel.lifetype.net
> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Ayalon
> Verzonden: vrijdag 15 december 2006 9:38
> Aan: plog-svn at devel.lifetype.net
> Onderwerp: RE: [pLog-svn] Process size
>
> Oscar,
>
> Do you need assistance for this? Let me know if it works and if I can help
> you put in the good values and files.
>
> Is there not a possibilty to have wildcard dns enabled on one of the
> lifetype server and test is this way? A central good test enviroment for all
> scenrario's.
>
> Regards
>
> -----Oorspronkelijk bericht-----
> Van: plog-svn-bounces at devel.lifetype.net
> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
> Verzonden: vrijdag 15 december 2006 7:45
> Aan: plog-svn at devel.lifetype.net
> Onderwerp: Re: [pLog-svn] Process size
>
> Hmm, I will see what I can do in my local instance...
>
> On 15 Dec 2006, at 03:02, Jon Daley wrote:
>
> >       It is probably just his custom URL settings, and you don't need his
> > server?  I guess maybe you need his server to see what it dies, but
> > you could at least analyze what happens when adding the extra slash -
> > see what articles show up, and if all the articles in the blog get
> > listed, then you at least know it is wrong.  It is probably too hard
> > to try to debug it remotely.
> >
> > On Fri, 15 Dec 2006, Oscar Renalias wrote:
> >
> >> No, I can't really send you anything because I've got nothing to
> >> send. In fact I don't know where to start, and my comments was more
> >> along the lines of whether any of us could be granted access to the
> >> server for debugging purposes instead of sending the scripts to you,
> >> then waiting to hear form you, analyze the results, rinse and repeat.
> >>
> >> On 14 Dec 2006, at 23:20, Ayalon wrote:
> >>
> >>> Ok, I'll ask to make it possible!
> >>> Is there any debug code I can run? Xdebug is not availble on the
> >>> server. You have some kind of debug scripts you can send me to test
> >>> out?
> >>> -----Oorspronkelijk bericht-----
> >>> Van: plog-svn-bounces at devel.lifetype.net
> >>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
> >>> Verzonden: donderdag 14 december 2006 22:16
> >>> Aan: plog-svn at devel.lifetype.net
> >>> Onderwerp: Re: [pLog-svn] Process size At the very least, we should
> >>> be able to upload classes with debug statements and be able to
> >>> browse the logs, so that we can see what's going on. It would be
> >>> even better if we could install something like xdebug for tracing
> >>> the code but we can also manage with debug code.
> >>> On 14 Dec 2006, at 20:49, Ayalon wrote:
> >>>> Yes I have a testserver, and yes it happens with every request like
> >>>> this.
> >>>> And it only happens if a categorie excist and is called with the
> >>>> full subdomain.domain.com/category/year/month/date/ So taking of
> >>>> the postname (slug), if I do like
> >>>> subdomain.domain.com/category/year/month/date
> >>>> it comes
> >>>> up with the mainpage subdomain.domain.com/category/year/month/ it
> >>>> comes up with the mainpage subdomain.domain.com/category/year/month
> >>>> aswell etc etc..
> >>>> If I look in the cache it looks like it's trying to load all the
> >>>> posts etc in the database...
> >>>> what kind of access do you need?
> >>>> As it is a managed server I only have ftp access at the
> >>>> testmachine..
> >>>> -----Oorspronkelijk bericht-----
> >>>> Van: plog-svn-bounces at devel.lifetype.net
> >>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
> >>>> Verzonden: donderdag 14 december 2006 19:35
> >>>> Aan: plog-svn at devel.lifetype.net
> >>>> Onderwerp: Re: [pLog-svn] Process size Do you have some sort of
> >>>> test/staging server with the same exact data as your live server
> >>>> that we could use to play around with this?
> >>>> How about the logs, could you analyze them to find out whether each
> >>>> single occurence of memory problems was related to this? Or the
> >>>> other way around, did all requests like this one cause memory
> >>>> issues?
> >>>> On 14 Dec 2006, at 16:41, Ayalon wrote:
> >>>>> Well what i did, as you can see in the previous screenshots, i
> >>>>> removed all the  /blog/{blogname} from the links. Thats it.
> >>>>> In my htaccess is only the forcetype part and error.php part. No
> >>>>> rewrite rules nothing..
> >>>>> Why does those url's have to do a redirect? I don't understand,
> >>>>> and with lifetype 1.0.6 and previous it was working perfectly.
> >>>>> (i'm using lifetype for years;))
> >>>>> ----- Original Message ----- From: "Jon Daley"
> >>>>> <plogworld at jon.limedaley.com>
> >>>>> To: "LifeType SVN" <plog-svn at devel.lifetype.net>
> >>>>> Sent: Thursday, December 14, 2006 3:29 PM
> >>>>> Subject: Re: [pLog-svn] Process size Ah - you don't have
> >>>>> /category/{category} in your URLs.
> >>>>> So
> >>>>> http://amsterdam.blog.nl/toerist_in_eigen_stad/2006/12/12/
> >>>>> economie-
> >>>>> amsterdam-staat-er-goed-voor
> >>>>> has to do a redirect.  What is in your .htaccess?  I am not seeing
> >>>>> a 400 returned, which means you either wrote some custom rules, or
> >>>>> the modrewrite stuff is being matched.
> >>>>> You have read this, right?
> >>>>> http://wiki.lifetype.net/index.php/Custom_URLs#Why_removing_.
> >>>>> 2Fblog.
> >>>>> 2F_from_the_URLs_is_evil
> >>>>> I bet if you put the URLs back to the defaults, your problem will
> >>>>> go away.
> >>>>> We should probably write a custom URL validator or something to
> >>>>> check when people write bad URLs, though I am not sure how it is
> >>>>> parsing it when you have the extra slash.
> >>>>> On Thu, 14 Dec 2006, Ayalon wrote:
> >>>>>> Ok, yours is fixed, now my problem..
> >>>>>> A not excisting url with the format categorie/year/month/day/ is
> >>>>>> causing memory errors..
> >>>>>> ----- Original Message ----- From: "Jon Daley"
> >>>>>> <plogworld at jon.limedaley.com>
> >>>>>> To: <plog-svn at devel.lifetype.net>
> >>>>>> Sent: Thursday, December 14, 2006 3:10 PM
> >>>>>> Subject: Re: [pLog-svn] Process size The slashes are now enabled,
> >>>>>> that is just a configuration choice, and I hadn't ever noticed
> >>>>>> before.
> >>>>>> I changed them to:
> >>>>>> category_link_format
> >>>>>> /category/{catname}/?$
> >>>>>> archive_link_format
> >>>>>> /archives/{year}/?{month}/?({day}/?)?
> >>>>>> actually, the archive format didn't work like that, because it
> >>>>>> started generating URLs like:
> >>>>>> http://jon.limedaley.com/plog/archives/2006/01//
> >>>>>> which caused it to only show one day's worth of posts.
> >>>>>> So, my archive_link_format is:
> >>>>>> /archives/{year}/?{month}/?{day}?$
> >>>>>> slash removed to avoid the double slash problem, and the $ added
> >>>>>> to distinguish from my permalink entry:
> >>>>>> /archives/{year}/{month}/{day}/{postname}$
> >>>>>> Probably it was matching with a null postname or something.
> >>>>>> In all of that, though, I never had any memory problems.  I
> >>>>>> believe I have it currently set to 16MB.  This is enough to show
> >>>>>> lots of comments on one page, but not to show thousands, and
> >>>>>> small enough to not kill the server if lots of pages are viewed
> >>>>>> simultaneously.
> >>>>>> On Thu, 14 Dec 2006, Ayalon wrote:
> >>>>>>> Using subdomain enabled, the rest is in the screenshot.
> >>>>>>> ----------------------------------------------------------------
> >>>>>>> ---
> >>>>>>> -
> >>>>>>> ---------------------------------------------------------------
> >>>>>>> Jon:
> >>>>>>> Also in your config there minor issues related to url's:
> >>>>>>> http://jon.limedaley.com/plog/category/pregnancy will show the
> >>>>>>> categorie http://jon.limedaley.com/plog/category/pregnancy/ is
> >>>>>>> showing the main page At my situation yahoo bot is for some
> >>>>>>> reason crawling:
> >>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07 (not a correct
> >>>>>>> url, and shows
> >>>>>>> mainpage)
> >>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07/ (not a correct
> >>>>>>> url and gets a mem error) Obvious yahoo crawler is doing
> >>>>>>> incorrect things, but this can't ofcourse get there serious
> >>>>>>> errors...
> >>>>>>> ----- Original Message ----- From: "Jon Daley"
> >>>>>>> <plogworld at jon.limedaley.com>
> >>>>>>> To: <plog-svn at devel.lifetype.net>
> >>>>>>> Sent: Thursday, December 14, 2006 1:50 PM
> >>>>>>> Subject: Re: [pLog-svn] Process size What are your custom URL
> >>>>>>> settings set to?  I can't duplicate this on my blog.
> >>>>>>> On Thu, 14 Dec 2006, Ayalon wrote:
> >>>>>>>> First i'm running freedbsd 6.0  with php 5.1.6 and apache 2.2.2
> >>>>>>>> The exact url, i make an exampe:
> >>>>>>>> i'm running a subdomain config with straight categories behind
> >>>>>>>> it.
> >>>>>>>> so: subdomain.domain.com/categorie This ofcourse excist, if i
> >>>>>>>> type a categorie behind it that doesn ´t excist it comes up
> >>>>>>>> with an error.
> >>>>>>>> Now i call a categorie but with a archive page look:
> >>>>>>>> subdomain.domain.com/categorie/year/month/date This works fine.
> >>>>>>>> But yahoo is searching it with a / behind the url:
> >>>>>>>> subdomain.domain.com/categorie/year/month/date/ The thing is
> >>>>>>>> that this will work, but if you put a / behind the url it's not
> >>>>>>>> working anymore and is causing memory errors etc.
> >>>>>>>> for example:
> >>>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07 works fine
> >>>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07/ gets errors in
> >>>>>>>> the log and out of memory issues, please don't click it :)
> >>>>>>>> Anybody an idea how to fix this?
> >>>>>>>> ----- Original Message ----- From: "Oscar Renalias"
> >>>>>>>> <oscar at renalias.net>
> >>>>>>>> To: <plog-svn at devel.lifetype.net>
> >>>>>>>> Sent: Thursday, December 14, 2006 9:51 AM
> >>>>>>>> Subject: Re: [pLog-svn] Process size
> >>>>>>>>> Can you provide the exact URLs? Or even better, the exact line
> >>>>>>>>> (s) from
> >>>>>>>>> the
> >>>>>>>>> logs.
> >>>>>>>>> How about your version of PHP? Are you running PHP 4 or PHP 5?
> >>>>>>>>> On 12/14/06, Ayalon <ayalon at blog.nl> wrote:
> >>>>>>>>>> Found it!
> >>>>>>>>>> Searched the logs for the yahoobot. It looks like yahoo is
> >>>>>>>>>> crawling a lot of pages that don't exist.
> >>>>>>>>>> for example blogname.mydomain.com/categoryname/year/month/
> >>>>>>>>>> date
> >>>>>>>>>> While a normal yahoo request is passing by normally. This
> >>>>>>>>>> page doesn't excist in my enviroment and this will cause some
> >>>>>>>>>> kind of loop in the script with eventually causing a lot of
> >>>>>>>>>> mem in php, a lot of buffering in apache etc. I looked in the
> >>>>>>>>>> php and apache forums and bug reports, but can't find
> >>>>>>>>>> anything. Is there something wrong in the way not excisting
> >>>>>>>>>> url's etc are being handled? Because i see in my logs normal
> >>>>>>>>>> request are also handle correct. Pages, and especially within
> >>>>>>>>>> subdirs are causing these problems..
> >>>>>>>>>> If you need more info, i'm happy to give. Probably also devel
> >>>>>>>>>> had this problem last weekend...
> >>>>>>>>>> ----- Original Message -----
> >>>>>>>>>> From: "Ayalon" <ayalon at blog.nl>
> >>>>>>>>>> To: <plog-svn at devel.lifetype.net>
> >>>>>>>>>> Sent: Thursday, December 14, 2006 8:32 AM
> >>>>>>>>>> Subject: RE: [pLog-svn] Process size
> >>>>>>>>>>> Hi Oscar,
> >>>>>>>>>>> Thanks for your email. But anyway, we're not allowing apache
> >>>>>>>>>> to grow >  >
> >>>>>>>>>> so
> >>>>>>>>>>> big,
> >>>>>>>>>>> it's php with e mem limit of 32. If I try to run it with a
> >>>>>>>>>> lower >
> >>>>>>>>>> value
> >>>>>>>>>> the
> >>>>>>>>>>> pages with comments are not shown (100+ comments). So do I
> >>>>>>>>>> have >
> >>>>>>>>>> another
> >>>>>>>>>>> choice?
> >>>>>>>>>>> Jon: I'll true to get the logs to show you what yahoo is
> >>>>>>>>>> hitting..
> >>>>>>>>>>> Jon2: Can you see anything on devel?
> >>>>>>>>>>> -----Oorspronkelijk bericht-----
> >>>>>>>>>>> Van: plog-svn-bounces at devel.lifetype.net
> >>>>>>>>>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar
> >>>>>>>>>> Renalias
> >>>>>>>>>>> Verzonden: donderdag 14 december 2006 8:28
> >>>>>>>>>>> Aan: plog-svn at devel.lifetype.net
> >>>>>>>>>>> Onderwerp: Re: [pLog-svn] Process size In addition to all I
> >>>>>>>>>>> said before, why do you allow up to
> >>>>>>>>>> 32mb to > each
> >>>>>>>>>>> Apache
> >>>>>>>>>>> project? I think that's too much, nowadays 8-12mb should be
> >>>>>>>>>> a more
> >>>>>>>>>>> reasonable figure. If there's memory leaks somewhere in PHP
> >>>>>>>>>> (not in >  >
> >>>>>>>>>> our
> >>>>>>>>>>> code, remember that there's no way to explicitely deallocate
> >>>>>>>>>> an >
> >>>>>>>>>> object
> >>>>>>>>>> in
> >>>>>>>>>>> PHP code as far as I know), 32mb isn't exactly going to
> >>>>>>>>>>> help...
> >>>>>>>>>>> On 12/14/06, Oscar Renalias <oscar at renalias.net> wrote:
> >>>>>>>>>>>> No, we're not checking anything. From Lifetype's point of
> >>>>>>>>>> view, we
> >>>>>>>>>>>> don't really care about who is making the request.
> >>>>>>>>>>>> Could it be that the Yahoo blog is performing searches? You
> >>>>>>>>>> should >>  >>
> >>>>>>>>>> be
> >>>>>>>>>>>> able to see what kind of requests the crawler is making by
> >>>>>>>>>>>> >>
> >>>>>>>>>> referencing
> >>>>>>>>>>>> the timestamps you posted below from apache's error log
> >>>>>>>>>> file with >>  >>
> >>>>>>>>>> the
> >>>>>>>>>>>> data you've got in the access log. The apache access log
> >>>>>>>>>> should
> >>>>>>>>>>>> contain the exact request, please find it and post it here,
> >>>>>>>>>>>> >>
> >>>>>>>>>> otherwise
> >>>>>>>>>>>> we're just guessing.
> >>>>>>>>>>>> On 14 Dec 2006, at 00:15, Ayalon wrote:
> >>>>>>>>>>>>> Nope, there's nobody who has so many post in the mainpage
> >>>>>>>>>> as I
> >>>>>>>>>>>>> configure everybody's blog.
> >>>>>>>>>>>>> It's not just an issue with all bots, it's only with
> >>>>>>>>>> yahoo bot. >> >  >>
> >>>>>>>>>> When
> >>>>>>>>>>>>> searching the net I found more similair problems. No
> >>>>>>>>>> check is >> > done
> >>>>>>>>>>>>> who is coming on the site or something??
> >>>>>>>>>>>>> -----Oorspronkelijk bericht-----
> >>>>>>>>>>>>> Van: plog-svn-bounces at devel.lifetype.net
> >>>>>>>>>>>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar
> >>>>>>>>>>>>> Renalias
> >>>>>>>>>>>>> Verzonden: woensdag 13 december 2006 23:10
> >>>>>>>>>>>>> Aan: plog-svn at devel.lifetype.net
> >>>>>>>>>>>>> Onderwerp: Re: [pLog-svn] Process size I guess it's the
> >>>>>>>>>>>>> same issue we had in devel.lifetype.net.
> >>>>>>>>>>>>> One thing you should check is whether you've got any user
> >>>>>>>>>> who has
> >>>>>>>>>>>>> configured his/her blog to display something like 80 or
> >>>>>>>>>> 100 posts
> >>>>>>>>>>> in
> >>>>>>>>>>>>> the front page, as that can cause performance problems.
> >>>>>>>>>> There's
> >>>>>>>>>>>>> already a fix for that in LT 1.2, but in the meantime you
> >>>>>>>>>> will >> >> >  >
> >>>>>>>>>> have
> >>>>>>>>>>>>> to keep an eye on it on your own.
> >>>>>>>>>>>>> Otherwise a crawler performs the exact same operations as
> >>>>>>>>>> a user >> >  >>
> >>>>>>>>>>> via
> >>>>>>>>>>>>> a browser would do.
> >>>>>>>>>>>>> On 13 Dec 2006, at 20:48, Ayalon wrote:
> >>>>>>>>>>>>>> Ok, I found the problem, and I tell you it's really
> >>>>>>>>>> strange but >> >>  >>
> >>>>>>>>>>>> true:
> >>>>>>>>>>>>>> The yahoo bot (inktomi bot) is hitting my site and then
> >>>>>>>>>>>>>> the Cache_lite.php is for some reason using to much
> >>>>>>>>>>>>>> memory:
> >>>>>>>>>>>>>> [Tue Dec 12 00:01:44 2006] [error] [client 74.6.85.156]
> >>>>>>>>>> PHP >> >> Fatal
> >>>>>>>>>>>>>> error:
> >>>>>>>>>>>>>> Allowed memory size of 33554432 bytes exhausted (tried
> >>>>>>>>>> to >> >>
> >>>>>>>>>> allocate
> >>>>>>>>>>>>>> 84 bytes)
> >>>>>>>>>>>>>> in /data/www/www.blog.nl/class/cache/Cache_Lite/Lite.php
> >>>>>>>>>> on line
> >>>>>>>>>>>>>> 352
> >>>>>>>>>>>>>> [Tue Dec 12 00:05:27 2006] [error] [client 74.6.86.205]
> >>>>>>>>>> PHP >> >> Fatal
> >>>>>>>>>>>>>> error:
> >>>>>>>>>>>>>> Allowed memory size of 33554432 bytes exhausted (tried
> >>>>>>>>>> to >> >>
> >>>>>>>>>> allocate
> >>>>>>>>>>>>>> 93 bytes)
> >>>>>>>>>>>>>> in /data/www/www.blog.nl/class/cache/Cache_Lite/Lite.php
> >>>>>>>>>> on line
> >>>>>>>>>>>>>> 352
> >>>>>>>>>>>>>> When a normal user is hitting the site, there's nothing
> >>>>>>>>>> at all.
> >>>>>>>>>>>>>> When the bot
> >>>>>>>>>>>>>> is hitting my site this is happening with a lot of the
> >>>>>>>>>> requests.
> >>>>>>>>>>>>>> Also the
> >>>>>>>>>>>>>> apache process is growing so big that at the end the
> >>>>>>>>>> process is
> >>>>>>>>>>>>>> using so much memory that it's starting to use the swap.
> >>>>>>>>>>>>>> What can cause this problem? Now I blocked yahoo bot via
> >>>>>>>>>>>>>> htaccess and the problem is not there anymore. It started
> >>>>>>>>>>>>>> to
> >>>>>>>>>> happen after
> >>>>>>>>>>>>>> the upgrade to the new lifetype platform. Are there some
> >>>>>>>>>> checks >> >>  >>
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>> who is coming in?
> >>>>>>>>>>>>>> Do you
> >>>>>>>>>>>>>> need more data?
> >>>>>>>>>>>>>> Anyway it's strange and interesting, anybody an
> >>>>>>>>>>>>>> idea....??
> >>>>>>>>>>>>>> -----Oorspronkelijk bericht-----
> >>>>>>>>>>>>>> Van: plog-svn-bounces at devel.lifetype.net
> >>>>>>>>>>>>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Jon
> >>>>>>>>>> Daley
> >>>>>>>>>>>>>> Verzonden: woensdag 13 december 2006 17:06
> >>>>>>>>>>>>>> Aan: plog-svn at devel.lifetype.net
> >>>>>>>>>>>>>> Onderwerp: RE: [pLog-svn] Process size On Wed, 13 Dec
> >>>>>>>>>>>>>> 2006, Ayalon wrote:
> >>>>>>>>>>>>>>> Where can I find the post of the rewrite?
> >>>>>>>>>> http://forums.lifetype.net/viewtopic.php?
> >>>>>>>>>> p=23240&highlight=htaccess
> >>>>>>>>>>>>>> +rewrite+
> >>>>>>>>>>>>>> error
> >>>>>>>>>>>>>>> Anyway my provider is telling me that it looks like
> >>>>>>>>>> lifetype >> >>> has
> >>>>>>>>>>>>>>> memoryleaks in various aspects of the script, starting
> >>>>>>>>>> with >> >> >>>  >>
> >>>>>>>>>> caching.
> >>>>>>>>>>>>>>> Is
> >>>>>>>>>>>>>> that possible?
> >>>>>>>>>>>>>>     It is certainly possible.  I would expect to see
> >>>>>>>>>> more >> >> memory
> >>>>>>>>>>>>>> usage in my setup if that were the case.
> >>>>>>>>>>>>>>> Is it possible to disable everything related to cache
> >>>>>>>>>> lift? >> >>> Just
> >>>>>>>>>>>>>>> to test some things...
> >>>>>>>>>>>>>>     Line 39 of class/cache/cachemanager.class.php.
> >>>>>>>>>>>>>> Change
> >>>>>>>>>>>>>> $cacheEnable to false.  I think that should do the trick.
> >>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>> _____________________________________________________________
> >>>>>>>>>> ___
> >>>>>>>>>> _
> >>>>>>>>>> _____
> >>>>>>>>>> This email has been scanned by the MessageLabs Email Security
> >>>>>>>>>> System.
> >>>>>>>>>> For more information please visit http://www.messagelabs.com/
> >>>>>>>>>> email
> >>>>>>>>>> _____________________________________________________________
> >>>>>>>>>> ___
> >>>>>>>>>> _
> >>>>>>>>>> _____
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>> _______________________________________________
> >>>>>>>>> pLog-svn mailing list
> >>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>> _______________________________________________________________
> >>>>>>>> ___
> >>>>>>>> _
> >>>>>>>> ___
> >>>>>>>> This email has been scanned by the MessageLabs Email Security
> >>>>>>>> System.
> >>>>>>>> For more information please visit http://www.messagelabs.com/
> >>>>>>>> email
> >>>>>>>> _______________________________________________________________
> >>>>>>>> ___
> >>>>>>>> _
> >>>>>>>> ___
> >>>>>>>> _______________________________________________
> >>>>>>>> pLog-svn mailing list
> >>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>> --
> >>>>>>> Jon Daley
> >>>>>>> http://jon.limedaley.com/
> >>>>>>> The difference between genius and stupidity is that genius has
> >>>>>>> its limits.
> >>>>>>> -- Anonymous
> >>>>>>> ----------------------------------------------------------------
> >>>>>>> ---
> >>>>>>> -
> >>>>>>> ------------
> >>>>>>>> _______________________________________________
> >>>>>>>> pLog-svn mailing list
> >>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>> ________________________________________________________________
> >>>>>>> ___
> >>>>>>> _
> >>>>>>> __
> >>>>>>> This email has been scanned by the MessageLabs Email Security
> >>>>>>> System.
> >>>>>>> For more information please visit http://www.messagelabs.com/
> >>>>>>> email
> >>>>>>> ________________________________________________________________
> >>>>>>> ___
> >>>>>>> _
> >>>>>>> __
> >>>>>> --
> >>>>>> Jon Daley
> >>>>>> http://jon.limedaley.com/
> >>>>>> Music: A safe kind of high
> >>>>>> -- Jimi Hendrix
> >>>>>> -----------------------------------------------------------------
> >>>>>> ---
> >>>>>> -
> >>>>>> -----------
> >>>>>>> _______________________________________________
> >>>>>>> pLog-svn mailing list
> >>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>> _________________________________________________________________
> >>>>>> ___
> >>>>>> _
> >>>>>> _
> >>>>>> This email has been scanned by the MessageLabs Email Security
> >>>>>> System.
> >>>>>> For more information please visit http://www.messagelabs.com/
> >>>>>> email
> >>>>>> _________________________________________________________________
> >>>>>> ___
> >>>>>> _
> >>>>>> _
> >>>>>> _______________________________________________
> >>>>>> pLog-svn mailing list
> >>>>>> pLog-svn at devel.lifetype.net
> >>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>> --
> >>>>> Jon Daley
> >>>>> http://jon.limedaley.com/
> >>>>> If everything is coming your way then you're in the wrong lane.
> >>>>> -- Anonymous
> >>>>> ------------------------------------------------------------------
> >>>>> ---
> >>>>> -
> >>>>> ----------
> >>>>>> _______________________________________________
> >>>>>> pLog-svn mailing list
> >>>>>> pLog-svn at devel.lifetype.net
> >>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>> __________________________________________________________________
> >>>>> ___
> >>>>> _
> >>>>> This email has been scanned by the MessageLabs Email Security
> >>>>> System.
> >>>>> For more information please visit http://www.messagelabs.com/email
> >>>>> __________________________________________________________________
> >>>>> ___
> >>>>> _
> >>>>> _______________________________________________
> >>>>> pLog-svn mailing list
> >>>>> pLog-svn at devel.lifetype.net
> >>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>> _______________________________________________
> >>>> pLog-svn mailing list
> >>>> pLog-svn at devel.lifetype.net
> >>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>> _______________________________________________
> >>>> pLog-svn mailing list
> >>>> pLog-svn at devel.lifetype.net
> >>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>> _______________________________________________
> >>> pLog-svn mailing list
> >>> pLog-svn at devel.lifetype.net
> >>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>> _______________________________________________
> >>> pLog-svn mailing list
> >>> pLog-svn at devel.lifetype.net
> >>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>
> >> _______________________________________________
> >> pLog-svn mailing list
> >> pLog-svn at devel.lifetype.net
> >> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >
> > --
> > Jon Daley
> > http://jon.limedaley.com/
> >
> > When dog food is new and improved tasting, who tests it?
> > _______________________________________________
> > pLog-svn mailing list
> > pLog-svn at devel.lifetype.net
> > http://devel.lifetype.net/mailman/listinfo/plog-svn
>
> _______________________________________________
> pLog-svn mailing list
> pLog-svn at devel.lifetype.net
> http://devel.lifetype.net/mailman/listinfo/plog-svn
>
>
> _______________________________________________
> pLog-svn mailing list
> pLog-svn at devel.lifetype.net
> http://devel.lifetype.net/mailman/listinfo/plog-svn
>
> _______________________________________________
> pLog-svn mailing list
> pLog-svn at devel.lifetype.net
> http://limedaley.com/mailman/listinfo/plog-svn
>

More information about the pLog-svn mailing list