[pLog-svn] Process size
Ayalon
ayalon at blog.nl
Fri Mar 2 08:06:00 EST 2007
Hi Oscar,
Thanks for your quick reply!
Yes, I'm aware of those settings and they're all correct.
The problem is that when I call
subdomain.domain.com/category/year/month/date from a browser I see cache
lite trying to grab all the articles from the database, which result in a to
big set. The url doesn't exist and is not in my url scheme. The problem is
that some searchengines etc are crawling the dirs.
In older versions this was not happening btw. From 1.1 it started.
-----Oorspronkelijk bericht-----
Van: plog-svn-bounces at devel.lifetype.net
[mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
Verzonden: vrijdag 2 maart 2007 13:47
Aan: plog-svn at devel.lifetype.net
Onderwerp: Re: [pLog-svn] Process size
Can you tell us how long your apache processes are running?
Specifically, look at the MaxRequestsPerChild parameter
(http://httpd.apache.org/docs/1.3/mod/core.html#maxrequestsperchild)
and I recently checked a site where they had this value to over 1000,
and a number of childs of over 100. As a result they had Apache
processes that were over 1 week old and consuming over 100mb of
memory, while the new ones only used about 15-20mb.
You should set this parameter to a relatively low value. It's pretty
much a fact that PHP leaks memory per every request so by setting this
value to a lower number you will ensure that apache processes are
periodically killed and their memory freed.
There's also other Apache parameter you should look at,
MaxSpareServers, MinSpareServers, StartServers, etc.
On 3/2/07, Ayalon <ayalon at blog.nl> wrote:
> The following problem also excist in 1.2 with an out of the box
> installation:
>
>
> Hi All,
>
> Concerning this issue I made a full test enviroment availble for custom
> url's. Wildcards dns etc is there, default lifetype is there, the error is
> there;)..
>
> Who wants access, send me private the details:
>
> Ip of your connection
>
> The I'll send login details in return..
>
>
>
> -----Oorspronkelijk bericht-----
> Van: plog-svn-bounces at devel.lifetype.net
> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Ayalon
> Verzonden: vrijdag 15 december 2006 9:38
> Aan: plog-svn at devel.lifetype.net
> Onderwerp: RE: [pLog-svn] Process size
>
> Oscar,
>
> Do you need assistance for this? Let me know if it works and if I can help
> you put in the good values and files.
>
> Is there not a possibilty to have wildcard dns enabled on one of the
> lifetype server and test is this way? A central good test enviroment for
all
> scenrario's.
>
> Regards
>
> -----Oorspronkelijk bericht-----
> Van: plog-svn-bounces at devel.lifetype.net
> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
> Verzonden: vrijdag 15 december 2006 7:45
> Aan: plog-svn at devel.lifetype.net
> Onderwerp: Re: [pLog-svn] Process size
>
> Hmm, I will see what I can do in my local instance...
>
> On 15 Dec 2006, at 03:02, Jon Daley wrote:
>
> > It is probably just his custom URL settings, and you don't need
his
> > server? I guess maybe you need his server to see what it dies, but
> > you could at least analyze what happens when adding the extra slash -
> > see what articles show up, and if all the articles in the blog get
> > listed, then you at least know it is wrong. It is probably too hard
> > to try to debug it remotely.
> >
> > On Fri, 15 Dec 2006, Oscar Renalias wrote:
> >
> >> No, I can't really send you anything because I've got nothing to
> >> send. In fact I don't know where to start, and my comments was more
> >> along the lines of whether any of us could be granted access to the
> >> server for debugging purposes instead of sending the scripts to you,
> >> then waiting to hear form you, analyze the results, rinse and repeat.
> >>
> >> On 14 Dec 2006, at 23:20, Ayalon wrote:
> >>
> >>> Ok, I'll ask to make it possible!
> >>> Is there any debug code I can run? Xdebug is not availble on the
> >>> server. You have some kind of debug scripts you can send me to test
> >>> out?
> >>> -----Oorspronkelijk bericht-----
> >>> Van: plog-svn-bounces at devel.lifetype.net
> >>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
> >>> Verzonden: donderdag 14 december 2006 22:16
> >>> Aan: plog-svn at devel.lifetype.net
> >>> Onderwerp: Re: [pLog-svn] Process size At the very least, we should
> >>> be able to upload classes with debug statements and be able to
> >>> browse the logs, so that we can see what's going on. It would be
> >>> even better if we could install something like xdebug for tracing
> >>> the code but we can also manage with debug code.
> >>> On 14 Dec 2006, at 20:49, Ayalon wrote:
> >>>> Yes I have a testserver, and yes it happens with every request like
> >>>> this.
> >>>> And it only happens if a categorie excist and is called with the
> >>>> full subdomain.domain.com/category/year/month/date/ So taking of
> >>>> the postname (slug), if I do like
> >>>> subdomain.domain.com/category/year/month/date
> >>>> it comes
> >>>> up with the mainpage subdomain.domain.com/category/year/month/ it
> >>>> comes up with the mainpage subdomain.domain.com/category/year/month
> >>>> aswell etc etc..
> >>>> If I look in the cache it looks like it's trying to load all the
> >>>> posts etc in the database...
> >>>> what kind of access do you need?
> >>>> As it is a managed server I only have ftp access at the
> >>>> testmachine..
> >>>> -----Oorspronkelijk bericht-----
> >>>> Van: plog-svn-bounces at devel.lifetype.net
> >>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
> >>>> Verzonden: donderdag 14 december 2006 19:35
> >>>> Aan: plog-svn at devel.lifetype.net
> >>>> Onderwerp: Re: [pLog-svn] Process size Do you have some sort of
> >>>> test/staging server with the same exact data as your live server
> >>>> that we could use to play around with this?
> >>>> How about the logs, could you analyze them to find out whether each
> >>>> single occurence of memory problems was related to this? Or the
> >>>> other way around, did all requests like this one cause memory
> >>>> issues?
> >>>> On 14 Dec 2006, at 16:41, Ayalon wrote:
> >>>>> Well what i did, as you can see in the previous screenshots, i
> >>>>> removed all the /blog/{blogname} from the links. Thats it.
> >>>>> In my htaccess is only the forcetype part and error.php part. No
> >>>>> rewrite rules nothing..
> >>>>> Why does those url's have to do a redirect? I don't understand,
> >>>>> and with lifetype 1.0.6 and previous it was working perfectly.
> >>>>> (i'm using lifetype for years;))
> >>>>> ----- Original Message ----- From: "Jon Daley"
> >>>>> <plogworld at jon.limedaley.com>
> >>>>> To: "LifeType SVN" <plog-svn at devel.lifetype.net>
> >>>>> Sent: Thursday, December 14, 2006 3:29 PM
> >>>>> Subject: Re: [pLog-svn] Process size Ah - you don't have
> >>>>> /category/{category} in your URLs.
> >>>>> So
> >>>>> http://amsterdam.blog.nl/toerist_in_eigen_stad/2006/12/12/
> >>>>> economie-
> >>>>> amsterdam-staat-er-goed-voor
> >>>>> has to do a redirect. What is in your .htaccess? I am not seeing
> >>>>> a 400 returned, which means you either wrote some custom rules, or
> >>>>> the modrewrite stuff is being matched.
> >>>>> You have read this, right?
> >>>>> http://wiki.lifetype.net/index.php/Custom_URLs#Why_removing_.
> >>>>> 2Fblog.
> >>>>> 2F_from_the_URLs_is_evil
> >>>>> I bet if you put the URLs back to the defaults, your problem will
> >>>>> go away.
> >>>>> We should probably write a custom URL validator or something to
> >>>>> check when people write bad URLs, though I am not sure how it is
> >>>>> parsing it when you have the extra slash.
> >>>>> On Thu, 14 Dec 2006, Ayalon wrote:
> >>>>>> Ok, yours is fixed, now my problem..
> >>>>>> A not excisting url with the format categorie/year/month/day/ is
> >>>>>> causing memory errors..
> >>>>>> ----- Original Message ----- From: "Jon Daley"
> >>>>>> <plogworld at jon.limedaley.com>
> >>>>>> To: <plog-svn at devel.lifetype.net>
> >>>>>> Sent: Thursday, December 14, 2006 3:10 PM
> >>>>>> Subject: Re: [pLog-svn] Process size The slashes are now enabled,
> >>>>>> that is just a configuration choice, and I hadn't ever noticed
> >>>>>> before.
> >>>>>> I changed them to:
> >>>>>> category_link_format
> >>>>>> /category/{catname}/?$
> >>>>>> archive_link_format
> >>>>>> /archives/{year}/?{month}/?({day}/?)?
> >>>>>> actually, the archive format didn't work like that, because it
> >>>>>> started generating URLs like:
> >>>>>> http://jon.limedaley.com/plog/archives/2006/01//
> >>>>>> which caused it to only show one day's worth of posts.
> >>>>>> So, my archive_link_format is:
> >>>>>> /archives/{year}/?{month}/?{day}?$
> >>>>>> slash removed to avoid the double slash problem, and the $ added
> >>>>>> to distinguish from my permalink entry:
> >>>>>> /archives/{year}/{month}/{day}/{postname}$
> >>>>>> Probably it was matching with a null postname or something.
> >>>>>> In all of that, though, I never had any memory problems. I
> >>>>>> believe I have it currently set to 16MB. This is enough to show
> >>>>>> lots of comments on one page, but not to show thousands, and
> >>>>>> small enough to not kill the server if lots of pages are viewed
> >>>>>> simultaneously.
> >>>>>> On Thu, 14 Dec 2006, Ayalon wrote:
> >>>>>>> Using subdomain enabled, the rest is in the screenshot.
> >>>>>>> ----------------------------------------------------------------
> >>>>>>> ---
> >>>>>>> -
> >>>>>>> ---------------------------------------------------------------
> >>>>>>> Jon:
> >>>>>>> Also in your config there minor issues related to url's:
> >>>>>>> http://jon.limedaley.com/plog/category/pregnancy will show the
> >>>>>>> categorie http://jon.limedaley.com/plog/category/pregnancy/ is
> >>>>>>> showing the main page At my situation yahoo bot is for some
> >>>>>>> reason crawling:
> >>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07 (not a correct
> >>>>>>> url, and shows
> >>>>>>> mainpage)
> >>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07/ (not a correct
> >>>>>>> url and gets a mem error) Obvious yahoo crawler is doing
> >>>>>>> incorrect things, but this can't ofcourse get there serious
> >>>>>>> errors...
> >>>>>>> ----- Original Message ----- From: "Jon Daley"
> >>>>>>> <plogworld at jon.limedaley.com>
> >>>>>>> To: <plog-svn at devel.lifetype.net>
> >>>>>>> Sent: Thursday, December 14, 2006 1:50 PM
> >>>>>>> Subject: Re: [pLog-svn] Process size What are your custom URL
> >>>>>>> settings set to? I can't duplicate this on my blog.
> >>>>>>> On Thu, 14 Dec 2006, Ayalon wrote:
> >>>>>>>> First i'm running freedbsd 6.0 with php 5.1.6 and apache 2.2.2
> >>>>>>>> The exact url, i make an exampe:
> >>>>>>>> i'm running a subdomain config with straight categories behind
> >>>>>>>> it.
> >>>>>>>> so: subdomain.domain.com/categorie This ofcourse excist, if i
> >>>>>>>> type a categorie behind it that doesn ´t excist it comes up
> >>>>>>>> with an error.
> >>>>>>>> Now i call a categorie but with a archive page look:
> >>>>>>>> subdomain.domain.com/categorie/year/month/date This works fine.
> >>>>>>>> But yahoo is searching it with a / behind the url:
> >>>>>>>> subdomain.domain.com/categorie/year/month/date/ The thing is
> >>>>>>>> that this will work, but if you put a / behind the url it's not
> >>>>>>>> working anymore and is causing memory errors etc.
> >>>>>>>> for example:
> >>>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07 works fine
> >>>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07/ gets errors in
> >>>>>>>> the log and out of memory issues, please don't click it :)
> >>>>>>>> Anybody an idea how to fix this?
> >>>>>>>> ----- Original Message ----- From: "Oscar Renalias"
> >>>>>>>> <oscar at renalias.net>
> >>>>>>>> To: <plog-svn at devel.lifetype.net>
> >>>>>>>> Sent: Thursday, December 14, 2006 9:51 AM
> >>>>>>>> Subject: Re: [pLog-svn] Process size
> >>>>>>>>> Can you provide the exact URLs? Or even better, the exact line
> >>>>>>>>> (s) from
> >>>>>>>>> the
> >>>>>>>>> logs.
> >>>>>>>>> How about your version of PHP? Are you running PHP 4 or PHP 5?
> >>>>>>>>> On 12/14/06, Ayalon <ayalon at blog.nl> wrote:
> >>>>>>>>>> Found it!
> >>>>>>>>>> Searched the logs for the yahoobot. It looks like yahoo is
> >>>>>>>>>> crawling a lot of pages that don't exist.
> >>>>>>>>>> for example blogname.mydomain.com/categoryname/year/month/
> >>>>>>>>>> date
> >>>>>>>>>> While a normal yahoo request is passing by normally. This
> >>>>>>>>>> page doesn't excist in my enviroment and this will cause some
> >>>>>>>>>> kind of loop in the script with eventually causing a lot of
> >>>>>>>>>> mem in php, a lot of buffering in apache etc. I looked in the
> >>>>>>>>>> php and apache forums and bug reports, but can't find
> >>>>>>>>>> anything. Is there something wrong in the way not excisting
> >>>>>>>>>> url's etc are being handled? Because i see in my logs normal
> >>>>>>>>>> request are also handle correct. Pages, and especially within
> >>>>>>>>>> subdirs are causing these problems..
> >>>>>>>>>> If you need more info, i'm happy to give. Probably also devel
> >>>>>>>>>> had this problem last weekend...
> >>>>>>>>>> ----- Original Message -----
> >>>>>>>>>> From: "Ayalon" <ayalon at blog.nl>
> >>>>>>>>>> To: <plog-svn at devel.lifetype.net>
> >>>>>>>>>> Sent: Thursday, December 14, 2006 8:32 AM
> >>>>>>>>>> Subject: RE: [pLog-svn] Process size
> >>>>>>>>>>> Hi Oscar,
> >>>>>>>>>>> Thanks for your email. But anyway, we're not allowing apache
> >>>>>>>>>> to grow > >
> >>>>>>>>>> so
> >>>>>>>>>>> big,
> >>>>>>>>>>> it's php with e mem limit of 32. If I try to run it with a
> >>>>>>>>>> lower >
> >>>>>>>>>> value
> >>>>>>>>>> the
> >>>>>>>>>>> pages with comments are not shown (100+ comments). So do I
> >>>>>>>>>> have >
> >>>>>>>>>> another
> >>>>>>>>>>> choice?
> >>>>>>>>>>> Jon: I'll true to get the logs to show you what yahoo is
> >>>>>>>>>> hitting..
> >>>>>>>>>>> Jon2: Can you see anything on devel?
> >>>>>>>>>>> -----Oorspronkelijk bericht-----
> >>>>>>>>>>> Van: plog-svn-bounces at devel.lifetype.net
> >>>>>>>>>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar
> >>>>>>>>>> Renalias
> >>>>>>>>>>> Verzonden: donderdag 14 december 2006 8:28
> >>>>>>>>>>> Aan: plog-svn at devel.lifetype.net
> >>>>>>>>>>> Onderwerp: Re: [pLog-svn] Process size In addition to all I
> >>>>>>>>>>> said before, why do you allow up to
> >>>>>>>>>> 32mb to > each
> >>>>>>>>>>> Apache
> >>>>>>>>>>> project? I think that's too much, nowadays 8-12mb should be
> >>>>>>>>>> a more
> >>>>>>>>>>> reasonable figure. If there's memory leaks somewhere in PHP
> >>>>>>>>>> (not in > >
> >>>>>>>>>> our
> >>>>>>>>>>> code, remember that there's no way to explicitely deallocate
> >>>>>>>>>> an >
> >>>>>>>>>> object
> >>>>>>>>>> in
> >>>>>>>>>>> PHP code as far as I know), 32mb isn't exactly going to
> >>>>>>>>>>> help...
> >>>>>>>>>>> On 12/14/06, Oscar Renalias <oscar at renalias.net> wrote:
> >>>>>>>>>>>> No, we're not checking anything. From Lifetype's point of
> >>>>>>>>>> view, we
> >>>>>>>>>>>> don't really care about who is making the request.
> >>>>>>>>>>>> Could it be that the Yahoo blog is performing searches? You
> >>>>>>>>>> should >> >>
> >>>>>>>>>> be
> >>>>>>>>>>>> able to see what kind of requests the crawler is making by
> >>>>>>>>>>>> >>
> >>>>>>>>>> referencing
> >>>>>>>>>>>> the timestamps you posted below from apache's error log
> >>>>>>>>>> file with >> >>
> >>>>>>>>>> the
> >>>>>>>>>>>> data you've got in the access log. The apache access log
> >>>>>>>>>> should
> >>>>>>>>>>>> contain the exact request, please find it and post it here,
> >>>>>>>>>>>> >>
> >>>>>>>>>> otherwise
> >>>>>>>>>>>> we're just guessing.
> >>>>>>>>>>>> On 14 Dec 2006, at 00:15, Ayalon wrote:
> >>>>>>>>>>>>> Nope, there's nobody who has so many post in the mainpage
> >>>>>>>>>> as I
> >>>>>>>>>>>>> configure everybody's blog.
> >>>>>>>>>>>>> It's not just an issue with all bots, it's only with
> >>>>>>>>>> yahoo bot. >> > >>
> >>>>>>>>>> When
> >>>>>>>>>>>>> searching the net I found more similair problems. No
> >>>>>>>>>> check is >> > done
> >>>>>>>>>>>>> who is coming on the site or something??
> >>>>>>>>>>>>> -----Oorspronkelijk bericht-----
> >>>>>>>>>>>>> Van: plog-svn-bounces at devel.lifetype.net
> >>>>>>>>>>>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar
> >>>>>>>>>>>>> Renalias
> >>>>>>>>>>>>> Verzonden: woensdag 13 december 2006 23:10
> >>>>>>>>>>>>> Aan: plog-svn at devel.lifetype.net
> >>>>>>>>>>>>> Onderwerp: Re: [pLog-svn] Process size I guess it's the
> >>>>>>>>>>>>> same issue we had in devel.lifetype.net.
> >>>>>>>>>>>>> One thing you should check is whether you've got any user
> >>>>>>>>>> who has
> >>>>>>>>>>>>> configured his/her blog to display something like 80 or
> >>>>>>>>>> 100 posts
> >>>>>>>>>>> in
> >>>>>>>>>>>>> the front page, as that can cause performance problems.
> >>>>>>>>>> There's
> >>>>>>>>>>>>> already a fix for that in LT 1.2, but in the meantime you
> >>>>>>>>>> will >> >> > >
> >>>>>>>>>> have
> >>>>>>>>>>>>> to keep an eye on it on your own.
> >>>>>>>>>>>>> Otherwise a crawler performs the exact same operations as
> >>>>>>>>>> a user >> > >>
> >>>>>>>>>>> via
> >>>>>>>>>>>>> a browser would do.
> >>>>>>>>>>>>> On 13 Dec 2006, at 20:48, Ayalon wrote:
> >>>>>>>>>>>>>> Ok, I found the problem, and I tell you it's really
> >>>>>>>>>> strange but >> >> >>
> >>>>>>>>>>>> true:
> >>>>>>>>>>>>>> The yahoo bot (inktomi bot) is hitting my site and then
> >>>>>>>>>>>>>> the Cache_lite.php is for some reason using to much
> >>>>>>>>>>>>>> memory:
> >>>>>>>>>>>>>> [Tue Dec 12 00:01:44 2006] [error] [client 74.6.85.156]
> >>>>>>>>>> PHP >> >> Fatal
> >>>>>>>>>>>>>> error:
> >>>>>>>>>>>>>> Allowed memory size of 33554432 bytes exhausted (tried
> >>>>>>>>>> to >> >>
> >>>>>>>>>> allocate
> >>>>>>>>>>>>>> 84 bytes)
> >>>>>>>>>>>>>> in /data/www/www.blog.nl/class/cache/Cache_Lite/Lite.php
> >>>>>>>>>> on line
> >>>>>>>>>>>>>> 352
> >>>>>>>>>>>>>> [Tue Dec 12 00:05:27 2006] [error] [client 74.6.86.205]
> >>>>>>>>>> PHP >> >> Fatal
> >>>>>>>>>>>>>> error:
> >>>>>>>>>>>>>> Allowed memory size of 33554432 bytes exhausted (tried
> >>>>>>>>>> to >> >>
> >>>>>>>>>> allocate
> >>>>>>>>>>>>>> 93 bytes)
> >>>>>>>>>>>>>> in /data/www/www.blog.nl/class/cache/Cache_Lite/Lite.php
> >>>>>>>>>> on line
> >>>>>>>>>>>>>> 352
> >>>>>>>>>>>>>> When a normal user is hitting the site, there's nothing
> >>>>>>>>>> at all.
> >>>>>>>>>>>>>> When the bot
> >>>>>>>>>>>>>> is hitting my site this is happening with a lot of the
> >>>>>>>>>> requests.
> >>>>>>>>>>>>>> Also the
> >>>>>>>>>>>>>> apache process is growing so big that at the end the
> >>>>>>>>>> process is
> >>>>>>>>>>>>>> using so much memory that it's starting to use the swap.
> >>>>>>>>>>>>>> What can cause this problem? Now I blocked yahoo bot via
> >>>>>>>>>>>>>> htaccess and the problem is not there anymore. It started
> >>>>>>>>>>>>>> to
> >>>>>>>>>> happen after
> >>>>>>>>>>>>>> the upgrade to the new lifetype platform. Are there some
> >>>>>>>>>> checks >> >> >>
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>> who is coming in?
> >>>>>>>>>>>>>> Do you
> >>>>>>>>>>>>>> need more data?
> >>>>>>>>>>>>>> Anyway it's strange and interesting, anybody an
> >>>>>>>>>>>>>> idea....??
> >>>>>>>>>>>>>> -----Oorspronkelijk bericht-----
> >>>>>>>>>>>>>> Van: plog-svn-bounces at devel.lifetype.net
> >>>>>>>>>>>>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Jon
> >>>>>>>>>> Daley
> >>>>>>>>>>>>>> Verzonden: woensdag 13 december 2006 17:06
> >>>>>>>>>>>>>> Aan: plog-svn at devel.lifetype.net
> >>>>>>>>>>>>>> Onderwerp: RE: [pLog-svn] Process size On Wed, 13 Dec
> >>>>>>>>>>>>>> 2006, Ayalon wrote:
> >>>>>>>>>>>>>>> Where can I find the post of the rewrite?
> >>>>>>>>>> http://forums.lifetype.net/viewtopic.php?
> >>>>>>>>>> p=23240&highlight=htaccess
> >>>>>>>>>>>>>> +rewrite+
> >>>>>>>>>>>>>> error
> >>>>>>>>>>>>>>> Anyway my provider is telling me that it looks like
> >>>>>>>>>> lifetype >> >>> has
> >>>>>>>>>>>>>>> memoryleaks in various aspects of the script, starting
> >>>>>>>>>> with >> >> >>> >>
> >>>>>>>>>> caching.
> >>>>>>>>>>>>>>> Is
> >>>>>>>>>>>>>> that possible?
> >>>>>>>>>>>>>> It is certainly possible. I would expect to see
> >>>>>>>>>> more >> >> memory
> >>>>>>>>>>>>>> usage in my setup if that were the case.
> >>>>>>>>>>>>>>> Is it possible to disable everything related to cache
> >>>>>>>>>> lift? >> >>> Just
> >>>>>>>>>>>>>>> to test some things...
> >>>>>>>>>>>>>> Line 39 of class/cache/cachemanager.class.php.
> >>>>>>>>>>>>>> Change
> >>>>>>>>>>>>>> $cacheEnable to false. I think that should do the trick.
> >>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>>> _____________________________________________________________
> >>>>>>>>>> ___
> >>>>>>>>>> _
> >>>>>>>>>> _____
> >>>>>>>>>> This email has been scanned by the MessageLabs Email Security
> >>>>>>>>>> System.
> >>>>>>>>>> For more information please visit http://www.messagelabs.com/
> >>>>>>>>>> email
> >>>>>>>>>> _____________________________________________________________
> >>>>>>>>>> ___
> >>>>>>>>>> _
> >>>>>>>>>> _____
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> pLog-svn mailing list
> >>>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>>> _______________________________________________
> >>>>>>>>> pLog-svn mailing list
> >>>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>>> _______________________________________________________________
> >>>>>>>> ___
> >>>>>>>> _
> >>>>>>>> ___
> >>>>>>>> This email has been scanned by the MessageLabs Email Security
> >>>>>>>> System.
> >>>>>>>> For more information please visit http://www.messagelabs.com/
> >>>>>>>> email
> >>>>>>>> _______________________________________________________________
> >>>>>>>> ___
> >>>>>>>> _
> >>>>>>>> ___
> >>>>>>>> _______________________________________________
> >>>>>>>> pLog-svn mailing list
> >>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>> --
> >>>>>>> Jon Daley
> >>>>>>> http://jon.limedaley.com/
> >>>>>>> The difference between genius and stupidity is that genius has
> >>>>>>> its limits.
> >>>>>>> -- Anonymous
> >>>>>>> ----------------------------------------------------------------
> >>>>>>> ---
> >>>>>>> -
> >>>>>>> ------------
> >>>>>>>> _______________________________________________
> >>>>>>>> pLog-svn mailing list
> >>>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>>> ________________________________________________________________
> >>>>>>> ___
> >>>>>>> _
> >>>>>>> __
> >>>>>>> This email has been scanned by the MessageLabs Email Security
> >>>>>>> System.
> >>>>>>> For more information please visit http://www.messagelabs.com/
> >>>>>>> email
> >>>>>>> ________________________________________________________________
> >>>>>>> ___
> >>>>>>> _
> >>>>>>> __
> >>>>>> --
> >>>>>> Jon Daley
> >>>>>> http://jon.limedaley.com/
> >>>>>> Music: A safe kind of high
> >>>>>> -- Jimi Hendrix
> >>>>>> -----------------------------------------------------------------
> >>>>>> ---
> >>>>>> -
> >>>>>> -----------
> >>>>>>> _______________________________________________
> >>>>>>> pLog-svn mailing list
> >>>>>>> pLog-svn at devel.lifetype.net
> >>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>>> _________________________________________________________________
> >>>>>> ___
> >>>>>> _
> >>>>>> _
> >>>>>> This email has been scanned by the MessageLabs Email Security
> >>>>>> System.
> >>>>>> For more information please visit http://www.messagelabs.com/
> >>>>>> email
> >>>>>> _________________________________________________________________
> >>>>>> ___
> >>>>>> _
> >>>>>> _
> >>>>>> _______________________________________________
> >>>>>> pLog-svn mailing list
> >>>>>> pLog-svn at devel.lifetype.net
> >>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>> --
> >>>>> Jon Daley
> >>>>> http://jon.limedaley.com/
> >>>>> If everything is coming your way then you're in the wrong lane.
> >>>>> -- Anonymous
> >>>>> ------------------------------------------------------------------
> >>>>> ---
> >>>>> -
> >>>>> ----------
> >>>>>> _______________________________________________
> >>>>>> pLog-svn mailing list
> >>>>>> pLog-svn at devel.lifetype.net
> >>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>>> __________________________________________________________________
> >>>>> ___
> >>>>> _
> >>>>> This email has been scanned by the MessageLabs Email Security
> >>>>> System.
> >>>>> For more information please visit http://www.messagelabs.com/email
> >>>>> __________________________________________________________________
> >>>>> ___
> >>>>> _
> >>>>> _______________________________________________
> >>>>> pLog-svn mailing list
> >>>>> pLog-svn at devel.lifetype.net
> >>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>> _______________________________________________
> >>>> pLog-svn mailing list
> >>>> pLog-svn at devel.lifetype.net
> >>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>>> _______________________________________________
> >>>> pLog-svn mailing list
> >>>> pLog-svn at devel.lifetype.net
> >>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>> _______________________________________________
> >>> pLog-svn mailing list
> >>> pLog-svn at devel.lifetype.net
> >>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>> _______________________________________________
> >>> pLog-svn mailing list
> >>> pLog-svn at devel.lifetype.net
> >>> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >>
> >> _______________________________________________
> >> pLog-svn mailing list
> >> pLog-svn at devel.lifetype.net
> >> http://devel.lifetype.net/mailman/listinfo/plog-svn
> >
> > --
> > Jon Daley
> > http://jon.limedaley.com/
> >
> > When dog food is new and improved tasting, who tests it?
> > _______________________________________________
> > pLog-svn mailing list
> > pLog-svn at devel.lifetype.net
> > http://devel.lifetype.net/mailman/listinfo/plog-svn
>
> _______________________________________________
> pLog-svn mailing list
> pLog-svn at devel.lifetype.net
> http://devel.lifetype.net/mailman/listinfo/plog-svn
>
>
> _______________________________________________
> pLog-svn mailing list
> pLog-svn at devel.lifetype.net
> http://devel.lifetype.net/mailman/listinfo/plog-svn
>
> _______________________________________________
> pLog-svn mailing list
> pLog-svn at devel.lifetype.net
> http://limedaley.com/mailman/listinfo/plog-svn
>
_______________________________________________
pLog-svn mailing list
pLog-svn at devel.lifetype.net
http://limedaley.com/mailman/listinfo/plog-svn
More information about the pLog-svn
mailing list