[pLog-svn] Process size

Ayalon ayalon at blog.nl
Fri Mar 2 07:41:07 EST 2007 

The following problem also excist in 1.2 with an out of the box
installation:


Hi All,

Concerning this issue I made a full test enviroment availble for custom
url's. Wildcards dns etc is there, default lifetype is there, the error is
there;)..

Who wants access, send me private the details:

Ip of your connection

The I'll send login details in return..

 

-----Oorspronkelijk bericht-----
Van: plog-svn-bounces at devel.lifetype.net
[mailto:plog-svn-bounces at devel.lifetype.net] Namens Ayalon
Verzonden: vrijdag 15 december 2006 9:38
Aan: plog-svn at devel.lifetype.net
Onderwerp: RE: [pLog-svn] Process size

Oscar,

Do you need assistance for this? Let me know if it works and if I can help
you put in the good values and files. 

Is there not a possibilty to have wildcard dns enabled on one of the
lifetype server and test is this way? A central good test enviroment for all
scenrario's.

Regards 

-----Oorspronkelijk bericht-----
Van: plog-svn-bounces at devel.lifetype.net
[mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
Verzonden: vrijdag 15 december 2006 7:45
Aan: plog-svn at devel.lifetype.net
Onderwerp: Re: [pLog-svn] Process size

Hmm, I will see what I can do in my local instance...

On 15 Dec 2006, at 03:02, Jon Daley wrote:

> 	It is probably just his custom URL settings, and you don't need his 
> server?  I guess maybe you need his server to see what it dies, but 
> you could at least analyze what happens when adding the extra slash - 
> see what articles show up, and if all the articles in the blog get 
> listed, then you at least know it is wrong.  It is probably too hard 
> to try to debug it remotely.
>
> On Fri, 15 Dec 2006, Oscar Renalias wrote:
>
>> No, I can't really send you anything because I've got nothing to 
>> send. In fact I don't know where to start, and my comments was more 
>> along the lines of whether any of us could be granted access to the 
>> server for debugging purposes instead of sending the scripts to you, 
>> then waiting to hear form you, analyze the results, rinse and repeat.
>>
>> On 14 Dec 2006, at 23:20, Ayalon wrote:
>>
>>> Ok, I'll ask to make it possible!
>>> Is there any debug code I can run? Xdebug is not availble on the 
>>> server. You have some kind of debug scripts you can send me to test 
>>> out?
>>> -----Oorspronkelijk bericht-----
>>> Van: plog-svn-bounces at devel.lifetype.net
>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
>>> Verzonden: donderdag 14 december 2006 22:16
>>> Aan: plog-svn at devel.lifetype.net
>>> Onderwerp: Re: [pLog-svn] Process size At the very least, we should 
>>> be able to upload classes with debug statements and be able to 
>>> browse the logs, so that we can see what's going on. It would be 
>>> even better if we could install something like xdebug for tracing 
>>> the code but we can also manage with debug code.
>>> On 14 Dec 2006, at 20:49, Ayalon wrote:
>>>> Yes I have a testserver, and yes it happens with every request like 
>>>> this.
>>>> And it only happens if a categorie excist and is called with the 
>>>> full subdomain.domain.com/category/year/month/date/ So taking of 
>>>> the postname (slug), if I do like 
>>>> subdomain.domain.com/category/year/month/date
>>>> it comes
>>>> up with the mainpage subdomain.domain.com/category/year/month/ it 
>>>> comes up with the mainpage subdomain.domain.com/category/year/month
>>>> aswell etc etc..
>>>> If I look in the cache it looks like it's trying to load all the 
>>>> posts etc in the database...
>>>> what kind of access do you need?
>>>> As it is a managed server I only have ftp access at the 
>>>> testmachine..
>>>> -----Oorspronkelijk bericht-----
>>>> Van: plog-svn-bounces at devel.lifetype.net
>>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar Renalias
>>>> Verzonden: donderdag 14 december 2006 19:35
>>>> Aan: plog-svn at devel.lifetype.net
>>>> Onderwerp: Re: [pLog-svn] Process size Do you have some sort of 
>>>> test/staging server with the same exact data as your live server 
>>>> that we could use to play around with this?
>>>> How about the logs, could you analyze them to find out whether each 
>>>> single occurence of memory problems was related to this? Or the 
>>>> other way around, did all requests like this one cause memory 
>>>> issues?
>>>> On 14 Dec 2006, at 16:41, Ayalon wrote:
>>>>> Well what i did, as you can see in the previous screenshots, i 
>>>>> removed all the  /blog/{blogname} from the links. Thats it.
>>>>> In my htaccess is only the forcetype part and error.php part. No 
>>>>> rewrite rules nothing..
>>>>> Why does those url's have to do a redirect? I don't understand, 
>>>>> and with lifetype 1.0.6 and previous it was working perfectly.
>>>>> (i'm using lifetype for years;))
>>>>> ----- Original Message ----- From: "Jon Daley"
>>>>> <plogworld at jon.limedaley.com>
>>>>> To: "LifeType SVN" <plog-svn at devel.lifetype.net>
>>>>> Sent: Thursday, December 14, 2006 3:29 PM
>>>>> Subject: Re: [pLog-svn] Process size Ah - you don't have 
>>>>> /category/{category} in your URLs.
>>>>> So
>>>>> http://amsterdam.blog.nl/toerist_in_eigen_stad/2006/12/12/
>>>>> economie-
>>>>> amsterdam-staat-er-goed-voor
>>>>> has to do a redirect.  What is in your .htaccess?  I am not seeing 
>>>>> a 400 returned, which means you either wrote some custom rules, or 
>>>>> the modrewrite stuff is being matched.
>>>>> You have read this, right?
>>>>> http://wiki.lifetype.net/index.php/Custom_URLs#Why_removing_. 
>>>>> 2Fblog.
>>>>> 2F_from_the_URLs_is_evil
>>>>> I bet if you put the URLs back to the defaults, your problem will 
>>>>> go away.
>>>>> We should probably write a custom URL validator or something to 
>>>>> check when people write bad URLs, though I am not sure how it is 
>>>>> parsing it when you have the extra slash.
>>>>> On Thu, 14 Dec 2006, Ayalon wrote:
>>>>>> Ok, yours is fixed, now my problem..
>>>>>> A not excisting url with the format categorie/year/month/day/ is 
>>>>>> causing memory errors..
>>>>>> ----- Original Message ----- From: "Jon Daley"
>>>>>> <plogworld at jon.limedaley.com>
>>>>>> To: <plog-svn at devel.lifetype.net>
>>>>>> Sent: Thursday, December 14, 2006 3:10 PM
>>>>>> Subject: Re: [pLog-svn] Process size The slashes are now enabled, 
>>>>>> that is just a configuration choice, and I hadn't ever noticed 
>>>>>> before.
>>>>>> I changed them to:
>>>>>> category_link_format
>>>>>> /category/{catname}/?$
>>>>>> archive_link_format
>>>>>> /archives/{year}/?{month}/?({day}/?)?
>>>>>> actually, the archive format didn't work like that, because it 
>>>>>> started generating URLs like:
>>>>>> http://jon.limedaley.com/plog/archives/2006/01//
>>>>>> which caused it to only show one day's worth of posts.
>>>>>> So, my archive_link_format is:
>>>>>> /archives/{year}/?{month}/?{day}?$
>>>>>> slash removed to avoid the double slash problem, and the $ added 
>>>>>> to distinguish from my permalink entry:
>>>>>> /archives/{year}/{month}/{day}/{postname}$
>>>>>> Probably it was matching with a null postname or something.
>>>>>> In all of that, though, I never had any memory problems.  I 
>>>>>> believe I have it currently set to 16MB.  This is enough to show 
>>>>>> lots of comments on one page, but not to show thousands, and 
>>>>>> small enough to not kill the server if lots of pages are viewed 
>>>>>> simultaneously.
>>>>>> On Thu, 14 Dec 2006, Ayalon wrote:
>>>>>>> Using subdomain enabled, the rest is in the screenshot.
>>>>>>> ----------------------------------------------------------------
>>>>>>> ---
>>>>>>> -
>>>>>>> ---------------------------------------------------------------
>>>>>>> Jon:
>>>>>>> Also in your config there minor issues related to url's:
>>>>>>> http://jon.limedaley.com/plog/category/pregnancy will show the 
>>>>>>> categorie http://jon.limedaley.com/plog/category/pregnancy/ is 
>>>>>>> showing the main page At my situation yahoo bot is for some 
>>>>>>> reason crawling:
>>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07 (not a correct 
>>>>>>> url, and shows
>>>>>>> mainpage)
>>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07/ (not a correct 
>>>>>>> url and gets a mem error) Obvious yahoo crawler is doing 
>>>>>>> incorrect things, but this can't ofcourse get there serious 
>>>>>>> errors...
>>>>>>> ----- Original Message ----- From: "Jon Daley"
>>>>>>> <plogworld at jon.limedaley.com>
>>>>>>> To: <plog-svn at devel.lifetype.net>
>>>>>>> Sent: Thursday, December 14, 2006 1:50 PM
>>>>>>> Subject: Re: [pLog-svn] Process size What are your custom URL 
>>>>>>> settings set to?  I can't duplicate this on my blog.
>>>>>>> On Thu, 14 Dec 2006, Ayalon wrote:
>>>>>>>> First i'm running freedbsd 6.0  with php 5.1.6 and apache 2.2.2 
>>>>>>>> The exact url, i make an exampe:
>>>>>>>> i'm running a subdomain config with straight categories behind 
>>>>>>>> it.
>>>>>>>> so: subdomain.domain.com/categorie This ofcourse excist, if i 
>>>>>>>> type a categorie behind it that doesn ´t excist it comes up 
>>>>>>>> with an error.
>>>>>>>> Now i call a categorie but with a archive page look:
>>>>>>>> subdomain.domain.com/categorie/year/month/date This works fine.
>>>>>>>> But yahoo is searching it with a / behind the url:
>>>>>>>> subdomain.domain.com/categorie/year/month/date/ The thing is 
>>>>>>>> that this will work, but if you put a / behind the url it's not 
>>>>>>>> working anymore and is causing memory errors etc.
>>>>>>>> for example:
>>>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07 works fine 
>>>>>>>> http://amsterdam.blog.nl/amsterdam/2006/12/07/ gets errors in 
>>>>>>>> the log and out of memory issues, please don't click it :) 
>>>>>>>> Anybody an idea how to fix this?
>>>>>>>> ----- Original Message ----- From: "Oscar Renalias"
>>>>>>>> <oscar at renalias.net>
>>>>>>>> To: <plog-svn at devel.lifetype.net>
>>>>>>>> Sent: Thursday, December 14, 2006 9:51 AM
>>>>>>>> Subject: Re: [pLog-svn] Process size
>>>>>>>>> Can you provide the exact URLs? Or even better, the exact line
>>>>>>>>> (s) from
>>>>>>>>> the
>>>>>>>>> logs.
>>>>>>>>> How about your version of PHP? Are you running PHP 4 or PHP 5?
>>>>>>>>> On 12/14/06, Ayalon <ayalon at blog.nl> wrote:
>>>>>>>>>> Found it!
>>>>>>>>>> Searched the logs for the yahoobot. It looks like yahoo is 
>>>>>>>>>> crawling a lot of pages that don't exist.
>>>>>>>>>> for example blogname.mydomain.com/categoryname/year/month/
>>>>>>>>>> date
>>>>>>>>>> While a normal yahoo request is passing by normally. This 
>>>>>>>>>> page doesn't excist in my enviroment and this will cause some 
>>>>>>>>>> kind of loop in the script with eventually causing a lot of 
>>>>>>>>>> mem in php, a lot of buffering in apache etc. I looked in the 
>>>>>>>>>> php and apache forums and bug reports, but can't find 
>>>>>>>>>> anything. Is there something wrong in the way not excisting 
>>>>>>>>>> url's etc are being handled? Because i see in my logs normal 
>>>>>>>>>> request are also handle correct. Pages, and especially within 
>>>>>>>>>> subdirs are causing these problems..
>>>>>>>>>> If you need more info, i'm happy to give. Probably also devel 
>>>>>>>>>> had this problem last weekend...
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Ayalon" <ayalon at blog.nl>
>>>>>>>>>> To: <plog-svn at devel.lifetype.net>
>>>>>>>>>> Sent: Thursday, December 14, 2006 8:32 AM
>>>>>>>>>> Subject: RE: [pLog-svn] Process size
>>>>>>>>>>> Hi Oscar,
>>>>>>>>>>> Thanks for your email. But anyway, we're not allowing apache
>>>>>>>>>> to grow >  >
>>>>>>>>>> so
>>>>>>>>>>> big,
>>>>>>>>>>> it's php with e mem limit of 32. If I try to run it with a
>>>>>>>>>> lower >
>>>>>>>>>> value
>>>>>>>>>> the
>>>>>>>>>>> pages with comments are not shown (100+ comments). So do I
>>>>>>>>>> have >
>>>>>>>>>> another
>>>>>>>>>>> choice?
>>>>>>>>>>> Jon: I'll true to get the logs to show you what yahoo is
>>>>>>>>>> hitting..
>>>>>>>>>>> Jon2: Can you see anything on devel?
>>>>>>>>>>> -----Oorspronkelijk bericht-----
>>>>>>>>>>> Van: plog-svn-bounces at devel.lifetype.net
>>>>>>>>>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar
>>>>>>>>>> Renalias
>>>>>>>>>>> Verzonden: donderdag 14 december 2006 8:28
>>>>>>>>>>> Aan: plog-svn at devel.lifetype.net
>>>>>>>>>>> Onderwerp: Re: [pLog-svn] Process size In addition to all I 
>>>>>>>>>>> said before, why do you allow up to
>>>>>>>>>> 32mb to > each
>>>>>>>>>>> Apache
>>>>>>>>>>> project? I think that's too much, nowadays 8-12mb should be
>>>>>>>>>> a more
>>>>>>>>>>> reasonable figure. If there's memory leaks somewhere in PHP
>>>>>>>>>> (not in >  >
>>>>>>>>>> our
>>>>>>>>>>> code, remember that there's no way to explicitely deallocate
>>>>>>>>>> an >
>>>>>>>>>> object
>>>>>>>>>> in
>>>>>>>>>>> PHP code as far as I know), 32mb isn't exactly going to 
>>>>>>>>>>> help...
>>>>>>>>>>> On 12/14/06, Oscar Renalias <oscar at renalias.net> wrote:
>>>>>>>>>>>> No, we're not checking anything. From Lifetype's point of
>>>>>>>>>> view, we
>>>>>>>>>>>> don't really care about who is making the request.
>>>>>>>>>>>> Could it be that the Yahoo blog is performing searches? You
>>>>>>>>>> should >>  >>
>>>>>>>>>> be
>>>>>>>>>>>> able to see what kind of requests the crawler is making by
>>>>>>>>>>>> >>
>>>>>>>>>> referencing
>>>>>>>>>>>> the timestamps you posted below from apache's error log
>>>>>>>>>> file with >>  >>
>>>>>>>>>> the
>>>>>>>>>>>> data you've got in the access log. The apache access log
>>>>>>>>>> should
>>>>>>>>>>>> contain the exact request, please find it and post it here,
>>>>>>>>>>>> >>
>>>>>>>>>> otherwise
>>>>>>>>>>>> we're just guessing.
>>>>>>>>>>>> On 14 Dec 2006, at 00:15, Ayalon wrote:
>>>>>>>>>>>>> Nope, there's nobody who has so many post in the mainpage
>>>>>>>>>> as I
>>>>>>>>>>>>> configure everybody's blog.
>>>>>>>>>>>>> It's not just an issue with all bots, it's only with
>>>>>>>>>> yahoo bot. >> >  >>
>>>>>>>>>> When
>>>>>>>>>>>>> searching the net I found more similair problems. No
>>>>>>>>>> check is >> > done
>>>>>>>>>>>>> who is coming on the site or something??
>>>>>>>>>>>>> -----Oorspronkelijk bericht-----
>>>>>>>>>>>>> Van: plog-svn-bounces at devel.lifetype.net
>>>>>>>>>>>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Oscar 
>>>>>>>>>>>>> Renalias
>>>>>>>>>>>>> Verzonden: woensdag 13 december 2006 23:10
>>>>>>>>>>>>> Aan: plog-svn at devel.lifetype.net
>>>>>>>>>>>>> Onderwerp: Re: [pLog-svn] Process size I guess it's the 
>>>>>>>>>>>>> same issue we had in devel.lifetype.net.
>>>>>>>>>>>>> One thing you should check is whether you've got any user
>>>>>>>>>> who has
>>>>>>>>>>>>> configured his/her blog to display something like 80 or
>>>>>>>>>> 100 posts
>>>>>>>>>>> in
>>>>>>>>>>>>> the front page, as that can cause performance problems.
>>>>>>>>>> There's
>>>>>>>>>>>>> already a fix for that in LT 1.2, but in the meantime you
>>>>>>>>>> will >> >> >  >
>>>>>>>>>> have
>>>>>>>>>>>>> to keep an eye on it on your own.
>>>>>>>>>>>>> Otherwise a crawler performs the exact same operations as
>>>>>>>>>> a user >> >  >>
>>>>>>>>>>> via
>>>>>>>>>>>>> a browser would do.
>>>>>>>>>>>>> On 13 Dec 2006, at 20:48, Ayalon wrote:
>>>>>>>>>>>>>> Ok, I found the problem, and I tell you it's really
>>>>>>>>>> strange but >> >>  >>
>>>>>>>>>>>> true:
>>>>>>>>>>>>>> The yahoo bot (inktomi bot) is hitting my site and then 
>>>>>>>>>>>>>> the Cache_lite.php is for some reason using to much
>>>>>>>>>>>>>> memory:
>>>>>>>>>>>>>> [Tue Dec 12 00:01:44 2006] [error] [client 74.6.85.156]
>>>>>>>>>> PHP >> >> Fatal
>>>>>>>>>>>>>> error:
>>>>>>>>>>>>>> Allowed memory size of 33554432 bytes exhausted (tried
>>>>>>>>>> to >> >>
>>>>>>>>>> allocate
>>>>>>>>>>>>>> 84 bytes)
>>>>>>>>>>>>>> in /data/www/www.blog.nl/class/cache/Cache_Lite/Lite.php
>>>>>>>>>> on line
>>>>>>>>>>>>>> 352
>>>>>>>>>>>>>> [Tue Dec 12 00:05:27 2006] [error] [client 74.6.86.205]
>>>>>>>>>> PHP >> >> Fatal
>>>>>>>>>>>>>> error:
>>>>>>>>>>>>>> Allowed memory size of 33554432 bytes exhausted (tried
>>>>>>>>>> to >> >>
>>>>>>>>>> allocate
>>>>>>>>>>>>>> 93 bytes)
>>>>>>>>>>>>>> in /data/www/www.blog.nl/class/cache/Cache_Lite/Lite.php
>>>>>>>>>> on line
>>>>>>>>>>>>>> 352
>>>>>>>>>>>>>> When a normal user is hitting the site, there's nothing
>>>>>>>>>> at all.
>>>>>>>>>>>>>> When the bot
>>>>>>>>>>>>>> is hitting my site this is happening with a lot of the
>>>>>>>>>> requests.
>>>>>>>>>>>>>> Also the
>>>>>>>>>>>>>> apache process is growing so big that at the end the
>>>>>>>>>> process is
>>>>>>>>>>>>>> using so much memory that it's starting to use the swap.
>>>>>>>>>>>>>> What can cause this problem? Now I blocked yahoo bot via 
>>>>>>>>>>>>>> htaccess and the problem is not there anymore. It started 
>>>>>>>>>>>>>> to
>>>>>>>>>> happen after
>>>>>>>>>>>>>> the upgrade to the new lifetype platform. Are there some
>>>>>>>>>> checks >> >>  >>
>>>>>>>>>>>> for
>>>>>>>>>>>>>> who is coming in?
>>>>>>>>>>>>>> Do you
>>>>>>>>>>>>>> need more data?
>>>>>>>>>>>>>> Anyway it's strange and interesting, anybody an 
>>>>>>>>>>>>>> idea....??
>>>>>>>>>>>>>> -----Oorspronkelijk bericht-----
>>>>>>>>>>>>>> Van: plog-svn-bounces at devel.lifetype.net
>>>>>>>>>>>>>> [mailto:plog-svn-bounces at devel.lifetype.net] Namens Jon
>>>>>>>>>> Daley
>>>>>>>>>>>>>> Verzonden: woensdag 13 december 2006 17:06
>>>>>>>>>>>>>> Aan: plog-svn at devel.lifetype.net
>>>>>>>>>>>>>> Onderwerp: RE: [pLog-svn] Process size On Wed, 13 Dec 
>>>>>>>>>>>>>> 2006, Ayalon wrote:
>>>>>>>>>>>>>>> Where can I find the post of the rewrite?
>>>>>>>>>> http://forums.lifetype.net/viewtopic.php?
>>>>>>>>>> p=23240&highlight=htaccess
>>>>>>>>>>>>>> +rewrite+
>>>>>>>>>>>>>> error
>>>>>>>>>>>>>>> Anyway my provider is telling me that it looks like
>>>>>>>>>> lifetype >> >>> has
>>>>>>>>>>>>>>> memoryleaks in various aspects of the script, starting
>>>>>>>>>> with >> >> >>>  >>
>>>>>>>>>> caching.
>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>> that possible?
>>>>>>>>>>>>>>     It is certainly possible.  I would expect to see
>>>>>>>>>> more >> >> memory
>>>>>>>>>>>>>> usage in my setup if that were the case.
>>>>>>>>>>>>>>> Is it possible to disable everything related to cache
>>>>>>>>>> lift? >> >>> Just
>>>>>>>>>>>>>>> to test some things...
>>>>>>>>>>>>>>     Line 39 of class/cache/cachemanager.class.php.   
>>>>>>>>>>>>>> Change
>>>>>>>>>>>>>> $cacheEnable to false.  I think that should do the trick.
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> pLog-svn mailing list
>>>>>>>>>>>>>> pLog-svn at devel.lifetype.net 
>>>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> pLog-svn mailing list
>>>>>>>>>>>>>> pLog-svn at devel.lifetype.net 
>>>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> pLog-svn mailing list
>>>>>>>>>>>>> pLog-svn at devel.lifetype.net 
>>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> pLog-svn mailing list
>>>>>>>>>>>>> pLog-svn at devel.lifetype.net 
>>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> pLog-svn mailing list
>>>>>>>>>>>> pLog-svn at devel.lifetype.net 
>>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> pLog-svn mailing list
>>>>>>>>>>> pLog-svn at devel.lifetype.net
>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> pLog-svn mailing list
>>>>>>>>>>> pLog-svn at devel.lifetype.net
>>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>>>>>> _____________________________________________________________
>>>>>>>>>> ___
>>>>>>>>>> _
>>>>>>>>>> _____
>>>>>>>>>> This email has been scanned by the MessageLabs Email Security 
>>>>>>>>>> System.
>>>>>>>>>> For more information please visit http://www.messagelabs.com/ 
>>>>>>>>>> email 
>>>>>>>>>> _____________________________________________________________
>>>>>>>>>> ___
>>>>>>>>>> _
>>>>>>>>>> _____
>>>>>>>>>> _______________________________________________
>>>>>>>>>> pLog-svn mailing list
>>>>>>>>>> pLog-svn at devel.lifetype.net
>>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>>>>> _______________________________________________
>>>>>>>>> pLog-svn mailing list
>>>>>>>>> pLog-svn at devel.lifetype.net
>>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>>>> _______________________________________________________________
>>>>>>>> ___
>>>>>>>> _
>>>>>>>> ___
>>>>>>>> This email has been scanned by the MessageLabs Email Security 
>>>>>>>> System.
>>>>>>>> For more information please visit http://www.messagelabs.com/ 
>>>>>>>> email 
>>>>>>>> _______________________________________________________________
>>>>>>>> ___
>>>>>>>> _
>>>>>>>> ___
>>>>>>>> _______________________________________________
>>>>>>>> pLog-svn mailing list
>>>>>>>> pLog-svn at devel.lifetype.net
>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>>> --
>>>>>>> Jon Daley
>>>>>>> http://jon.limedaley.com/
>>>>>>> The difference between genius and stupidity is that genius has 
>>>>>>> its limits.
>>>>>>> -- Anonymous
>>>>>>> ----------------------------------------------------------------
>>>>>>> ---
>>>>>>> -
>>>>>>> ------------
>>>>>>>> _______________________________________________
>>>>>>>> pLog-svn mailing list
>>>>>>>> pLog-svn at devel.lifetype.net
>>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>>> ________________________________________________________________
>>>>>>> ___
>>>>>>> _
>>>>>>> __
>>>>>>> This email has been scanned by the MessageLabs Email Security 
>>>>>>> System.
>>>>>>> For more information please visit http://www.messagelabs.com/ 
>>>>>>> email 
>>>>>>> ________________________________________________________________
>>>>>>> ___
>>>>>>> _
>>>>>>> __
>>>>>> --
>>>>>> Jon Daley
>>>>>> http://jon.limedaley.com/
>>>>>> Music: A safe kind of high
>>>>>> -- Jimi Hendrix
>>>>>> -----------------------------------------------------------------
>>>>>> ---
>>>>>> -
>>>>>> -----------
>>>>>>> _______________________________________________
>>>>>>> pLog-svn mailing list
>>>>>>> pLog-svn at devel.lifetype.net
>>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>>> _________________________________________________________________
>>>>>> ___
>>>>>> _
>>>>>> _
>>>>>> This email has been scanned by the MessageLabs Email Security 
>>>>>> System.
>>>>>> For more information please visit http://www.messagelabs.com/ 
>>>>>> email 
>>>>>> _________________________________________________________________
>>>>>> ___
>>>>>> _
>>>>>> _
>>>>>> _______________________________________________
>>>>>> pLog-svn mailing list
>>>>>> pLog-svn at devel.lifetype.net
>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>> --
>>>>> Jon Daley
>>>>> http://jon.limedaley.com/
>>>>> If everything is coming your way then you're in the wrong lane.
>>>>> -- Anonymous
>>>>> ------------------------------------------------------------------
>>>>> ---
>>>>> -
>>>>> ----------
>>>>>> _______________________________________________
>>>>>> pLog-svn mailing list
>>>>>> pLog-svn at devel.lifetype.net
>>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>>> __________________________________________________________________
>>>>> ___
>>>>> _
>>>>> This email has been scanned by the MessageLabs Email Security 
>>>>> System.
>>>>> For more information please visit http://www.messagelabs.com/email 
>>>>> __________________________________________________________________
>>>>> ___
>>>>> _
>>>>> _______________________________________________
>>>>> pLog-svn mailing list
>>>>> pLog-svn at devel.lifetype.net
>>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>> _______________________________________________
>>>> pLog-svn mailing list
>>>> pLog-svn at devel.lifetype.net
>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>>> _______________________________________________
>>>> pLog-svn mailing list
>>>> pLog-svn at devel.lifetype.net
>>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>> _______________________________________________
>>> pLog-svn mailing list
>>> pLog-svn at devel.lifetype.net
>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>> _______________________________________________
>>> pLog-svn mailing list
>>> pLog-svn at devel.lifetype.net
>>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>>
>> _______________________________________________
>> pLog-svn mailing list
>> pLog-svn at devel.lifetype.net
>> http://devel.lifetype.net/mailman/listinfo/plog-svn
>
> --
> Jon Daley
> http://jon.limedaley.com/
>
> When dog food is new and improved tasting, who tests it? 
> _______________________________________________
> pLog-svn mailing list
> pLog-svn at devel.lifetype.net
> http://devel.lifetype.net/mailman/listinfo/plog-svn

_______________________________________________
pLog-svn mailing list
pLog-svn at devel.lifetype.net
http://devel.lifetype.net/mailman/listinfo/plog-svn


_______________________________________________
pLog-svn mailing list
pLog-svn at devel.lifetype.net
http://devel.lifetype.net/mailman/listinfo/plog-svn

More information about the pLog-svn mailing list