Saturday, September 6, 2008

New Site Launch



Support independent publishing: buy this book on Lulu.


Launched a new site to sort of centralize information about the book The Path Into Darkness. There's not much content there yet, mostly because the book is still winding it's way through the distributor channels before it can get listed and sold on online bookstores, although it is available at the author's storefront.

Next week, excerpts from the book will be available.

Friday, September 5, 2008

Adventures in Webpage Content Injection.

I run several web sites and recently decided that I wanted to add some common text to the bottom of all the pages. Since I don't generate the content, it would be best if the server did this as it served the pages.

A quick search through Apache's module directory and I saw that mod_layout fit the bill ... in theory, at least. I had tried, in vain, for the last two days to get mod_layout working with my antiquated Linux server (fedora core 3), before I decided that I should really look into my other options -- especially since mod_layout hadn't been updated and from what I could see from forums, it's developer wasn't exactly interested in making it working with Apache 2.0+.

Then I saw a post about how mod_rewrite could be cajoled into doing this sort of a task. Essentially it does this by rewriting the request into a CGI call, passing the original requested file name as a paramter.

Like so:


RewriteRule /(.*) /wrapper.cgi?file=$1 [nc,l,qsa]


All the examples were using PHP. My server is old (as mentioned above) and you can't really find RPM's for older Fedora's. So, since I don't have PHP installed, I looked at my options ... and it immediately struck me that Python would be up to the task.

So, you need to create a Python scripted named
wrapper.cgi
which contains:

import os
import os
import re
import urllib

print "Content-type: text/html"
print
docroot = os.getenv( 'DOCUMENT_ROOT' )
fname = docroot + urllib.unquote( os.getenv( 'REQUEST_URI' ) )
buff = open( fname ).read( )

mobj = re.compile( '<body[^>]*>', re.IGNORECASE | re.VERBOSE )
mobj2 = re.compile( '</body>', re.IGNORECASE | re.VERBOSE )
obj = mobj.search( buff )
obj2 = mobj2.search( buff )
header = buff[:obj.end()]
body = buff[obj.start( ):obj2.start( )]
footer = buff[obj2.start():]

print header
if os.path.exists( docroot + '/header.inc' ):
print open( docroot + '/header.inc' ).read( )
print body
if os.path.exists( docroot + '/footer.inc' ):
print open( docroot + '/footer.inc' ).read( )
print footer


So, now I have a framework and method for wrapping all the assorted web pages with some common header & footer code (such as some essential support links, for example).

Thursday, September 4, 2008

Rider-Waite tarot card images

Hmm...just noticed that the images used on Wikipedia are from the original deck, which is in the public domain. Which is useful, since I want to use them as part of a book cover, and now I don't have to pay royalties or seek permissions.

Happy day.

Wednesday, September 3, 2008

Plone and the annoying ATAmazon...

So, I recently tried to make more use out of my Amazon Associate Id that I had gotten way back in 2001. Since I run Plone on a few of my sites, I tried to to install the ATAmazon product.

It didn't work.

Not only didn't it work, but it didn't even give me very helpful error messages to try to fix the problem myself. Other people seem to use it ... or at least know about it. I enter in the proper ASIN number (I know this, because you can click on the "Buy" button that it shows and go to the proper item's page), but none of the details seem to either retrieved or parsed properly.

Any tips or clues to help resolve this would be most welcome.

(EDIT: Silly amazon "deprecated" that version of their web services. Unlike the usual method of deprecation, they simply turned it off!)

Monday, September 1, 2008

Stackless Python meets Twisted Matrix....




Sometimes, you come across two programming toolkits that would go great together. However, in the case of Twisted Matrix and Stackless python, there's some legwork required to get these two great systems to work together.
Twisted requires that it's reactor runs in the main "tasklet", but if there is no network activity or other deferred code to execute, the reactor loop will stop the entire application and thus defeat the purpose behind using tasklets and Stackless.

There is some setup required to get this all working together.

import stackless
from twisted.internet import reactor, task
reactor_tasklet = None

def reactor_run( ):
reactor_tasklet = stackless.getcurrent( )
# repeatedly call stackless.schedule every 0.0001 seconds
schedulingTask = task.LoopingCall( stackless.schedule )
# this prevents the reactor from blocking out the other tasklets
schedulingTask.start( 0.0001 )
reactor.run( )

t = stackless.tasklet( reactor_run )
t.run( )
# run the stackless scheduler.
stackless.run( )

Now, extending out this simple case to a more general solution involves the use of Python's function decorators. (I use the great decorator.py module to make decorators a little easier to write.)
def __filter( d ):
if isinstance( d, failure.Failure ):
if isinstance( d.value, TaskletExit ):
print "ignore taskletexit"
return None
return d
return d

def __wrapper( d, f, *args, **kwargs ):
try:
rv = defer.maybeDeferred( f, *args, **kwargs )
rv.addCallback( __filter )
rv.addCallback( d.callback )
rv.addErrback( __filter )
except TaskletExit:
pass
except Exception, e:
print e, dir( e )
d.errback( e )


Above is just some boiler-plate code. __filter screens out the TaskletExit exception that gets sent to Tasklets; if this isn't done, the Twisted framework wraps it up in an instance of twisted.python.failure.Failure and you get "Unhandled error in Deferred" exceptions at the calling point. Since this is almost never what you want, it's easiest to just filter it out. Of course, in real code you'll remove the line reading 'print "ignore taskletexit"'.

__wrapper does the actual heavy lifting of the function call. It uses the maybeDeferred function to ensure that after the function call we are only dealing with Deferred's. __wrapper uses Twisted's usual callback mechanism to ensure that the Deferred that it received as a function paramater is called once the results of the actual function call is available. This parameter Deferred is essential for the function decorators described next to work.

reactor_tasklet = None

@decorator
def deferred_tasklet( f, *args, **kwargs ):
d = defer.Deferred( )
t = stackless.tasklet( __wrapper )
t( d, f, *args, **kwargs )
t.run( )
return d


@decorator
def blocking_tasklet( f, *args, **kwargs ):
f2 = deferred_tasklet( f )
d = f2( *args, **kwargs )
if reactor_tasklet != stackless.getcurrent( )
and stackless.getcurrent( ) != stackless.getmain( ):
return block_on( d )
raise RuntimeError( "Cannot block in reactor task" )


def block_on( d ):
chan = stackless.channel( )
d.addBoth( lambda x,y=chan: y.send( x ) )
return chan.receive( )

Here we have the two main function decorators deferred_tasklet and blocking_tasklet, as well as the utiliity function block_on. The first of these simply returns a Deferred, suspiciously the very same Deferred that it passes as a parameter to the __wrapper function; which, if you've been paying attention, will be triggered once the results of the wrapped-up function are available. All we're really doing here is creating a stackless.tasklet and running __wrapper in that new microthread.

blocking_tasklet goes one step beyond this, and takes the Deferred that we were passed earlier and converts it into a blocking function call. First, it does some sanity checks to ensure that it's not blocking in the same tasklet that Twisted's reactor is running in. Somewhere you need to store the value of stackless.getcurrent() when called from with the reactor's tasklet. We also need to make sure that our current tasklet is not the "main" Stackless tasklet; this should never happen, but I like to be safe at times.

The utility function block_on sets up a Stackless channel. It then adds a simple little lambda closure. This closure only send's it's parameter to the stackless channel. This closure is added to both the callback and errback chain of the Deferred that we're going to wait on. After this is all set up, we then call receive, which blocks this tasklet until the Deferred is finished and the callbacks/errback is fired off. At this point, we receive the return value of the original function through the channel and can return it as the return value of our function.

As long as we are not in the same tasklet as Twisted's reactor, we can use this block_on function to turn our other wise asynchronous code into a sequentially execute synchronous code. This can also be done using Twisted's inlineCallbacks decorator, but that turns the decorated function into a generator, which isn't always what we want.