Associated Content

I've heard good things about Associated Content, where you get paid for publishing content and articles. So, I submitted two pieces of flash fiction (short stories of up to 2000 words). They're now available for viewing.

Timmy's father shipped out on active military duty the previous Christmas. Before he left, he gave Timmy a collection of little green army men; most of which were his favorite toys when he was younger.

Toy Soldiers Off to War
A piece of flash fiction (a short story up to 2000 words).
View more »

The Montoya was berthed in Fueling Bay 2-A. James Sandersen had the annoying habit of liking to run the ship down below manufacturer's recommendation on fuel levels. He also ran the ship past suggested maintenance schedules, but that was due to finances more than any other factor. Sandersen always insisted that the shipbuilders were in a cartel with the fuelers, but that was neither here nor there.

Deal Done Bad
A piece of science fiction flash fiction (a short story of up to 2000 words).
View more »

EVE Online's Stackless Architecture

I just saw this article about EVE Online's Server Model. Very interesting and informative read, and a testament to Stackless's reliability and usability.

Weekly Roundup

This week has seen some interesting news in the Python world.

• In particular, the release of Python 2.6, with support for json and a new multiprocessing paradigm.
  
• In my research for other projects, I ran across an older article by the ineffable effbot about thread synchronization.
• One of the more interesting, and sometimes confusing, parts of Python is list comprehensions, explained in good detail in this article on comprehending list comprehensions.
• I don't often need to write a parser, but the ideas presented in the "Zen of Parsing" seem very useful for those rare instances.
  

On the other hand, not everything I did this week was programming related.

• I stumbled across this list of words of wisdom from Ernest Hemmingway.
  
• Questions about copyright? You know you have them, and here are the big myths about copyright.

Perspective Broker Authentication for Stackless Python

Twisted Matrix's Perspective Broker is, to me at least, the main compelling reason to use the entire framework. It's also an area that is lacking in documentation that is usable to a newbie to the framework.

One of the things provided by the Perspective Broker is an authentication framework, based around the concept of an avatar. We're going to create a Perspective Broker authenticator that is more complex than what is presented in the Twisted Matrix documentation.

In one of the projects I worked on, we needed to have a central authentication server, to which all the other object brokers would authenticate against. Due to the way that Twisted operates, a user needs to authenticate against the broker that they are currently connected to. We need to build a authenticator that can authenticate for itself, or if needed, recursively authenticate against another server.

To achieve this goal, we run into the first hurdle, Twisted doesn't pass a meaningful password around. This changes the order of authentication slightly then, as we now need to authenticate against the main authenticating server first. That server then gives us a cookie, known as authentication_cookie in the code below. This cookie is good for this login session with the main authenticator. When connecting and authenticating against any other broker in the system, we pass the username and this cookie as our credentials.

class UsernamePasswordDatabaseChecker:
 interface.implements(checkers.ICredentialsChecker)

 credentialInterfaces = (credentials.IUsernamePassword,
                         credentials.IUsernameHashedPassword)

 def __init__( self, login_service ):
 self.login_service = login_service
 
 def passcheck( self, account, credentials ):
     if not account:
         return failure.Failure( NoSuchAccount( ) )
     c1 = credentials.checkPassword( account.password )
     c2 = credentials.checkPassword( account.authentication_cookie )
     if c1:
         return login_server.MasterAuthenticationStatus( account.username,
                                                         account, 1 )
     elif c2:
         return login_server.MasterAuthenticationStatus( account.username,
                                                         account, 0 )
     elif not c1 and not c2 and account.authentication_cookie:
         return failure.Failure( AuthenticationCookieMismatch( ) )
     elif not c1 and not c2 and not account.authentication_cookie:
         return failure.Failure( InvalidPassword( ) )
     return failure.Failure( NoSuchAccount( ) )

 @reactor.deferred_tasklet
 def store_auth_cookie( self, auth_status ):
     bcr = reactor.blocking_tasklet( self.login_service.callRemote )
     return bcr( "store_authentication_cookie", auth_status )

 @reactor.deferred_tasklet
 def requestAvatarId(self, credentials ):
     if hasattr( self.login_service, "callRemote" ):
         bcr = reactor.blocking_tasklet( self.login_server.callRemote )
         acct = bcr( "get_account", credentials.username )
         self.passcheck( acct, credentials )
         status = bcr( "account_status", acct )
         rv = self.store_auth_cookie( status )
         return rv
     else:
         rv = self.login_service.authenticate_account( credentials )
         return rv

We use one of the function decorators from when we first started integrating Twisted into Stackless python, the deferred_tasklet. If you haven't read that article, the deferred_tasklet decorator wraps the function to run in a seperate Stackless tasklet, while still returning a Deferred that can be used by the rest of the Twisted machinery.

The referenced member login_service is a straight-forward object that implements the function authenticate_account. If the credentials it is passed are good, it returns an object -- this object is the avatar required by the Perspective Broker. If the credentials are bad, it returns None, which then gets translated into an appropriate remote exception.

The core ingredient in this is whether or not UsernamePasswordDatabaseChecker's member login_service has a member method callRemote. In this scenario, we make remote calls to do the authentication. We use another one of function decorators, blocking_tasklet, to do this so that our code can remain synchronous in style.

All in all, using Stackless tasklets to implement this setup results in less lines of code, and code that is straight-forward and easy to understand. The purely Twisted incarnation of this setup result in a crazy amount of nested callbacks that were very difficult to follow during initial testing, let alone a few months later.

The examples for login_service can be provided, if anyone wants, I just need to dig through the CVS repository to find them.

Stuff in the refrigerator

Now that we've looked at the stuff in my living room, let's wander over into the kitchen and check out the fridge.

1. Grapes: There's some other fruit in there as well (peaches, strawberries, and some pineapple). But I like the grapes, especially the seedless ones. You can just keep popping them into your mouth, one after the other after the other. Before you know it, you've eaten way more than you should have. My rating: A+
2. Milk: Combined with some chocolate chip cookies, this is my ideal after-dinner treat. Oh, and it's useful for coffee and cereal. But, really, it's all about combining it with cookies. My rating: A
3. Diet Dr. Pepper: Of all the different sodas I've tried, this is the one that tastes least like diet soda. This, of course, is important as some diet soda ends up tasting horrible reminding you that you've opted for lower calories over good taste. It also happens to be my favorite mixer with certain alcohol. My rating: A
4. Salad: I don't usually eat all that healthy. I mean, I try to. I want to. However, I happen to like the way certain foods -- usually the ones that are not good for you at all -- taste. I do like salad though, particularly simple just a simple combo of tomatoes, lettuce, and radishes. My rating: B+
5. Southern Comfort: I haven't drank Southern Comfort straight up since college, and it was the leading cause of some of my stupider decisions. I do like to, occasionally, mix it with the aforementioned Diet Dr. Pepper into a very tasty adult beverage. My rating: A+

A quick update...

I planned on having an entry tomorrow showing a more in-depth example of combining Twisted with Stackless python, but it's taking a little longer than expected to get all the pieces together. The example will be using what is in my opinion the most useful part of Twisted: the Perspective Broker. I have several working systems that utilize it, but pulling out just enough to demonstrate how to use the PB with Stackless is proving a little troublesome.

It should be ready by Thursday.

Multi-Threaded Twisted / Stackless Integration

Another way to integrate Twisted with Stackless python, is to use multiple threads. One thread handles Twisted's reactor while Stackless tasklets run in at least one other thread. This lowers the deterministic nature of Stackless, but for certain conditions may be more effective than trying to integrate Twisted and Stackless into a single thread. Communication between the threads is handled through a channel, which according to the documentation for Stackless python is thread-safe.

the_channel = stackless.channel( )

def a_func( *args ):
    print "a_func:", args
    return args

def dispatch( d, func, *args, **kwargs ):
    d1 = defer.maybeDeferred( func, *args, **kwargs )
    d1.addCallback( lambda x: reactor.callFromThread( d.callback, x ) )
    d1.addErrback( lambda x: reactor.callFromThread( d.errback, x ) )

For our example, we'll be calling a_func to run in the Stackless thread. This is handled through the helper function dispatch. The result of the function will be wrapped up in a Deferred. Through the reactor's callFromThread method we'll be able to fire the callback chain inside the thread running the main Twisted reactor loop.

the_channel is our cross-thread communication channel, through which the requests for function invocation will be passed.

def stackless_dispatcher( chan ):
    while True:
       try:
           d, func, args, kwargs = chan.receive( )
           t = stackless.tasklet( dispatch )
           t( d, func, args, kwargs )
           print threading.currentThread( )
           stackless.schedule( )
       except:
           break

This is the main loop of the Stackless thread. This method loops until an error condition occurs -- in this simplified version that is enough. It blocks on the main channel, until it receives a function, function arguments, and a Deferred that will be fired upon function completion. It then creates a tasklet to run the function in.

def call_in_stackless( chan, func, *args, **kwargs ):
    d = defer.Deferred( )
    t1 = stackless.tasklet( chan.send )
    t1( (d,func,args,kwargs) )
    stackless.schedule( )
    return d

This function is called from within the reactor's thread to cause a function invocation inside the Stackless thread. Because Stackless complains about deadlock when sending on channels, we have to create a tasklet to send on the main channel. The function, arguments, and a newly created Deferred are sent via the channel and the Deferred is returned from the function. Ultimately, this Deferred will have it's callback chain fired so at this point, traditional Twisted-style programming can continue.

def test( chan ):
    print threading.currentThread( )
    d = call_in_stackless( chan, a_func, 1 )
    d2 = call_in_stackless( chan, a_func, 1, 2 )
    dl = defer.DeferredList( [ d, d2 ] )
    dl.addCallback( lambda x: reactor.callLater( 0, reactor.stop ) )
    def ender( x, chan ):
        t = stackless.tasklet( chan.send )
        t( x )
        stackless.schedule( )
    dl.addCallback( ender, chan )

reactor.callInThread( stackless_dispatcher, the_channel )
reactor.callLater( 0, test, the_channel )
reactor.run( )

This is just a test of the premise. It should operate as expected, with a_func being invoked twice inside the Stackless thread before the reactor is stopped. We force a shutdown of the Stackless loop by passing a single argument through the channel -- since the receiving side is expecting a 4-part tuple this will cause an exception and stop the loop.

This form of integration does allow for some more concurrency than the previously discussion integration method. While we have to worry about Python's GIL (global interpretor lock) cutting down our actual concurrency, if the application is heavily I/O-bound this is not much of an issue since the GIL is released whenever control passes into the Twisted's reactor loop.

Bestsellers and Book Sales Expectations

Note: I found the following while surfing and googling around. The article's terms of use allows for copying it, as long as I maintain the proper attributions.


Most new authors have no idea how book sales happen or what numbers to expect. A typical successful book from a major publisher sells between 20,000 and 40,000 copies. Healthy book sales from a smaller publisher may number at least 5,000 copies, whereas a self-published book may be considered promising if it sells as few as 1,000 copies to a wide audience.

Set your sales goals high but your expectations lower. Obviously you want high sales for you book; otherwise, why do all the work? But be clear about your vision. If you plan to author a book for your own seminar business, your expectations will necessarily be different from those of the author who wants to become a world-class speaker and subject-matter guru. Clearly a book about diets will have broader market appeal than a book on corporate strategic planning.

Your ability to drive book sales will also depend on the name recognition you’ve already established. Someone like Dr. Phil McGraw can realistically expect his next book to sell millions of copies regardless of the subject matter. How’s your name recognition?

Most people are surprised to hear that less than half of book sales each year are through bookstores, leaving other possibilities for authors to pursue.

Many new authors just assume their book will appear in bookstores. If the book is self-published, it is very unlikely you will be able to crack the bookstore market; therefore, it will be important to consider:

* Sales through your own Web site
* Sales through Amazon.com and BarnesandNoble.com
* Sales through other Web sites and affiliate programs
* Sales through print catalogs
* Sales to specialty markets for premium incentives (organizations and corporations)
* Sales to mass merchandisers or other retailers

Almost the only way to get your book into national bookstore chains is to be published by a traditional publisher. They have the distribution methods in place to make this possible.

You may choose to self-publish and work on sales through some of these non-bookstore channels, get to credible sales figures, and then approach a traditional publisher to finally hit that desirable bookstore market.

If you sell your own book, you know exactly how many books you printed, how many have been sold, and how many are left to be sold in the printing.

Oddly enough, when books are published by traditional publishers and go through the bookstore system, there are very few avenues open for authors and even publishers to track "real" sales of books. The Bookscan report from Neilsen Broadcast Data Systems, available since 2001, provides weekly sales by the major book retailers. But only the very largest companies can afford this expensive service. It is virtually impossible to estimate the real sales of a book from the quantities that are shipped to the stores. Remember, all "sales” to bookstores are made with a long-established, full-price return policy anytime. That means all the books could be returned to you, even a year later. Major publishers expect overall return rates of 20 percent to 30 percent, but individual books can be much, much higher.

When your book is returned, it doesn’t mean that someone bought it, didn’t like it, and sent it back through the bookstore. Far more likely, it was never sold in the first place, and the bookstore returned it to the distributor after sixty to ninety days.

You will see sales numbers on your royalty statement, but generally there is also a reserve for expected returns as well.

If your book is published by a traditional publisher or printed by Lightning Source, you can call the distributor Ingram’s sales tracking phone number, 615-213-6803, to hear about sales for both the previous week and last year.

It seems that most authors these days say their book is a best seller, and that’s because there is no established, objective criteria for what that means. There are roughly forty national and regional best-seller lists in the United States. The New York Times’ list is probably the most well-known and respected. Another one that is well-known, especially inside the publishing industry, is provided by Publishers Weekly (PW). In 2005, 442 adult titles were declared best-sellers, according to the Publisher’s Weekly compilations.

Some best-seller lists such as those belonging to Publisher’s Weekly and USA Today are based on national surveys, with others on much less objective criteria. The New York Times bases their list on a poll of both chain and independent bookstores. The list from the American Booksellers Association polls only independent bookstores. The Los Angeles Times polls thirty bookstores in the Los Angeles metropolitan area to compile its list.

Only eight self-published books have made it to number one on the Publishers Weekly bestseller list.


Copyright (c) 2007 Gail Richards

Sunday Funnies

They're not necessarily new, but these are some strips I like.






The Order of the Stick (doesn't allow hot linking :( ).

I know, no great new findings here, but I thought it'd be nice to share to those few out there who may not actually have seen either of these strips before.