Sorting through the Twisted framework is reminiscent of the old story about blind men and elephants. Twisted has many capabilities, and it takes a bit of a paradigm switch to get a good sense of why they are all there. In fact, as I write this first installment, I am probably only halfway toward getting my mind fully around Twisted. We can work through it together.

One of the strengths of recent versions of Python is that they come with "batteries included" -- that is, the standard distribution includes modules to do just about everything you want to accomplish in most programming tasks. For the most part, when you want a third-party Python module or package, it is to accomplish some specialized and unusual task. Twisted is one of few exceptions to the pattern described; developed by Twisted Matrix Laboratories, it is a well-designed and general-purpose collection of modules for performing all manner of network programming tasks, in ways not easily facilitated by Python's standard library.

It is not quite true that Python's standard library lacks support for asynchronous, non-blocking network applications. The module asyncore provides basic support for switching among I/O channels within a single thread. But Twisted takes the style to a higher level and provides a huge collection of pre-built and reusable protocols, interfaces, and components.

A first server

The documentation that accompanies Twisted is quite extensive, but hard to get a handle on. Let's start with a simple server, and build on that. In a recent developerWorks tip (see Resources for a link), I demonstrated an XML-based "Weblog server" that presents to a client a stream of records about the latest hits to a Web server. The XML aspect is not important here, but the use of SocketServer and its ThreadingTCPServer class is useful as a baseline. This pre-Twisted server consists of:

from SocketServer import BaseRequestHandler, ThreadingTCPServer

from time import sleep

import sys, socket

from webloglib import log_fields, hit_tag


class WebLogHandler(BaseRequestHandler):

    def handle(self):

        print "Connected from", self.client_address

        self.request.sendall('<hits>')

        try:

            while True:

                for hit in LOG.readlines():

                    self.request.sendall(hit_tag % log_fields(hit))

                sleep(5)

        except socket.error:

            self.request.close()

        print "Disconnected from", self.client_address


if __name__=='__main__':

    global LOG

    LOG = open('access-log')

    LOG.seek(0, 2)     # Start at end of current access log

    srv = ThreadingTCPServer(('',8888), WebLogHandler)

    srv.serve_forever()

Other than that overhead of its per-client thread creation, a notable feature of the SocketServer-based server is its use of a blocking call to time.sleep() within its handler. For Twisted's non-blocking select() loop, such a block is not permissible.

A first non-blocking approach pushes any artificial delays onto the client, and lets the client specifically request each new batch of Weblog records (and also sends a message to indicate their absence, rather than send nothing). This Twisted server looks like:

from twisted.internet import reactor

from twisted.internet.protocol import Protocol, Factory

from webloglib import hit_tag, log_fields


class WebLog(Protocol):

    def connectionMade(self):

        print "Connected from", self.transport.client

        self.transport.write('<hits>')

    def dataReceived(self, data):

        newhits = LOG.readlines()

        if not newhits:

            self.transport.write('<none/>')

        for hit in newhits:

            self.transport.write(hit_tag % log_fields(hit))

    def connectionLost(self, reason):

        print "Disconnected from", self.transport.client


factory = Factory()

factory.protocol = WebLog


if __name__=='__main__':

    global LOG

    LOG = open('access-log')

    LOG.seek(0, 2)     # Start at end of current access log

    reactor.listenTCP(8888, factory)

    reactor.run()

Readers should refer to my prior tip for details on the client application. But the following change should be noted. The main client loop adds two lines:

while 1:

    xml_data = sock.recv(8192)

    parser.feed(xml_data)

    sleep(5)          # Delay before requesting new records

    sock.send('NEW?') # Send signal to indicate readiness

Back to top

The parts of a Twisted server

A Twisted server consists of several modular elements. At a bytestream level, a server implements a protocol, often by inheriting from twisted.internet.protocol.Protocol or from some previously specialized child of it. For example, provided subclasses (in twisted.protocols) include dns, ftp, gnutella, http, nntp, shoutcast, and many others. Basically, a protocol should know how to handle making and losing connections, and receiving and sending data within a connection. These responsibilities are not much different than in a SocketServer-based server, except in being slightly more modular in defining methods for each element.

The next level of a Twisted server is a factory. In our twisted-weblog-1.py example, the factory really does nothing besides store a protocol. In a more sophisticated server, however, a factory is a good place to perform initialization and finalization related to a protocol server. And probably of greatest interest, a factory can be persisted within applications (we will see those soon).

Neither a protocol nor a factory knows anything about the network the server runs on. Instead, a reactor is a class that actually listens on a network (utilizing a factory instance for its protocol). Basically, a reactor is just a loop that listens on a given port and network interface (which one is chosen by calling a method like .listenTCP(), .listenSSL(), or .listenUDP()). The thing to understand is that the basic reactor in Twisted, SelectReactor, runs in a single thread; each connection is checked for new data, and the data is delivered to the relevant protocol object. An upshot is that a protocol object is really not allowed to block, or even just take too long to complete (protocols must be programmed appropriately).

Back to top

An enhanced server

Let's try to enhance the Twisted Weblog server so that it follows the pattern of SocketServer-weblog.py in feeding new records to clients without the need for repeated requests from those clients. The problem here is inserting a time.sleep() call into a method of WebLog(Protocol) causes it to block, and so is not allowed. While we are at it, notice that the prior servers probably do the wrong thing in that they feed each new batch of records only to one client. Presumably, if you want to allow multiple clients to monitor a Weblog, you want them all to receive ongoing updates.

The way you delay actions in Twisted without blocking is to add callbacks to a reactor, using the .callLater() method. A callback added this way is added to the queue of events to service, but it will not actually be processed until after a specified delay. Putting both changes together, an enhanced Weblog server looks like:

from twisted.internet import reactor

from twisted.internet.protocol import Protocol, Factory

from webloglib import hit_tag, log_fields

import time


class WebLog(Protocol):

    def connectionMade(self):

        print "Connected from", self.transport.client

        self.transport.write('<hits>')

        self.ts = time.time()

        self.newHits()

    def newHits(self):

        for hit in self.factory.records:

            if self.ts <= hit[0]:

                self.transport.write(hit_tag % log_fields(hit[1]))

        self.ts = time.time()

        reactor.callLater(5, self.newHits)

    def connectionLost(self, reason):

        print "Disconnected from", self.transport.client


class WebLogFactory(Factory):

    protocol = WebLog

    def __init__(self, fname):

        self.fname = fname

        self.records = []

    def startFactory(self):

        self.fp = open(self.fname)

        self.fp.seek(0, 2) # Start at end of current access log

        self.updateRecords()

    def updateRecords(self):

        ts = time.time()

        for rec in self.fp.readlines():

            self.records.append((ts, rec))

        self.records = self.records[-100:]  # Only keep last 100 hits

        reactor.callLater(1, self.updateRecords)

    def stopFactory(self):

        self.fp.close()


if __name__=='__main__':

    reactor.listenTCP(8888, WebLogFactory('access-log'))

    reactor.run()

In this case, we define a custom factory and move some of the initialization from the _main_ block to the factory. Notice also that the clients for this server need not (and should not) sleep or send new requests -- in fact, I use the exact client application I discussed in the XML tip (see Resources).

The factory and the protocol use the same technique in their custom methods .updatedRecords() and .newHits(), respectively. That is, if a method wants to run periodically, its last line can schedule it to run again at a specified delay. On its face, this pattern looks a lot like recursion -- but it is not (moreover, the repeat scheduling need not occur on the last line; it just makes sense there). The method .newHits(), for example, simply lets the controlling reactor loop know that it wants to be called in another 5 seconds, but the method itself terminates. There is no requirement that a method schedule only itself -- it can schedule whatever it wants to occur, and functions quite apart from factory or protocol methods can be added to a reactor loop, if you wish.

Back to top

Persistence and scheduling

Besides reactor.callLater() scheduling, Twisted contains a general class twisted.internet.defer.Deferred. In essence, deferreds are a generalization of scheduled callbacks, but allow techniques such as chaining dependent callbacks and handling error conditions in these chains. The idea behind a Deferred object is that when you call a method, rather than wait for its results (which may take a while to arrive), the method can immediately return a Deferred object that the reactor/scheduler can call again later, when results are expected to be available.

I have not really played with Deferred objects yet, but it feels like getting them right will be slightly tricky. If you need to wait on a blocking action -- say, the results from a remote database query -- it is not clear exactly how long you will need to wait for results to be available. Deferred objects do have a timeout mechanism, but I will have to come back to that in a later installment. Interested readers should at least know that the Twisted Matrix developers have attempted to provide a standard API for wrapping blocking actions. Of course, the worst case is to fall back to using threads for blocking actions that really cannot be converted into asynchronous callbacks.

Another important element to Twisted servers is their easy support for persistence. A reactor is a loop that monitors and responds to I/O events. An application is much like an enhanced reactor that is able to pickle its state for later re-starting. Moreover, applications can be statefully saved into ".tap" files, and can be managed and daemonized using the tool twistd. Here's a simple example that illustrates the usage (modelled on the Twisted documentation's OneTimeKey example). This server delivers distinct Fibonacci numbers to all interested clients, without repeating numbers between them -- even if the server is stopped and started:

from twisted.internet.app import Application

from twisted.internet.protocol import Protocol, Factory


class Fibonacci(Protocol):

    "Serve a sequence of Fibonacci numbers to all requesters"

    def dataReceived(self, data):

        self.factory.new = self.factory.a + self.factory.b

        self.transport.write('%d' % self.factory.new)

        self.factory.a = self.factory.b

        self.factory.b = self.factory.new


def main():

    import fib_server    # Use script as namespace

    f = Factory()

    f.protocol = fib_server.Fibonacci

    f.a, f.b = 1, 1

    application = Application("Fibonacci")

    application.listenTCP(8888, f)

    application.save()


if '__main__' == __name__:

    main()

You can see that mostly all we have changed is replacing reactor with application throughout. While the class Application also has a .run() method, we use its .save() method to create a Fibonacci.tap file. Running this server is done as:

% python fib_server.py

% twistd -f Fibonacci.tap

...let server run, then shut it down...

% kill `cat twistd.pid`

...re-start server where it left off...

% twistd -f Fibonacci-shutdown.tap

...serve numbers where we left off...

The client that connects to this server should use a time.sleep() in its loop if it only wants a new number intermittently rather than as fast as possible. Obviously, a more useful server can provide a more interesting stateful datastream.

CS395T: Special Topics in Computer Vision, Spring 2010

Object Recognition

Meets: Wednesdays 3:30-6:30 pm
ACES 3.408
Unique # 54470
 
Instructor: Kristen Grauman 
Email: grauman@cs
Office: CSA 114
 
TA: Sudheendra Vijayanarasimhan
Email: svnaras@cs
Office: CSA 106

When emailing us, please put CS395 in the subject line.

Announcements:

Course overview:

Topics: This is a graduate seminar course in computer vision.   We will survey and discuss current vision papers relating to object recognition, auto-annotation of images, and scene understanding.  The goals of the course will be to understand current approaches to some important problems, to actively analyze their strengths and weaknesses, and to identify interesting open questions and possible directions for future research.

See the syllabus for an outline of the main topics we'll be covering.

Requirements: Students will be responsible for writing paper reviews each week, participating in discussions, completing one programming assignment, presenting once or twice in class (depending on enrollment, and possibly done in teams), and completing a project (done in pairs). 

Note that presentations are due one week before the slot your presentation is scheduled.  This means you will need to read the papers, prepare experiments, make plans with your partner, create slides, etc. more than one week before the date you are signed up for.  The idea is to meet and discuss ahead of time, so that we can iterate as needed the week leading up to your presentation. 

More details on the requirements and grading breakdown are here.

Prereqs:  Courses in computer vision and/or machine learning (378 Computer Vision and/or 391 Machine Learning, or similar); ability to understand and analyze conference papers in this area; programming required for experiment presentations and projects. 

Please talk to me if you are unsure if the course is a good match for your background.  I generally recommend scanning through a few papers on the syllabus to gauge what kind of background is expected.  I don't assume you are already familiar with every single algorithm/tool/image feature a given paper mentions, but you should feel comfortable following the key ideas.

1. Single-object recognition fundamentals: representation, matching, and classification
1. Specific objects
   
2. Classification and global models
3. Objects composed of parts
4. Region-based methods
2. Beyond single objects: recognizing categories in context and learning their properties
1. Context
2. Attributes
3. Actions and objects/scenes
   
3. Scalability issues in category learning, detection, and search
1. Too many pixels!
2. Too many categories!
3. Too many images!
   
4. Recognition and "everyday" visual data
1. Landmarks, locations, and tourists
2. Alignment with text
3. Pictures of people

Other useful links:

• Compiled list of recognition datasets
  
• OpenCV (open source computer vision library)
• Weka (Java data mining software)
• Netlab (Matlab toolbox for data analysis techniques, written by Ian Nabney and Christopher Bishop)
• CV Online
• Annotated Computer Vision Bibliography
• Computer vision conferences
• ICCV 2005 / CVPR 2007 / ICCV 2009 Short Course on Recognition
• AAAI 2008 Tutorial on Recognition
• ICML 2008 Tutorial on Recognition

• CS 395T Spring 2007: Object Recognition
• CS 395T Spring 2008: Visual Recognition and Search
• CS 395T Spring 2009: Visual Recognition and Search

• Object Recognition and Scene Understanding, MIT, Antonio Torralba
• Learning-based Methods in Vision, CMU, Alyosha Efros
• Selected Topics in Vision & Learning, UCSD, Serge Belongie
• Recognition Problems in Computer Vision, SFU, Greg Mori
• High-Level Recognition in Computer Vision, Princeton, Fei-Fei Li
• Computer Vision and the Web, UNC, Svetlana Lazebnik