I have been playing with tulip in python 3.3 while developing archangel and come across a couple of interesting problems that i thought were worth documenting mainly for my own personal usage, But also with the hope that it may be useful for others.

Archangel uses the new event loop being developed by Guido van Rossum called tulip. It is a standardization of event loops for Python that allows multiple pieces of code to cooperate without each having to bring their own (possibly incompatible) event loop

While playing with archangel, I decided to incorporate Operating system like concepts such as the idea of a 'Process' and extend the protocol layer to register themselves with a manager when created so that i may list which requests are currently 'in flight' and which have completed. In a long polling situation this could be very handy to see who is connected and have a handle to manually kill a connection from a management interface (telnet or ssh shell in process or via a web interface). The event loop makes it incredibly easy to plug these sorts of different interfaces in without having to specifically support them with glue code to wire them into your event loop.

The main problem i encountered was how to register a process with the manager. While i don't mind a single object for an instance (for some definition of instance) a do not like singletons at the module level. Tulip makes the situation of running dissimilar apps running side by side MORE likely and actually possible so that solution is out.

I could have a Process object that has a reference to a Manager that objects could inherit, but then as above the manager needs to be a singleton as the scope you create the manager instance in may occur after the scope the object was defined in. I do have a solution for this in my dyno library for 'late binding' implementations of a dependency however while i wrote it, I would use it very sparingly and i feel there could be a better way to do it (play smart not hard in this case).

I could also use a decorator (which would be ideal), but once again i run into the same approaches as above.

What i ended up doing was a combination of the first 2 approaches. you create your protocol object as normal but inherit from Process. this adds some basic functions and hooks for things such as catching a kill signal to terminate a process early and registering of at_exit hooks for handling cleanup work.

To tie this into the ProcessManager i expose a register method that can also act as a decorator (for those cases where you are defining code inline after creating the manager as i do in my xing framework). As a decorator just takes a function and returns a (possibly modified) function it makes it very easy to plug into the start_serving method on an event loop, as an example the following:

server = loop.start_serving(lambda: ArchangelServer(), 'localhost', 8080)

becomes:

server = loop.start_serving(lambda: manager.register(ArchangelServer()), 'localhost', 8080)

Just like a decorator, the register function can be plugged in as if it was a component in a unix pipeline (ArchangelServer() | manager.register).

This register function acts as a 2nd stage __init__ and assigns a PID to the process and gives it a reference to the manager so that it may deregister itself on exit.

One lesson i learned and the reason for this entire article is that some of the things i am doing in __init__ should be done in __new__. Anything that is being set to a default value and not being passed in at initialization time (ie as Obj(<args>)) should be set using __new__. __init__ is specifically for initializing the object by doing setup work so it can be used and __new__ is for creating the object itself.

If this was a building, __new__ would be putting up the frame of the building (which is fairly standardized and doesn't change from building to building) and __init__ would be putting up the walls, putting in fixtures and basically doing anything that the buyer can change or specify. To put it in terms of roles, __new__ is for changes by the Architect (the writer of the code) while __init__ is for the buyer (the programmer using your code).

I hope to write more of these articles and talk about the design and my thoughts and musings of my other projects as well. hopefully others will be able to take something away from these posts. So stay tuned and let me know if you want to see more or hear about something specific ethier here or here.