Archangel Design Notes: Part 1
I have been playing with tulip in python 3.3 while developing archangel and come across a couple of interesting problems that i thought were worth documenting mainly for my own personal usage, But also with the hope that it may be useful for others.
Archangel uses the new event loop being developed by Guido van Rossum called tulip. It is a standardization of event loops for Python that allows multiple pieces of code to cooperate without each having to bring their own (possibly incompatible) event loop
While playing with archangel, I decided to incorporate Operating system like concepts such as the idea of a 'Process' and extend the protocol layer to register themselves with a manager when created so that i may list which requests are currently 'in flight' and which have completed. In a long polling situation this could be very handy to see who is connected and have a handle to manually kill a connection from a management interface (telnet or ssh shell in process or via a web interface). The event loop makes it incredibly easy to plug these sorts of different interfaces in without having to specifically support them with glue code to wire them into your event loop.
The main problem i encountered was how to register a process with the manager. While i don't mind a single object for an instance (for some definition of instance) a do not like singletons at the module level. Tulip makes the situation of running dissimilar apps running side by side MORE likely and actually possible so that solution is out.
I could have a
Process object that has a reference to a
objects could inherit, but then as above the manager needs to be a singleton
as the scope you create the manager instance in may occur after the scope the
object was defined in. I do have a solution for this in my
dyno library for 'late binding'
implementations of a dependency however while i wrote it, I would use it very
sparingly and i feel there could be a better way to do it (play smart not hard
in this case).
I could also use a decorator (which would be ideal), but once again i run into the same approaches as above.
What i ended up doing was a combination of the first 2 approaches. you create
your protocol object as normal but inherit from Process. this adds some basic
functions and hooks for things such as catching a
kill signal to terminate a
process early and registering of
at_exit hooks for handling cleanup work.
To tie this into the ProcessManager i expose a
register method that can also
act as a decorator (for those cases where you are defining code inline after
creating the manager as i do in my
xing framework). As a
decorator just takes a function and returns a (possibly modified) function it
makes it very easy to plug into the
start_serving method on an event loop,
as an example the following:
server = loop.start_serving(lambda: ArchangelServer(), 'localhost', 8080)
server = loop.start_serving(lambda: manager.register(ArchangelServer()), 'localhost', 8080)
Just like a decorator, the register function can be plugged in as if it was a
component in a unix pipeline (
ArchangelServer() | manager.register).
This register function acts as a 2nd stage
__init__ and assigns a PID to the
process and gives it a reference to the manager so that it may deregister
itself on exit.
One lesson i learned and the reason for this entire article is that some of
the things i am doing in
__init__ should be done in
__new__. Anything that is
being set to a default value and not being passed in at initialization time (ie
Obj(<args>)) should be set using
__init__ is specifically for
initializing the object by doing setup work so it can be used and
for creating the object itself.
If this was a building,
__new__ would be putting up the frame of the
building (which is fairly standardized and doesn't change from building to
__init__ would be putting up the walls, putting in fixtures
and basically doing anything that the buyer can change or specify. To put it
in terms of roles,
__new__ is for changes by the Architect (the writer of
the code) while
__init__ is for the buyer (the programmer using your code).
I hope to write more of these articles and talk about the design and my thoughts and musings of my other projects as well. hopefully others will be able to take something away from these posts. So stay tuned and let me know if you want to see more or hear about something specific ethier here or here.