Archangel Design Notes: Part 1
I have been playing with tulip in python 3.3 while developing archangel and come across a couple of interesting problems that i thought were worth documenting mainly for my own personal usage, But also with the hope that it may be useful for others.
Archangel uses the new event loop being developed by Guido van Rossum called tulip. It is a standardization of event loops for Python that allows multiple pieces of code to cooperate without each having to bring their own (possibly incompatible) event loop
While playing with archangel, I decided to incorporate Operating system like concepts such as the idea of a 'Process' and extend the protocol layer to register themselves with a manager when created so that i may list which requests are currently 'in flight' and which have completed. In a long polling situation this could be very handy to see who is connected and have a handle to manually kill a connection from a management interface (telnet or ssh shell in process or via a web interface). The event loop makes it incredibly easy to plug these sorts of different interfaces in without having to specifically support them with glue code to wire them into your event loop.
The main problem i encountered was how to register a process with the manager. While i don't mind a single object for an instance (for some definition of instance) a do not like singletons at the module level. Tulip makes the situation of running dissimilar apps running side by side MORE likely and actually possible so that solution is out.
I could have a Process
object that has a reference to a Manager
that
objects could inherit, but then as above the manager needs to be a singleton
as the scope you create the manager instance in may occur after the scope the
object was defined in. I do have a solution for this in my
dyno library for 'late binding'
implementations of a dependency however while i wrote it, I would use it very
sparingly and i feel there could be a better way to do it (play smart not hard
in this case).
I could also use a decorator (which would be ideal), but once again i run into the same approaches as above.
What i ended up doing was a combination of the first 2 approaches. you create
your protocol object as normal but inherit from Process. this adds some basic
functions and hooks for things such as catching a kill
signal to terminate a
process early and registering of at_exit
hooks for handling cleanup work.
To tie this into the ProcessManager i expose a register
method that can also
act as a decorator (for those cases where you are defining code inline after
creating the manager as i do in my
xing framework). As a
decorator just takes a function and returns a (possibly modified) function it
makes it very easy to plug into the start_serving
method on an event loop,
as an example the following:
server = loop.start_serving(lambda: ArchangelServer(), 'localhost', 8080)
becomes:
server = loop.start_serving(lambda: manager.register(ArchangelServer()), 'localhost', 8080)
Just like a decorator, the register function can be plugged in as if it was a
component in a unix pipeline (ArchangelServer() | manager.register
).
This register function acts as a 2nd stage __init__
and assigns a PID to the
process and gives it a reference to the manager so that it may deregister
itself on exit.
One lesson i learned and the reason for this entire article is that some of
the things i am doing in __init__
should be done in __new__
. Anything that is
being set to a default value and not being passed in at initialization time (ie
as Obj(<args>)
) should be set using __new__
. __init__
is specifically for
initializing the object by doing setup work so it can be used and __new__
is
for creating the object itself.
If this was a building, __new__
would be putting up the frame of the
building (which is fairly standardized and doesn't change from building to
building) and __init__
would be putting up the walls, putting in fixtures
and basically doing anything that the buyer can change or specify. To put it
in terms of roles, __new_
_ is for changes by the Architect (the writer of
the code) while __init__
is for the buyer (the programmer using your code).
I hope to write more of these articles and talk about the design and my thoughts and musings of my other projects as well. hopefully others will be able to take something away from these posts. So stay tuned and let me know if you want to see more or hear about something specific ethier here or here.