============ PyBackground ============ .. contents:: Table of Contents License ======= Copyright (c) 2005-2008, David Baird All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Executive Summary ================= This module provides very, very easy multithreading. With this module, you will have a hard time *resisting* creating threads - you'll have to restrain yourself because it is so easy and so much fun. My friends and I like to call it "adhoc multithreading." See ExampleUsage_ for a demonstration. Since PyBackground is based on Python's threading module, it will work in Linux, Mac OS, Windows and anywhere else Python's threading module works. By the way, if you have too much fun with this module, perhaps that is a sign you should be using a language such as Erlang or Io instead of Python. History ======= Revision 12 (2008 February 16), David Baird Module was renamed from ``background`` to ``pybackground``. Background ========== This module was inspired by the Io_ programming language (which in turn was inspired by other programming languages). In Io, by merely putting the '@' (at) sign in front of a method to create an "actor," Io will execute that method in a background thread and return a "transparent future." A transparent future is a place holder object that represents the return value of the method. When you try to actually access the transparent future, it will block until the method terminates. But as long as you don't access the future, then you can keep doing stuff in the foreground while the background threads happily execute. This "background" module for Python accomplishes a very similar function. Instead of an '@' symbol, the "background" function is used instead as described in the next section. Io, unlike this module, is also capable of another trick: deadlock detection. .. _Io: http://www.iolanguage.com/ Technical Considerations ======================== Asynchronous I/O: Threads v. Select ----------------------------------- This module uses the threading implementation of Python to accomplish this. Python makes use of operating system threads rather than managing its own light-weight cooperative threads. With NPTL, multithreading in Linux became a lot more fun, but there are other ways to deal with high-latency tasks. One way in particular is the "select" system call which is not directly addressed by this module. (But you could use this module to wrap up something else that used the "select" system call). Possible Problems with the Proxy -------------------------------- The transparent future is based on a Proxy. Proxies can be tricky to design. In particular, some private methods (methods written like ``__foobar__(...)``) might not be proxied properly. If you have a problem, let me know and/or send me a patch and I will check it out. Deadlocks --------- This module has no ability (yet) to resolve certain cases of deadlocks; A deadlock is created when two modules are waiting on each other before they can complete - but they will never complete! That is a deadlock. Global Interpreter Lock (GIL) ----------------------------- Pybackground is based on Python's threading. Therefore, it is still subject to the GIL. If there is a way around this, can someone ping me with an email? I will then rewrite PyBackground so that it can interoperate with other threading implementations. .. ExampleUsage_ .. _ExampleUsage: Example Usage ============= No Timeouts ----------- Here is an example of how to use the background module:: import time from pybackground import background def very_slow_task(arg): # do something slow like download a webpage # (or sleep for 5 seconds) time.sleep(5) return arg t0 = time.time() # Instead of direcly calling very_slow_task(1), do this: x = background(very_slow_task)(1) y = background(very_slow_task)(2) z = background(very_slow_task)(3) print z, y, x # >>> (wait 5 seconds then print) 3 2 1 print time.time() - t0 # >>> 5.00204801559 You can also use this as a Python decorator:: import time from pybackground import background @background def very_slow_task(arg): # do something slow like download a webpage # (or sleep for 5 seconds) time.sleep(5) return arg t0 = time.time() x = very_slow_task(1) y = very_slow_task(2) z = very_slow_task(3) print z, y, x # >>> 3 2 1 print time.time() - t0 # >>> 5.00449609756 This also works with classes:: import time from pybackground import background class SomeClass(object): def method(self, arg): time.sleep(5) return arg my_object = SomeClass() x = background(my_object.method)(1) y = background(my_object.method)(2) z = background(my_object.method)(3) print z, y, x class SomeOtherClass(object): @background def method(self, arg): time.sleep(5) return arg my_other_object = SomeOtherClass() x = my_other_object.method(1) y = my_other_object.method(2) z = my_other_object.method(3) print z, y, x Timeouts -------- .. WARNING:: The timeout feature currently lacks the ability to kill processes. So, it is possible that even though a background processes timed-out that it might still be running and consuming system resources. Timeouts are currently implemented by passing a time parameter to the ``join`` method of threading.Thread objects. Here is an example of using timeouts:: import time from pybackground import background, Timeout def very_slow_task(arg): # do something slow like download a webpage # (or sleep for 5 seconds) time.sleep(5) return arg t0 = time.time() # This will cause a timeout after 1 second: x = background(very_slow_task, timeout=1.0)('foobar1') # This will cause a timeout after 6 seconds: y = background(very_slow_task, timeout=6.0)('foobar2') print x == Timeout # >>> True print x, y # >>> Timeout foobar2 print time.time() - t0 # >>> 4.97759008408 Credits ======= http://storytotell.org/ - this is the guy who introduces me to many neat languages including, you guessed it, Io. http://auriga.wearlab.de/~alb/python/ - has some neat Python scripts including "dataflow.py" which essentially does the same thing as this module. http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/84317 - "Easy threading with Futures" circa 2002, by David Perry Download ======== Latest version: - `pybackground-16.tar.gz `_ (or just have a look at `pybackground.py `_ directly) Older versions: - (none available)