TITLE OF PAPER: Translucent Inter-Process Service Migration URL OF PRESENTATION: _URL_of_powerpoint_presentation_ PRESENTED BY: Jean-Paul Calderone REPRESENTING: divmod.org CONFERENCE: PyCON 2004 DATE: 20040324 LOCATION: _venue_and_room_in_venue_ -------------------------------------------------------------------------- REAL-TIME NOTES / ANNOTATIONS OF THE PAPER: {If you've contributed, add your name, e-mail & URL at the bottom} What is migration? hand-off of application level duties from one process to another database server moving load to another process web server moving connection to user process existing stat passed from old code to new code add features incorporate new features why do we need it buggy software needs to be fixed new features need to be rolled out too simple solutions disconnect all users shutdown old server restart reconnect issues are that all network connections are broken startup overhead incurred unexpected difficulties leave service offline Upgrading code In-process with reload() difficult to reverse leaves two version of code side-by-side in one process upgrade on a permodule basis Intra-Process Incremental updates Cheaper code updates means more frequent code updates More frequent code updates means smaller code updates Upgrade on a per-user basis Testable state can be migratged from live server to test server for testing, user connections can be ignored failed migration test leads to defunct test/staging server, leave production server unchanged Problems with intra-process migration requires all state to be serializable or otherwise persistent most data-driven services require this anyway if user data doesn't persist, the service isn't very useful but migration can intoduce unexpedted serialization requirements Twisted Factories / Twisted Protocols execl() replace existing process with a new one file descriptors remain open Used by numerous MUDs Players happily slashing up monsters Administrator enters command Entire game is paused (players are interrupted) Player data is saved, world-state is not Process re-executes Player data loaded from disk, world-state initialized from defaults Game-play continues (players complain) Problems with execl() Monolithic (you can only replace the entire process) Difficult to test Irreversible Failure can lead to long periods of downtime Migration between processes on one machine Bring up new server Serialize and transfer state from old server to new server Bring down old server ... but doesn't scale up without massively parallel hardware Migration between processes on multiple machines Scales way, way up Additional protection from hardware failure More benefits such as useful load balancing Problems with mltiple machines Socket tricks don't work Client disconnnect is almost always required Server port passes from old to new Front-end machines can mask over back-end switching Example of simple ( Example of complex(stateful)migration start a python interpreter accessible via telnet connect and populate a namespace start a new process client connection and server port pass from old to new old process exits Migration - a demonstration - part 1 >>>> import sys, os >>>> sys.version >>> os.getpid() Example: Python process "upgraded" from 2.2 to 2.3; user doesn't notice unless he looks at the version number or the process ID. Followed by migration walkthrough explaining internal Twisted workings. Integration with exisiting software serialization code should be kept separat from application code Few changes to application code are required some are unavoidable class ShellFactory(telnet.ShellFactory): def __init__(self Designe for migratibility Maintain clear separation between code and data consider generators, which tend to conflate them pickle/jelly cannot serialize generators - (current stackless can serialize generators because it also serializes the code which is being run along with all the stack frames)(dp) Maintain clear separation between different services Conclusions Multitude of migration options with Python Migration without service interruption Future Directions Increase flexibility of existing migration code support more special objects (eg. scheduled events) port to new version of PB, when available UI for managing migrations render an object graph Allow user to select objects to migrate Allow user to select references to break Load Balancing Migrate services between different machines Questions Q: Why was Jelly separate from pickle A: Security reasons (Jelly is more secure than pickle). Q: is there a java implementation of Jelly A: yes Q: is the state of the code migrated A: no - only data Q: Socket migration A: Unix-only library for within a system, proxy-based approach over multiple machines (requires a common proxy server to allow the ip information to be passed) generators are a problem because they contain code and data -------------------------------------------------------------------------- REFERENCES: {as documents / sites are referenced add them below} Paper: http://www.python.org/pycon/dc2004/papers/51/migration.html -------------------------------------------------------------------------- QUOTES: -------------------------------------------------------------------------- CONTRIBUTORS: {add your name, e-mail address and URL below} Ted Leung, twl@osafoundation.org, http://www.sauria.com/blog Donovan Preston, dp@divmod.org, http://ulaluma.com/pyx Mike Taylor, bear@code-bear.com Abhay Saxena, ark3@email.com -------------------------------------------------------------------------- E-MAIL BOUNCEBACK: {add your e-mail address separated by commas } -------------------------------------------------------------------------- NOTES ON / KEY TO THIS TEMPLATE: A headline (like a field in a database) will be CAPITALISED This differentiates from the text that follows A variable that you can change will be surrounded by _underscores_ Spaces in variables are also replaced with under_scores This allows people to select the whole variable with a simple double-click A tool-tip is lower case and surrounded by {curly brackets / parentheses} These supply helpful contextual information. -------------------------------------------------------------------------- Copyright shared between all the participants unless otherwise stated...