Title: Hacking Guide to Lyntin Author: Will Guaraldi VERSION ======= $Id: HACKING,v 1.22 2003/01/01 00:36:25 willhelm Exp $ For more up-to-date information, please check the web-site: http://lyntin.sourceforge.net/ NOTES ===== This document provides a rough overview of how things in Lyntin are architected and how they work. The API documentation which is viewable online on the Lyntin site and is in the source code itself which comes with every copy of Lyntin (because it _is_ Lyntin) is where you will find the answers to all your questions and more! If you have questions, ask on the lyntin-devl mailing list. In depth knowledge of the Python programming language is required to really get into Lyntin since it's all written in Python. If you don't know Python, go visit http://www.python.org and familiarize yourself with the language, its constructs. and the Python community. SYNOPSIS ======== Lyntin was built for the hacker mudder. This is a special breed of person who muds and has a series of complex problems they want to solve and mere aliases and triggers just aren't going to do the trick. (At this point I should mention that I also use Lyntin to debug applications that use ASCII-oriented protocols like HTTP.) Tintin has a series of commands that implements a macro language. Building "macros" in this way is complex and solves a small set of problems, but the more complex problems really need a full language and a library to solve. The Tintin way of solving this is to keep adding commands that fill out the macro language to handle string manipulations and such. Lyntin takes a different approach leaving those sorts of things to Python. Lyntin is inextricably intertwined with Python so you can extend Lyntin to do anything without going through hoops: check your email, record URLs you see and open up a browser to visit them, export mudmail, building complex bots, web-servers, AIM clients, automappers.... There are a few different methods of hacking Lyntin to suit your needs. They are briefly outlined here and described in more detail in their respective sections: 1. editing the source code 2. building modules 3. command line Before we continue, we should ramble on about Lyntin's architecture first and how to do things. Before that, we'll cover Lyntin's philosophy. PHILOSOPHY ========== If you enter into the Python interactive mode (by typing "python" at the prompt) and type "import this" you get a list of Python's design philosophies. When working on Lyntin, we have similar thoughts: * Beautiful is better than ugly. * Simple is better than complex. * Readability before most things. * Even readable things need documentation. * Optimization if we have to. * Leave power and choice in the hands of the user. * Get it right the third time--the first two times explore the issue and perhaps implement solutions until the true solution is obvious. * Discussion before implementation. * Possibly specification and test cases before implementation. LYNTIN ARCHITECTURE =================== FILE STRUCTURE -------------- The Lyntin directory structure looks like this: lyntinng/ |-- ui/ -- user interfaces |-- modules/ -- Lyntin modules (dynamically loaded at runtime) |-- help/ -- Lyntin help files |- errata/ -- random stuff for testing and other stuff DESIGN ------ Lyntin is multi-threaded and as such is built around an event queue. The event queue is synchronized and allows us to have multiple threads going without creating mutexes for everything involved. As such, spin things off into events whenever you can--especially when you're interacting with the various data structures. The engine holds the event queue and drives Lyntin. Most Lyntin things occur via an event. Lyntin starts with the StartupEvent and ends with the ShutdownEvent. The event queue is a synchronized queue and thus gives us a centralized way of synchronizing all the threads accessing Lyntin internals without causing them to block. The engine holds: * the HistoryManager * the ThreadManager * the HelpManager * the list of Sessions * the user interface * a series of managers which manage aliases, actions, gags, highlights, substitutes, variables and other such things All these things exist on the _engine scoping_. Each session has: * its own SocketCommunicator which abstracts socket connections and network issues * a Ticker which allows you to do things at intervals These things exist on a _session scoping_ -- there are one of these for every session instance. API --- The API for accessing Lyntin's internals is in the "exported" module. This module contains interfaces to Lyntin's internals with the additional social contract that we Lyntin developers will not change the API between minor versions of Lyntin. Lyntin's internals actually use the exported module for accessing its own internals. The reasons for this being that it gives us a centralized place for fixing behavior bugs. If you plan on building Lyntin modules, do use the exported API--it will save you much consternation when dealing with multiple Lyntin releases. MANAGERS -------- Managers manage things. Most managers subclass the "manager.Manager" class. It doesn't really provide a lot of functionality, but it allows us to group them all and treat them all the same. Adding new managers is much easier because of this. Also, managers register themselves with the engine via the "exported.add_manager" function. The engine will cycle through registered managers for things like status and persistence. In addition, registered managers get told when the user has created a new session and when the user has ended a session through the "addSession" and "removeSession" methods. For more information, check out the "manager" module in the API. COMMANDS -------- Lyntin comes with a series of commands for manipulating aliases, actions, highlights, and various other things. Both the commands and the managers that hold the data are all defined in various Lyntin modules in the "modules/" subdirectory and are loaded up at Lyntin startup. Lyntin has an extremely powerful argument parser that allows Lyntin module writers to worry about the functionality of their command without having to deal with parsing out the arguments, type-checking, raising exceptions with command syntax help, and data conversion. Commands are stored internally in the "CommandManager". They are global to Lyntin--there are no commands that only exist in the session they were declared in. Commands, however, are executed in a specific session. When they are executed, they are given three arguments: the session object for the session they were executed in, an argument dictionary from the argument parser, and the exact input the user typed. Look at the modules/lyntincmds.py and modules/tintincmds.py modules for command examples. Additionally, check out the Lyntin module repository on http://lyntin.sourceforge.net/ for more examples. ARGUMENT PARSER --------------- tbd. LYNTIN HELP ----------- Lyntin has a comprehensive help system that can be accessed in-game through the "#help" command. The help material is all organized and manipulated by the "HelpManager". Help is organized as topics which exist in a hierarchy of categories and it comes from a couple of sources: 1) dynamically loaded at Lyntin startup from files located in the help subdirectory and ending with .tpc extension 2) when commands are registered via the exported.add_command function 3) by any module using the exported.add_help function There has been some effort to move the README into in-game help topics and then to actually produce the README file from the in-game help topics. At some point this will allow us to create a Lyntin manual which details all the commands and their syntaxes. USER DATA HANDLING ------------------ User data (sometimes called "input") is received from the "ui.BaseUI" subclasses and tossed into an "event.InputEvent". When the engine handles the event, it is first passed to the engine's "handleUserData" method which deals with global things such as splitting the input into a series of separate commands. Read the docstrings for more information. The user input is then passed to the session in question's "handleUserData" method which does some stuff and passes it through the "user_filter_hook". This is where the magic happens. All user input handlers in Lyntin (the alias manager, the variable manager ...) register with the user_filter_hook at various priorities. The "user_filter_hook" then cycles through the user filters in order of priority. For more specific information, read through the docstrings of the methods and classes involved. MUD DATA HANDLING ----------------- Data can also come from the mud. Mud data comes from the net module's "SocketCommunicator" and is encapsulated in an "event.MudEvent". tbd. THE UI ------ All user interfaces extend the "ui.BaseUI" class in the "ui" module. It has a series of methods which need to be overridden to give the ui functionality. In addition, ui's can specify additional Lyntin commands, they can bind to Lyntin hooks (most ui's bind to the "startup_hook" allowing them to perform certain functionality at Lyntin startup rather than when the module is imported or when the ui is instantiated), and anything else that modules can do. Lyntin comes with a text ui which can be used over telnet/ssh or in a command shell. It also comes with a tk gui. The tk gui has functionality like split history screen (pgup and pgdown), its own command history buffer (up and down) and numpad bindings (VK__NUMPAD0 through VK__NUMPAD9). There's also a very rudimentary and unstable curses ui. Feel free to extend the ui code and send in patches. Or even write your own ui based on another widget set or whatever you want to do. For more information, read the "ui" module. HOOKS ----- The engine is augmented by a series of hooks which allow modules to execute functions without having to change Lyntin's internals. Examples of this would be the "startup_hook" and "shutdown_hook". Any functions that hook into these hooks will be executed upon startup or shutdown of Lyntin. Lyntin also uses these hooks to implement its functionality. For example, the Tk ui uses the "startup_hook" to register with the "to_user_hook", add the tkui help topic, and start the ui user listener thread. Hooks and the Hook class are defined in the "hooks" module as is a whole lot of documentation on which hooks exist, and what is passed to them. SOURCE CODE HACKING =================== Lyntin is entirely written in Python. The source code is well organized (as of 3.0) and documentation exists on the web-site as well as in the source code distribution on how Lyntin is architected and details on the internals. Lyntin is distributed under the GPL. You should read this prior to making changes to the source code and distributing them so you don't accidentally find yourself in violation. If you patch Lyntin, we'd love to see your work! Send it to the Lyntin developers list and depending on what the patch does, it may get incorporated into the Lyntin base code. Note, you must disclaim your copyright on all code you submit so we can assign copyright to the FSF. If you don't, we won't accept it. LYNTIN MODULE HACKING ===================== Lyntin modules are cool and are what gives Lyntin much of its functionality. Because Lyntin modules are written in Python, they have full access to the Python API--they're not run in any kind of rexec bastion or anything. Every Lyntin module is written in Python. Lyntin modules can be loaded by Lyntin in two different ways. The first is at Lyntin startup when Lyntin lookes through all the Python modules in the Lyntin "modules/" directory excluding modules whose names start with a _. The second is using the "#import" Lyntin command: #import lyntinuser #import modules.highlight When the Lyntin modules is loaded, Lyntin will check the module for a "load" function and if it exists, Lyntin will execute it. This function lets you separate Python importing of the module from the Lyntin initialization. All the Lyntin modules that come with Lyntin use the "load" function for binding to Lyntin hooks, adding new commands, adding new help text, and doing other initialization sorts of things. When using the "#import" command to load a module, it first checks to see if the module is already loaded. If it is and the module has an "unload" function it will call this prior to reloading the module and calling the "load" function if it exists. We typically use the "unload" function for unregistering functions with Lyntin hooks as well as removing commands. This allows you to cleanly reload modules you're testing/debugging without stopping and restarting your Lyntin session. Lyntin modules should use the "exported" module to access Lyntin internals. This API is guaranteed not to change (much) over versions of Lyntin. In the modules/ subdirectory, there's a module called modutils. This module holds some helper functions for making things a bit easier to deal with. Consult the documentation there for more guidance. Use the modules that come with Lyntin as a series of examples on writing your own modules. WRITING MESSAGES TO THE UI -------------------------- In your modules, you should import the "exported" module. In there are a series of write_* functions which write various Message types to the user interface. For example: import exported def fancy_cmd(ses, args, input): text = args[0] if not text: exported.write_error("error: there is not text") return exported.write_message("text: %s" % text) See the documentation in the "exported" module for more details. ANSI colors are supported in the text you write to the ui. One way of doing this is as follows: exported.write_message("\033[1;34mhellothere\033[0m") However, that's difficult to read and subject to many potential tyops. As such, there's a get_color function in the "ansi" module which makes this a little easier: import ansi exported.write_message("%shello%s" % ansi.get_color("light blue"), ansi.get_color("default")) ANSI coloring is not a binary thing--there's no end tag. It's also not very nestable. Make sure to use "default" when ending your ANSI coloring--that returns the ANSI color to the default values for options, foreground, and background. COMMAND LINE HACKING ==================== Lyntin allows you to execute arbitrary Python at the command line. If you have a "lyntinuser" Lyntin module, then the code will be executed in there. Otherwise the code is executed in the "advanced" Lyntin module. For example: #@print "hello" will print hello to the console. There are some significant differences between Lyntin 2.x and 3.x in how this works. In the 2.x series, raw Python was executed in the "user" module. It was encouraged that you add your own functions to the "user" module and then these would be available to you at the Lyntin command line. In 3.0, "lyntinuser" is a Lyntin module in the modules directory. It does not come with Lyntin--it's something you can create if you want to build your own environment to execute Python commands in. If you don't have a "lyntinuser" module, raw Python code gets executed in the "advanced" module where the #@ functionality exists. A basic "lyntinuser" module could be as follows:: """ This is my lyntinuser.py module. I can do whatever I want in here. I can store this module in my Lyntin moduledir (set at the command line using --moduledir ) or in Lyntin's modules subdirectory along with alias.py, action.py and the rest of the gang. """ import exported myvar = "hello" def myfunc(): exported.write_message(myvar) def load(): """ You can put code here for initializing your module when it's loaded here. Typically this is registering with Lyntin hooks and adding Lyntin commands. Look at the other Lyntin modules for examples. """ pass def unload(): """ Put code here for uninitializing your module when it's unloaded. Typically this is the reverse of the load module: unregistering functions with Lyntin hooks, remove Lyntin commands, removing Lyntin help topics... Look at other Lyntin modules for examples. """ pass Now at the Lyntin command prompt I can do things like:: > #@print myvar hello > #@myfunc() lyntin: hello THE END ======= That about covers it. Read through the API and the source code. If you have questions, subscribe to the lyntin-devl mailing list and ask us. http://sourceforge.net/mail/?group_id=6730 Happy hacking! the Lyntin development folks http://lyntin.sourceforge.net/