#LyX 1.6.9 created this file. For more info see http://www.lyx.org/ \lyxformat 345 \begin_document \begin_header \textclass report \begin_preamble \usepackage{indentfirst} \usepackage{tocloft} \usepackage{calc} \date{} \usepackage[section] {placeins} \def\myClearpage{% \ifvmode \ifnum \@dbltopnum =\m@ne \ifdim \pagetotal <\topskip \hbox{}% \fi \fi \fi % \newpage \write\m@ne{}% \vbox{}% \penalty -\@Mi } \def\myCleardoublepage{\myClearpage\if@twoside \ifodd\c@page\else \hbox{}\if@twocolumn\hbox{}\fi\fi\fi} \usepackage{morefloats} \usepackage{graphicx} \usepackage{subfig} \usepackage{tocloft} \renewcommand{\cftchapfont}{\bfseries} \renewcommand{\cftchappagefont}{\bfseries} \renewcommand{\cftchappresnum}{Chapter } \renewcommand{\cftchapnumwidth}{6em} \oddsidemargin 0.5in \textwidth 6in \topmargin 0.0in \textheight 8.0in \setlength\topskip{24pt} \footskip 0.75in \usepackage[compact]{titlesec} \titleformat{\chapter}[display] {\vskip-8em\normalfont\bfseries} {\LARGE\raggedright\thechapter} {14ex} {\vspace{-20ex}% \LARGE\raggedleft} [\vspace{1ex}% {\titlerule[1pt]}] \usepackage[absolute]{textpos} \usepackage{fancyheadings} \pagestyle{fancy} \lhead{\thechapter} \usepackage[hmargin=3cm,vmargin=3.5cm]{geometry} \end_preamble \use_default_options true \language english \inputencoding auto \font_roman lmodern \font_sans default \font_typewriter default \font_default_family rmdefault \font_sc false \font_osf false \font_sf_scale 100 \font_tt_scale 100 \graphics default \paperfontsize default \spacing onehalf \use_hyperref false \papersize a4paper \use_geometry false \use_amsmath 1 \use_esint 1 \cite_engine natbib_authoryear \use_bibtopic false \paperorientation portrait \paperwidth 11in \paperheight 8.5in \leftmargin 1.25in \topmargin 0in \rightmargin 1in \bottommargin 1.7in \secnumdepth 2 \tocdepth 2 \paragraph_separation indent \defskip medskip \quotes_language english \papercolumns 1 \papersides 1 \paperpagestyle empty \tracking_changes false \output_changes false \author "" \author "" \end_header \begin_body \begin_layout Standard \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{textblock*}{ \backslash paperwidth}(0mm,40mm) \end_layout \begin_layout Plain Layout \backslash begin{center} \end_layout \begin_layout Plain Layout \backslash includegraphics[width= \backslash paperwidth/2]{graphics/logo} \end_layout \begin_layout Plain Layout \backslash end{center} \end_layout \begin_layout Plain Layout \backslash end{textblock*} \end_layout \end_inset \end_layout \begin_layout Title Generic Game Server \end_layout \begin_layout Author Niklas Landin \begin_inset Newline newline \end_inset Richard Pannek \begin_inset Newline newline \end_inset Matias Petterson \begin_inset Newline newline \end_inset Jonatan Pålsson \end_layout \begin_layout Abstract This is the abstract! \end_layout \begin_layout Standard \begin_inset ERT status open \begin_layout Plain Layout \backslash pagenumbering{roman} \end_layout \begin_layout Plain Layout \backslash setcounter{page}{5} \end_layout \begin_layout Plain Layout \backslash renewcommand \backslash contentsname{Table of Contents} \end_layout \begin_layout Plain Layout \backslash renewcommand{ \backslash cfttoctitlefont}{ \backslash hfill \backslash Large} \backslash renewcommand{ \backslash cftaftertoctitle}{ \backslash hfill} \end_layout \begin_layout Plain Layout \backslash renewcommand \backslash cftpartdotsep{6.6} \end_layout \begin_layout Plain Layout \backslash renewcommand \backslash cftchapdotsep{6.6} \end_layout \end_inset \begin_inset CommandInset toc LatexCommand tableofcontents \end_inset \end_layout \begin_layout Standard \begin_inset Newpage newpage \end_inset \end_layout \begin_layout Standard \begin_inset ERT status open \begin_layout Plain Layout \backslash pagenumbering{arabic} \end_layout \begin_layout Plain Layout \backslash setcounter{page}{1} \end_layout \end_inset \end_layout \begin_layout Chapter Introduction \end_layout \begin_layout Section Background \end_layout \begin_layout Section Purpose \end_layout \begin_layout Standard The purpose of the GGS project is to create a \emph on scalable \emph default and \emph on fault tolerant \emph default server, while still allowing the server to be as \emph on generic \emph default as possible. These three italicised terms need some explanation. \end_layout \begin_layout Standard Scalability in computer science is a large topic and is commonly divided into sub-fields, two of which are \emph on structural scalability \emph default and \emph on load scalability \emph default \begin_inset CommandInset citation LatexCommand citet key "Bondi:2000:CSI:350391.350432" \end_inset . These two issues are addressed in this thesis. Structural scalability means expanding an architecture, e.g. adding nodes to a system, without requiring modification of the system. Load scalability means using the available resources in a way which allows handling increasing load, e.g more users, gracefully. \end_layout \begin_layout Standard Fault tolerance is used to raise the level of \emph on dependability \emph default in a system, so that the dependability is high even in presence of errors. Dependability is defined as the statistical probability of the system functioni ng as intended at a given point in time. Fault tolerance is defined as the property of a system to always follow a specification, even in the presence of errors. The specification could take the form of error handling procedures which activate when an error occurs. This means that a fault tolerant, dependable system, will have a very high probability of functioning at a given point in time, and is exactly what is desired. \begin_inset CommandInset citation LatexCommand citet key "Gartner:1999:FFD:311531.311532" \end_inset \end_layout \begin_layout Subsection Generic \end_layout \begin_layout Standard A generic game server has to be able to run different client-server network games regardless of the platform the clients are running on. It should be able to run network games of different type, a very rough separation would be in \emph on real time games \emph default and turn based games. \end_layout \begin_layout Subsubsection Application server targeted on games \end_layout \begin_layout Standard The server is something like a application server designed to help to run games. A application server is different from a file or print server, which only serves resources to the clients, it serves processing ability and time. \end_layout \begin_layout Standard The most common type of application servers are webservers where you run a web-application within the server. The application-server handles all the network-/inter process-stuff and offers hooks and helpers for your application to use the resources, some examples for such web-application-servers are the Glassfish-Server which allows you to run applications written in Java, or the Google App Engine where you can run applications written in Python or some language which runs in the Java Virtual Machine. \end_layout \begin_layout Standard But you can find specialized application servers within the industry where one central server serves processing power for different independent robotic-cl ients. Or in Academia where one mainframe is used as a cerver for different clients to run different applications which are doing havy calculations and return the result to the clients. \end_layout \begin_layout Standard You could too think of a database server as a application server. You send it different \begin_inset Quotes eld \end_inset scripts \begin_inset Quotes erd \end_inset , e.g. bis SQL queries or JavaScript, the server runs them and returns the evaluated data to the clients. It is perhaps not the best example but just so you see how different applicatio n servers can be. \end_layout \begin_layout Standard One of the purposes of this thesis is to investigate how we can make a game server as generic, that is so that you are not limited to just one game, as possible. Some important helpers which are needed in network games and therefore should be offered by a generic game server are for example the abstraction of the network layer, a data store, and others, which will be discussed more in detail later in the thesis. \end_layout \begin_layout Subsubsection Different types of games \end_layout \begin_layout Standard In real time games all players are playing at the same time simultanously together. Latency is a big problem here, a typical round trip time for such games is one of 50 to 150 ms and everything above 200 ms is reported to be intolerabl e \begin_inset CommandInset citation LatexCommand citet key "Farber:2002:NGT:566500.566508" \end_inset . Examples for such games are most of the first person shoters with multiplayer ability like Counter Strike, Call Of Duty and \emph on MMORPG's \emph default (Massively multiplayer online role-playing game) like World Of Warcraft, Starcraft, Ultima Online, etc. \end_layout \begin_layout Standard In turn based games each player has to wait for her turn. Latency is not a problem because even if a round trip takes a bigger amount of time, the gameplay does not require fast interactions between the players and it will not be noticed. Examples are board and card games like chess, poker or Carcassonne played online, as well as multiplayer games like \begin_inset Quotes erd \end_inset Hattrick - The online football manager game \begin_inset Quotes erd \end_inset or the \begin_inset Quotes eld \end_inset Wheel of Fortune \begin_inset Quotes erd \end_inset . \end_layout \begin_layout Standard Both game types have varying difficulties and needs when it comes to implementin g them, a Generic Game Server should address all of them and help the developer to acomplish his goal. \end_layout \begin_layout Section Challenges \end_layout \begin_layout Standard The word \emph on generic \emph default in GGS implies that the system is able to run a very broad range of different code, for instance code written in different programming languages, in addition to a broad range of different game types. In order to support this, a virtual machine (VM) for each \emph on game development language \emph default (hereafter GDL for brevity) is used. \end_layout \begin_layout Standard No hard limit has been set on which languages can be used for game development on GGS, but there are several factors which decide the feasibility of a language; \end_layout \begin_layout Itemize How well it integrates with Erlang, which is used in the core GGS system \end_layout \begin_layout Itemize How easy it is to send messages to the virtual machine of the GDL from GGS \end_layout \begin_layout Itemize How easy it is to send messages from the GDL VM to GGS \end_layout \begin_layout Standard Internally, the GDL VM needs to interface with GGS to make use of the helpers and tools that GGS provides. Thus an internal API has to be designed for use in interacting with GGS. This API is ideally completely independent of the GDL, and reusable for any GDL. \end_layout \begin_layout Standard The communication with gaming clients has to take place over a protocol. Ideally a standard protocol should be used, in order to shorten the learning curve for developers, and also make the system as a whole less obscure. A large challenge during this project is to decide whether an existing protocol can be used, and if not, how a new protocol can be designed which performs technically as desired, while still being familiar enough to existing developers. \end_layout \begin_layout Standard A great deal of work is devoted to make GGS \emph on reliable \emph default . This includes ensuring that the system scales well, and to make sure it is fault tolerant. In order to facilitate scalability, we need a storage platform which is accessible and consistent among all of GGS, this is also investigated. \end_layout \begin_layout Section Delimitations \end_layout \begin_layout Standard The implementation of the GGS protocol, together with storage possibilities, server capacity, and game language support imposes some limitations on the project. To get a functional prototype some limits must be set on the types games that can be played on the prototype. \end_layout \begin_layout Standard The UDP protocol will not be implemented, only TCP, the main reason behind this is a strict timetable. This decision means that games that requires a high speed protocol will not be supported by the GGS prototype. Another limitation necessary to set on the system is the possibility to have huge game worlds. Due to the limited capability of threading in many GDL VM:s, GGS will not support \emph on massively multiplayer online role playing games \emph default (MMORPG) games such as \emph on World of Warcraft \emph default or \emph on EVE Online \emph default as it is not possible to implement and test something that complex within the projects timetable. \end_layout \begin_layout Standard The GGS is only a small prototype and tests will be performed on simple games like pong or chess, thus there are no need to implement more advanced features in the system. It is important to note that these limitations only apply for the prototype of the project, and that further developments to GGS could be to implement these features. \end_layout \begin_layout Section Method \end_layout \begin_layout Subsection Development process \end_layout \begin_layout Standard May be Extreme Programming(XP), need to check this out further. Maybe adapt so we can say that we use a standardized software development method. \end_layout \begin_layout Subsubsection Demand specification \end_layout \begin_layout Subsection Design \end_layout \begin_layout Subsection Testing and evaluation \end_layout \begin_layout Standard Can we use quickcheck? \end_layout \begin_layout Chapter Theory \end_layout \begin_layout Section Performance \end_layout \begin_layout Standard How many players can we have on a server? Performance differences between games? e.g can one game have thousands players on a server and another only have hundreds? Questions to be discussed here. \begin_inset Note Note status open \begin_layout Plain Layout Create a game with several thousand players, see how our server scales, how can we improve the performance? Sharding isn’t very nice.. alternatives? Improve the speed of sharding? \end_layout \end_inset \end_layout \begin_layout Section Choice of network protocol \end_layout \begin_layout Standard There are three main ways in which computer communication over the Internet usually takes place; TCP, UDP and HTTP. The first two are transport layer protocols, which are commonly used to transport application layer protocols, such as HTTP. TCP and UDP can not be used on their own, without an application layer protocol on top. Application layer protocols such as HTTP on the other hand needs a transport layer protocol in order to work. \end_layout \begin_layout Subsection HTTP \end_layout \begin_layout Standard Since HTTP is so widely used on the Internet today in web servers, it is available on most Internet connected devices. This means that if HTTP is used in GGS, firewalls will not pose problems, which is a great benefit. However, due to the intended usage of HTTP in web servers, the protocol was designed to be stateless and client-initiated. In order to maintain a state during a game session using HTTP, some sort of token would have to be passed between client and server at all times, much like how a web server works. These facts combined makes HTTP unsuitable for our purposes, since GGS requires a state to be maintained throughout a session, and also needs to push data from the server to clients without the clients requesting data. It should also be mentioned that HTTP uses the TCP protocol for transport, and what is said about TCP also applies to HTTP. \end_layout \begin_layout Subsection UDP \end_layout \begin_layout Standard Many online games use UDP as the carrier for their application layer protocol. UDP moves data across a network very quickly, however it does not ensure that the data transferred arrives in consistent manner. Data sent via UDP may be repeated, lost or out of order. To ensure the data transferred is in good shape, some sort of error checking mechanisms must be implemented. UDP is a good choice for applications where it is more important that data arrives in a timely manner than that all data arrives undamaged, it is thus very suitable for media streaming, for example. In GGS reliability of transfer was chosen before the speed of the transfer, ruling out UDP as the transport later protocol. \end_layout \begin_layout Subsection TCP \end_layout \begin_layout Standard For reliable transfers, TCP is often used on the Internet. Built in to the protocol are the error checking and correction mechanisms missing in UDP. This ensures the consistency of data, but also makes the transfer slower than if UDP had been used. In GGS, data consistency is more important than transfer speeds, and thus TCP is a better alternative than UDP. \end_layout \begin_layout Section Fault Tolerant \end_layout \begin_layout Subsubsection Performance penalties \end_layout \begin_layout Section Availability \end_layout \begin_layout Standard One important factor of a server is the availability, a server that you can not connect to is a bad server. Erlang has several features to increase the availability, for example hot code replacement. It is also critical to have a good design, we want to separate each part of the server and thus avoiding that the whole server will crash. \end_layout \begin_layout Standard \begin_inset Note Note status open \begin_layout Plain Layout Players are unsatisfied with the service of WoW Telecoms have the same problem of having to migrate users from one node to another, this is called handover \end_layout \end_inset \end_layout \begin_layout Section Scalability \end_layout \begin_layout Standard Each instance of GGS contains several tables. Each table is an isolated instance of a game, for example a chess game or a poker game. The way that GGS scales is to distribute these tables on different servers. In many games it is not necessary for a player to move between tables during games. This is for example not a common occurrence in chess, where it would be represented as a player standing up from her current table and sitting down at a new table, all within the same game session. With this in mind, the main focus of GGS is not to move players between tables, but to keep a player in a table, and to start new tables instead. When a server has reached a certain amount of players the performance will start to decrease. To avoid this GGS will start new tables on another server, using this technique the players will be close to evenly distributed between the servers. It is important to investigate and find out how many players that are optimal for each server. This approach makes it possible to utilize all resources with moderate load, instead of having some resources with heavy load and some with almost no load. \end_layout \begin_layout Standard As mentioned in the purpose section there are two different types of scalability , structural scalability and load scalability. To make GGS scalable both types of scalability are needed. Structural scalability means in our case that it should be possible to add more servers to an existing cluster of servers. By adding more servers the limits of how many users a system can have is increased. Load scalability in contrast to structural scalability is not about how to increase the actual limits of the system. Instead it means how good the system handles increased load. GGS should be able to scale well in both categories. \end_layout \begin_layout Standard \begin_inset Note Note status open \begin_layout Plain Layout Because P2P game architectures are a constant goal for cheaters and because “Cheating is a major concern in network games as it degrades the experience of the majority of players who are honest” and preventing cheating in P2P game architectures is very difficult game developers try to use Client - Server architectures which have a natural problem to scale. In this paper we want to show some strategies to achieve scalability. \end_layout \end_inset \end_layout \begin_layout Subsubsection UUID \end_layout \begin_layout Section Security \end_layout \begin_layout Standard We only support languages running in a sandboxed environment. Each game session is started in its own sandbox. The sandboxing isolates the games in such a way that they can not interfere with each other. If sandboxing was not in place, one game could potentially modify the contents of a different game. A similar approach is taken with the persistent storage we provide. In the storage each game has its own namespace, much like a table in a relational database. A game is not allowed to venture outside this namespace, and can because of this not modify the persistent data of other games. \end_layout \begin_layout Subsection Encryption \end_layout \begin_layout Chapter Overview \end_layout \begin_layout Section Techniques for ensuring reliability \end_layout \begin_layout Standard One of the main goals of the project is to achieve high reliability. A highly reliable application is one crashes very, very rarely \begin_inset Note Note status open \begin_layout Plain Layout CITATION NEEDED \end_layout \end_inset . There are some tools for creating reliable applications built in to Erlang. \end_layout \begin_layout Itemize Links between processes. When a process spawns a new child process, and the child process later exits, the parent process is notified of the exit. \end_layout \begin_layout Itemize Transparent distribution over a network of processors. When several nodes participate in a network, it does not matter on which of these machines a process is run. Communication between processes does not depend on the node in which each process is run. \end_layout \begin_layout Itemize Hot code replacements. Two versions of the same module can reside in the memory of Erlang at any time. This means that a simple swap between these versions can take place very quickly, and without stopping the machine. \end_layout \begin_layout Standard These three features are some of the basic building blocks for more sophisticate d reliability systems in Erlang. Many times it is not necessary to use these features directly, but rather through the design patterns described below. \end_layout \begin_layout Subsection Supervisor structure \end_layout \begin_layout Standard By linking processes together and notifying parents when children exit, we can create supervisors. A supervisor is a common approach in ensuring that an application functions in the way it was intended. When a process misbehaves, the supervisor takes some action to restore the process to a functional state. \end_layout \begin_layout Standard There are several approaches to supervisor design in general (when not just considering how they work in Erlang). One common approach is to have the supervisor look in to the state of the process(es) it supervises, and let the supervisor make decisions based on this state. The supervisor has a specification of how the process it supervises should function, and this is how it makes decisions. \end_layout \begin_layout Standard In Erlang, we have a simple version of supervisors. We do not inspect the state of the processes being supervised. We do have a specification of how the supervised processes should behave, but on a higher level. The specification describes things such as how many times in a given time interval a child process may crash, which processes need restarting when crashes occur, and so forth. \end_layout \begin_layout Standard When the linking of processes in order to monitor exit behaviour is coupled with the transparent distribution of Erlang, a very powerful supervision system is created. For instance, we can restart a failing process on a different, new node, with minimal impact on the system as a whole. \end_layout \begin_layout Standard In GGS, we have separated the system in to two large supervised parts. We try to restart a crashing child separately, if this fails too many times, we restart the nearest supervisor of this child. This ensures separation of the subsystems so that a crash is as isolated as possible. \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \begin_inset Note Note status open \begin_layout Plain Layout We should really do this graphic in EPS instead of PNG \end_layout \end_inset \end_layout \begin_layout Plain Layout \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{centering} \end_layout \end_inset \end_layout \begin_layout Plain Layout \begin_inset Graphics filename graphics/Supervisor_tree_GGS.eps scale 40 \end_inset \end_layout \begin_layout Plain Layout \begin_inset ERT status open \begin_layout Plain Layout \backslash end{centering} \end_layout \end_inset \end_layout \begin_layout Plain Layout \begin_inset Caption \begin_layout Plain Layout The supervisor structure of GGS \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Standard The graphic above shows our two subsystems, the coordinator subsystem and the dispatcher subsystem. Since these two systems perform very different tasks they have been separated. Each subsystem has one worker process, the coordinator or the dispatcher. The worker process keeps a state which should not be lost upon a crash. \end_layout \begin_layout Standard We have chosen to let faulty processes crash very easily when they receive bad data, or something unexpected happens. The alternative to crashing would have been to try and fix this faulty data, or to foresee the unexpected events. We chose not to do this because it is so simple to monitor and restart processes, and so difficult to try and mend broken states. This approach is something widely deployed in the Erlang world, and developers are often encouraged to “Let it crash”. \end_layout \begin_layout Standard To prevent any data loss, the good state of the worker processes is stored in their respective backup processes. When a worker process (re)starts, it asks the backup process for any previous state, if there is any that state is loaded in to the worker and it proceeds where it left off. If on the other hand no state is available, a special message is delivered instead, making the worker create a new state, this is what happens when the workers are first created. \end_layout \begin_layout Subsection Hot code replacement \end_layout \begin_layout Section Implementation \end_layout \begin_layout Subsubsection User interface \end_layout \begin_layout Chapter Problems \end_layout \begin_layout Section Erlang JS \end_layout \begin_layout Standard To be able to run JavaScript on our server we needed to embed a JavaScript engine within the server. After a thorough investigation erlang_js became our choice. erlang_js provides direct communication with a JavaScript VM (Virtual Machine). This was exactly what we wanted, but we also needed the possibility to communicate from erlang_js to Erlang. This functionality was not yet implemented in erlang_js, due to lack of time. \end_layout \begin_layout Standard There were two possible solutions to the problem. We could rewrite some part of erlang_js, or we could switch erlang_js for some other JavaScript engine. Searching for other engines we found erlv8 and beam.js which provided the functionality that we wanted. As we tested beam.js it occurred random crashes of the whole Erlang environment. These crashes were related to the use of erlv8 in beam.js and we decided that the use of erlv8 was not an alternative due to the stability issues. \end_layout \begin_layout Standard To get the functionality needed we decided to implement this in erlang_js. \end_layout \begin_layout Subsection UUID \end_layout \begin_layout Standard Erlang identifies processes uniquely throughout the entire Erlang network using process IDs (PID). When we wish to refer to erlang processes from outside our erlang system, for example in a virtual machine for a different language, possibly on a different machine, these PIDs are no longer useful. \end_layout \begin_layout Standard This problem is not new, and a common solution is to use a Universally Unique Identifier, a UUID. These identifiers are generated both using randomization and using time. A reasonably large number of UUIDs can be generated before a collision should occur. There are standard tools in many UNIX systems to generate UUIDs, we chose to use the uuidgen command, which employs an equidistributed combined Tausworth e generator. \end_layout \begin_layout Section Design choices \end_layout \begin_layout Standard When designing concurrent applications, it is useful to picture them as real world scenarios, and to model each actor# as a real world process. A real world process is a process which performs some action in the real world, such as a mailbox receiving a letter, a door being opened, a person translating a text, a soccer player kicking the ball, just to name a few examples. Since we focus on games in this project, it is suitable to model our system as a place where games take place. We imagined a chess club. \end_layout \begin_layout Standard The clients pictured as green circles can be thought of as the physical chess players. \end_layout \begin_layout Standard When a player wants to enter the our particular chess club, he must first be let in by the doorman, called the Dispatcher in GGS. \end_layout \begin_layout Standard He then gets a name badge, and thus becomes a Player process in the system. He is also guided in to the lobby by the Coordinator, which has the role of the host of the chess club. \end_layout \begin_layout Standard When players wish to play against each other, they talk to the Coordinator who pairs them up, and places them at a table. Once they have sat down at the table, they no longer need the assistance of the Coordinator, all further communication takes place via the table. This can be thought of as the actual chess game commencing. \end_layout \begin_layout Standard All the moves made in the game are recorded by the table, such that the table can restore the game in case something would happen, such as the table tipping over, which would represent the table process crashing. \end_layout \begin_layout Standard Once a player wishes to leave a game, or the entire facility, he should contact the Coordinator, who revokes his name badge and the Dispatcher will let the player out. \end_layout \begin_layout Standard With the information kept in the tables and the Coordinator combined, we can rebuild the entire state of the server at a different location. This can be thought of the chess club catching fire, and the Coordinator rounding up all the tables, running to a new location and building the club up in the exact state it was prior to the fire. \begin_inset Float figure wide false sideways false status open \begin_layout Plain Layout \begin_inset ERT status open \begin_layout Plain Layout \backslash begin{centering} \end_layout \end_inset \end_layout \begin_layout Plain Layout \begin_inset Graphics filename graphics/Chess_no_text.eps width 100text% \end_inset \end_layout \begin_layout Plain Layout \begin_inset ERT status open \begin_layout Plain Layout \backslash end{centering} \end_layout \end_inset \end_layout \begin_layout Plain Layout \begin_inset Caption \begin_layout Plain Layout The layout of GGS \end_layout \end_inset \end_layout \end_inset \end_layout \begin_layout Section Understanding OTP \end_layout \begin_layout Section Usability \end_layout \begin_layout Chapter Results and discussion \end_layout \begin_layout Section Software development methodology \end_layout \begin_layout Section Statistics \end_layout \begin_layout Chapter Conclusion \end_layout \begin_layout Standard \begin_inset CommandInset bibtex LatexCommand bibtex bibfiles "bibliography" options "plainnat" \end_inset \end_layout \end_body \end_document