Author: Frans Lundberg, www.linova.com, 8 June 2004

LinoGrid Manual

SUMMARY: LinoGrid is a simple Java-based computational grid focused on using normally vasted CPU-cycles of ordinary office computers for distributed computations. This manual describes how to use and develop the framework. An updated version of this manual may be avalable at www.linogrid.com.

Contents

In this document

Links

Introduction

The Idea

Most desktop machines are busy less than 5 per cent of the time ("Fundamentals of Grid Computing" by Viktor Berstis, ibm.com/redbooks). Linova believes this is an enormous vaste and therefore created LinoGrid. The purpose of LinoGrid is to harness this otherwise vasted computational power to use it for real computations instead of for running a screen saver.

The heart of LinoGrid is the LinoGrid Engine. It lets other people run tasks on your desktop machine when you are not using it. These tasks will never cause you any trouble because they run in a restricted environment (sandbox) that does not allow them access to the filesystem or the network. The tasks will not disturb normal programs since they run at a low priority and with a limited amount of memory allocated to it.

By installing LinoGrid Engines on computers in an intranet we create a LinoGrid that makes it possible to use the idle time of the computers for real computations.

What is LinoGrid?

A LinoGrid is a simple computational grid consisting of a set of LinoGrid Engines running on computers connected with a standard computer network.

Engines are typically installed on the computers in an intranet. The engines are simple to install and runs on nearly any operating system (Java 1.3 required). All intranet users have access to all Engines. This is possible with maintained security due to the the restricted sandbox that the tasks run in. This design is extremely simple, there is no need for user accounts, certificates, encryption keys, authentication, or authorization mechanisms. After installation, there is zero administration. To upgrade the LinoGrid simply install engines on more or faster computers.

The LinoGrid Engine with supporting software is free and open source. Note however that "LinoGrid" is a trademark belonging to Linova.

It is the responsibility of the grid user to locate suitable engines and to distribute the tasks to the engines. The LinoGrid may however include an Engine Supervisor to help users find suitable engines. The Engine Supervisor regularly scans the intranet to locate Engines and to keep statistics of how much the engines are used and how fast they are.

Other optional components in a LinoGrid are adapters. The Engines themselves are not interoperable with other types of grids, but with adapters they can interface to other systems, such as those based on the Globus toolkit version 3.0 (www.globus.org) which is compliant with OGSI 1.0 (www.gridforum.org).

This Manual

This manual describes how to use and develop LinoGrid. The document is divided in sections for different types of readers.

  1. Engine Runner, someone who runs LinoGrid Engine
  2. LinoGrid User, or just User, someone who uses LinoGrid for computations
  3. LinoGrid Developer, someone who modifies the LinoGrid Engine or supporting software.

The Engine Runner only needs to read the introduction and the section "Running an Engine".

The User should read all sections exception for the "For the LinoGrid Developer" and the LinoGrid Developer should read everything.

Running the Engine

System Requirements

A 400 MHz Pentium processor or better and 128 MB (256 MB recommended) of RAM memory is needed to run a LinoGrid engine. Java Runtime (Java Virtual Machine) 1.3 or later needs to be installed. See www.java.com or www.java.sun.com for how to download and install the Java Virtual Machine from Sun Microsystems. It is free.

Distributions

The LinoGrid software comes in two distributions: linogrid_engine.zip and linogrid_dev.zip. Possibly a version number is appended to the file names. The files can be downloaded from Linova.

linogrid_engine.zip

This distribution is for someone who wants to install and run LinoGrid Engine. All files and documentation for this purpose is included, just download the file and unzip it. Read index.html first.

The zip archive contains an executable jar file with the engine application. This file is called "linogrid_engine.jar". See below for how to use it.

The distribution contains the file linogrid_engine_install_win.jar which is also an executable jar file and is used to install the engine as a Windows service. This is the recommended way of running the engine on a Windows system. Start the install program by double-clicking on the file or by typing

    java -jar linogrid_engine_install_win.jar

on your command prompt. Then follow the instructions.

linogrid_dev.zip

This distribution contains java code and an ANT build script (build.xml) that is used to build everything. The distribution is both for someone who wants to write programs to run on a LinoGrid (a LinoGrid User), and for someone who wants to enhance or modify the LinoGrid software itself (the LinoGrid Developer).

After unzipping the archive first read readme.txt.

Starting the Engine

Simple Start

The engine application is packed as an executable Java jar file called linogrid_engine.jar, or linogrid_engineX.Y.jar where X and Y are the engine version and subversion. It is often possible to double-click on the file to start it. Otherwise it can be started by typing

    java -jar linogrid_engine.jar

in the command prompt of your system. This example assumes the jar file is located in current directory, otherwise type the full filename of the jar file.

Parameters

The above way of starting the LinoGrid engine creates a graphical window. To avoid this start the application with at least one parameter. For example

    java -jar linogrid_engine.jar normal

For a full description of the engine parameters, type

    java -jar linogrid_engine.jar help

Limited Memory

The Engine program runs in a Java virtual machine (JVM). For Suns JVM the maximum amount of memory used by JVM can be set with the -Xmx flag. For example:

    java -Xmx256m -jar linogrid_engine.jar normal

runs the engine with a maximum of 256 MB of RAM allocated to it.

Forbid Graphics

Tasks that generate graphical output is normally not allowed by the engine itself, but to be 100 per cent sure the engine will not draw anything on the screen it recommended that the java.awt.headless property is set. This works with JVM 1.4 from Sun. Start the engine like this:

    java -Djava.awt.headless=true -jar linogrid_engine.jar normal

To allow tasks to load GUI classes use the "-allowGraphics" flag as a startup parameter.

Allow Localhost

Tasks from users that run on the same computer as the engine, localhost tasks may be granted all permissions by the "-allowLocalhost" flag. Note this feature is currently not well-tested and was introduced for LinoGrid 2.2.

Windows

Windows Service

The recommended way of running the engine in Windows XP or Windows 2000 is to install the engine as a Windows service. This can be accomplished by running the linogrid_engine_install_win.jar installation program, just double-click on the file and follow the instructions.

If the engine is installed as a service it will run in the background even if no one is logged in.

Low Priority

The engine should be started at a low process priority, this can be accomplished with the Windows "start" command, like this.

    start /LOW java -jar linogrid_engine.jar normal

All together a command line to start the engine could look like this.

    start /LOW java -Djava.awt.headless=true -Xmx256m -jar linogrid_engine.jar normal

MSI-package

It is possible to create an installation file (*.msi file) that lets an administrator of a Windows domain install engines on a whole set of computers in a Windows domain. Contact Linova if this functionality is needed.

Unix, Linux

To set a low priority of the engine process the "nice" command can be used. To run the engine as a "service" in the background simply add the engine startup command to a suitable script that runs when the computer is started. A suitable command line could look like this.

    nice -n 15 java -Djava.awt.headless=true -Xmx256m -jar linogrid_engine.jar silent &

This run the engine on a low priority.

Engine Description

This section describes the LinoGrid engine.

The engine is a single-process Java application. Its input/output is solely through TCP network connections. The application works as a server waiting for clients (users) to connect to a standard TCP port (port 12300, 17 May 2004). A new thread is created for each user that connects.

The User may use the linogrid.user.User class to communicate with the engine. See the Javadoc documentation for details. The engine communication protocol is based on BaseStream, see the internet-draft draft-flundberg-basestream-03.txt (or later) for details on BaseStream. For details on the engine protocol see the linogrid.engine package documentation.

Tasks

The purpose of the engine is to run tasks for the user. A LinoGrid task is simply a Java class that implements Runnable and Serializable. No special API classes are needed to define tasks. Here is an example of a class that computes the average of 1 miljon random numbers.

public class RandomAverage implements Runnable, java.io.Serializable {
    public double average = 0.0;
    public void run() {
        int N = 1000000;
        for(int i=0; i<N; i++) {
            average += Math.random();
        }
        average = average / N;
    }
}

The package linogrid.user contains a number of useful utility tasks.

Sessions

A user that has connected to an engine can start a session. A session belongs to a single user. The engine can run several sessions simultaneouosly.

Within a session a user runs tasks. The user can run several parallel tasks in the same session or choose to run tasks in serial order.

Session variables can be used to store information between different tasks in the same session. Tasks are able to access only session variables in the session the task executes in.

To add/delete session variables and to handle the tasks of the session the Engine Interface (the linogrid.ei package) can be used.

The task classes do not have to be known by the engine in advance, the needed classes will automatically be loaded from the user by the session class loader. Also class and resource files, for example a jar file, can be added to the session class repository to avoid frequent single-class loading over the network.

Engine Properties

A connected user not running a session can querry the engine for properties with the "getProperty" command. The following properties are supported (18 May 2004).

 * 
 * A DESCRIPTION OF ENGINE PROPERTIES
 * 
 * version - the version of the engine, for example "1.2"
 *     the string is formatted like X.Y where X and Y are 
 *     (possibly negative) integers.
 * 
 * java.version - the string returned from System.getProperty("java.version")
 * 
 * sessionCount - the current number of running sessions
 *
 * maxSessionCount - the maximum number of running session allowed by this 
 *     engine
 *
 * totalMemory - the amount of memory used by the JVM as returned by 
 *     java.lang.Runtime.totalMemory
 *
 * maxMemory - the maximum memory that this JVM can use as returned by
 *     the method java.lang.Runtime.maxMemory if that method exists 
 *     (java1.4 or later), if the method does not exist (Java1.3) -1 is 
 *     returned to indicate an unknown max memory amount
 *
 * allowGraphics - returns either "true" or "false", true if the engine
 *     was started in a a mode that allows graphical classes (eg. JFrame)
 *     to be loaded by the session class loader
 *
 * debug - returns a debug string describing the state of the engine
 *     Note, this information is only available for a user on the same 
 *     computer as the engine (the user must be "localhost").
 *     If another user tries to get this property the empty string ("") is
 *     returned.
 *

See the Javadoc documentation for the linogrid.engine.Server class for updated information.

Using LinoGrid

This section describes how to use a LinoGrid for computations. Also study the source code for the class linogrid.user.Examples. It contains a number of examples and is a good starting point to learn how to create and run tasks.

Connecting to an Engine

The User class in package linogrid.User can be used to connect to an engine. The following code lines will connect to an engine on localhost, print the version of it and close the connection.

User u = new User("localhost");
u.open();
System.out.println(u.getProperty("version"));
u.close();

Running a Task

The lines below connects to an engine, starts a session, runs the RandomAverage (see above) task in it, stops the session and closes the connection with the engine.

RandomAverage task = new RandomAverage();
User u = new User("localhost");
u.open();
u.startSession("java1");
u.runSerialTask(task);
task = (RandomAverage) u.getResult();
System.out.println("The average is: " + task.average);
u.stopSession();
u.close();

The above example is a simple example that runs one serial task. More advanced computations may need to be able to run tasks in parallel. See the User.runTask method for more information.

Using Session Variables

See the source code of the Get and Set classes in linogrid.user package for an example on how to use session variables.

Compatibility for Tasks

The tasks we create should be possible to run in future versions of the engine, of course. Simple tasks that do not use the Engine Interface (linogrid.ei) will never be a problem. Tasks should be able to use the Engine Interface without the need to worry about whether task will be possible to run in new versions of the engine. The Engine Interface is intended to be very stable. If new methods are introduced that are not compatible with the old interface a new package (linogrid.ei2) will be created, but the old one will be kept for backward compatibility.

The tasks should use the static public method

    linogrid.ei.Access.getSessionInfo()

to get access to session information.

Note that the engine is has no knowledge of the classes in the linogrid.user package. This means that as long as the network interface to the engine is intact there is no problem to modify the engine and the linogrid.user classes independently.

For the LinoGrid Developer

This section contains information for the LinoGrid Developer, that is, someone who modifies or extends the LinoGrid software itself, such as the LinoGrid engine or supporting software.

Goals of the Engine Application

Releases

How to Create a New Engine Release

* Change the version in linolab/engine/data/version.txt (only major.minor version are used)
* compile, create jar, and test (use ANT build.xml script)
* When testing engine run it with a Java1.3 VM.
* Compile engine classes with Java1.3 compiler.
* Create signiture of new jar file with Signer program (FL)
* Upload to www.linova.com/linogrid_engine_download/ and www.linova.com/linogrid/
* Update the version.txt file on www.linova.com.
* Update documentation if needed

Release History

Conventions

The LinoGrid developer should follow Sun's recommendation: "Code Conventions for the Java Programming Language" loosly. Absolute requirements: 4-space indentation, javadoc comments for public class members and the class itself. Write in English. See http://java.sun.com/docs/codeconv/html/CodeConvTOC.doc.html.

ANT build script are used to build all software and distribution files.

Update Mechanism

This section describes how the Engine can securily update itself. The update mechanism consists of the following parts.

This works fine, but the problem is when a proxy server is used to be able to access the external internet. Some useful info is available at: http://www.javaworld.com/javaworld/javatips/jw-javatip42.html.

See http://forum.java.sun.com/thread.jsp?forum=30&thread=364342 for automatic proxy detection. Also check out the class linogrid.frans.ProxyTester. We will wait and see if newer version of Java will include automatic proxy detection.

Security

Security for engine computer

The engine computer cannot be attacked by the task run in the engine. This is mainly due to the Java sandbox. The following security features are considered.

Security for user

All code sent between the user and the engine is in compiled form (java byte code) so the program itself is protected as much as when deployed as a jar file. An obfuscator is recommended to make if difficult to reverse-engineer the byte code.

The SessionLoader class loader cannot load resources from the user. It would be a nice feature if this could be allowed. However, often Java source code is in the classpath directories and the source code could then be loaded to the engine computer if a malicious engine is installed.

Currently the communication between the user and the engine is not encrypted. This could of course be accomplished (using SSL), but it do require some extra CPU cycles. When the data has arrived at the engine, there is of course no way to protect the data from being read by someone who has control over the engine computer. The computations could however be made in such a way that a single engine will only see a fraction of the data.

Note that in a grid computing framework, there could be modified versions of the engines. An engine could be modified to return incorrect results for computations. There is not much that can be done about this accept that identical tasks may be sent to several engines to check whether or not the results are identical. This approach is taken by ceti@home. LinoGrid is mainly targetted at intranets where this is unlikely to be a problem.

Security Implementation

The linogrid.engine.Security class is responsible for setting the permissions for the tasks. The permissions granted are none except for read access to the "java.version" system property, and possibly read access to a few other such properties. All other permissions are denied. See guide/security/permissions.html in J2SDK documentation for a list of these permissions. The Security class creates an AccessControlContext suitable for using when running the tasks.

The linogrid.engine.TaskRunner class is responsible for actually running the tasks. The tasks can be stopped and they are forced to run at low thread priority. The class is loaded by the SessionLoader and not the default system loader. This is so that the classes loaded is loaded with SessionLoader. SessionLoader must be able to load dependent classes from the user. TaskRunner implements TaskRunnerInterface. The class Session1 loads the TaskRunner dynamically and accesses its methods through the TaskRunnerInterface. Session1 is loaded with the system class loader and can therefore not contain any static references to the TaskRunner class since it must be loaded by the SessionLoader.

Possible Enhancements and Changes