A Swift script describes data, application components, invocations of applications components, and the inter-relations (data flow) between those invocations.

Data is represented in a script by strongly-typed single-assignment variables. The syntax superficially resembles C and Java. For example, { and } characters are used to enclose blocks of statements.

Types in Swift can be atomic or composite. An atomic type can be either a primitive type or a mapped type. Swift provides a fixed set of primitive types, such as integer and string. A mapped type indicates that the actual data does not reside in CPU addressable memory (as it would in conventional programming languages), but in POSIX-like files. Composite types are further subdivided into structures and arrays. Structures are similar in most respects to structure types in other languages. Arrays use numeric indices, but are sparse. They can contain elements of any type, including other array types, but all elements in an array must be of the same type. We often refer to instances of composites of mapped types as datasets.

Mapped type and composite type variable declarations can be annotated with a mapping descriptor indicating the file(s) that make up that dataset. For example, the following line declares a variable named photo with type image. It additionally declares that the data for this variable is stored in a single file named shane.jpeg.

  image photo <"shane.jpeg">;

Component programs of scripts are declared in an app declaration, with the description of the command line syntax for that program and a list of input and output data. An app block describes a functional/dataflow style interface to imperative components.

For example, the following example lists a procedure which makes use of the ImageMagick convert command to rotate a supplied image by a specified angle:

  app (image output) rotate(image input) {
    convert "-rotate" angle @input @output;
  }

A procedure is invoked using the familiar syntax:

  rotated = rotate(photo, 180);

While this looks like an assignment, the actual unix level execution consists of invoking the command line specified in the app declaration, with variables on the left of the assignment bound to the output parameters, and variables to the right of the procedure invocation passed as inputs.

The examples above have used the type image without any definition of that type. We can declare it as a marker type which has no structure exposed to SwiftScript:

  type image;

This does not indicate that the data is unstructured; but it indicates that the structure of the data is not exposed to SwiftScript. Instead, SwiftScript will treat variables of this type as individual opaque files.

With mechanisms to declare types, map variables to data files, and declare and invoke procedures, we can build a complete (albeit simple) script:

 type image;
 image photo <"shane.jpeg">;
 image rotated <"rotated.jpeg">;

 app (image output) rotate(image input, int angle) {
    convert "-rotate" angle @input @output;
 }

 rotated = rotate(photo, 180);

This script can be invoked from the command line:

  $ ls *.jpeg
  shane.jpeg
  $ swift example.swift
  ...
  $ ls *.jpeg
  shane.jpeg rotated.jpeg

This executes a single convert command, hiding from the user features such as remote multisite execution and fault tolerance that will be discussed in a later section.

Figure 1. shane.jpeg

Figure 2. rotated.jpeg

Arrays of values can be declared using the [] suffix. An array be mapped to a collection of files, one element per file, by using a different form of mapping expression. For example, the filesys_mapper maps all files matching a particular unix glob pattern into an array:

  file frames[] <filesys_mapper; pattern="*.jpeg">;

The foreach construct can be used to apply the same block of code to each element of an array:

   foreach f,ix in frames {
     output[ix] = rotate(frames, 180);
   }

Sequential iteration can be expressed using the iterate construct:

   step[0] = initialCondition();
   iterate ix {
     step[ix] = simulate(step[ix-1]);
   }

This fragment will initialise the 0-th element of the step array to some initial condition, and then repeatedly run the simulate procedure, using each execution's outputs as input to the next step.

Non-array variables are single-assignment, which means that they must be assigned to exactly one value during execution. A procedure or expression will be executed when all of its input parameters have been assigned values. As a result of such execution, more variables may become assigned, possibly allowing further parts of the script to execute.

In this way, scripts are implicitly parallel. Aside from serialisation implied by these dataflow dependencies, execution of component programs can proceed in parallel.

In this fragment, execution of procedures p and q can happen in parallel:

  y=p(x);
  z=q(x);

while in this fragment, execution is serialised by the variable y, with procedure p executing before q.

 y=p(x);
 z=q(y);

Arrays in SwiftScript are more monotonic - a generalisation of being assignment. Knowledge about the content of an array increases during execution, but cannot otherwise change. Each element of the array is itself single assignment or monotonic (depending on its type). During a run all values for an array are eventually known, and that array is regarded as closed.

Statements which deal with the array as a whole will often wait for the array to be closed before executing (thus, a closed array is the equivalent of a non-array type being assigned). However, a foreach statement will apply its body to elements of an array as they become known. It will not wait until the array is closed.

Consider this script:

 file a[];
 file b[];
 foreach v,i in a {
   b[i] = p(v);
 }
 a[0] = r();
 a[1] = s();

Initially, the foreach statement will have nothing to execute, as the array a has not been assigned any values. The procedures r and s will execute. As soon as either of them is finished, the corresponding invocation of procedure p will occur. After both r and s have completed, the array a will be closed since no other statements in the script make an assignment to a.

As with many other programming languages, procedures consisting of SwiftScript code can be defined. These differ from the previously mentioned procedures declared with the app keyword, as they invoke other SwiftScript procedures rather than a component program.

 (file output) process (file input) {
   file intermediate;
   intermediate = first(input);
   output = second(intermediate);
 }

 file x <"x.txt">;
 file y <"y.txt">;
 y = process(x);

This will invoke two procedures, with an intermediate data file named anonymously connecting the first and second procedures.

Ordering of execution is generally determined by execution of app procedures, not by any containing compound procedures. In this code block:

 (file a, file b) A() {
   a = A1();
   b = A2();
 }
 file x, y, s, t;
 (x,y) = A();
 s = S(x);
 t = S(y);

then a valid execution order is: A1 S(x) A2 S(y). The compound procedure A does not have to have fully completed for its return values to be used by subsequent statements.

Each variable and procedure parameter in SwiftScript is strongly typed. Types are used to structure data, to aid in debugging and checking program correctness and to influence how Swift interacts with data.

The image type declared in previous examples is a marker type. Marker types indicate that data for a variable is stored in a single file with no further structure exposed at the SwiftScript level.

Arrays have been mentioned above, in the arrays section. A code block may be applied to each element of an array using foreach; or individual elements may be references using [] notation.

There are a number of primitive types:

Complex types may be defined using the type keyword:

  type headerfile;
  type voxelfile;
  type volume {
    headerfile h;
    voxelfile v;
  }

Members of a complex type can be accessed using the . operator:

  volume brain;
  o = p(brain.h);

Sometimes data may be stored in a form that does not fit with Swift's file-and-site model; for example, data might be stored in an RDBMS on some database server. In that case, a variable can be declared to have external type. This indicates that Swift should use the variable to determine execution dependency, but should not attempt other data management; for example, it will not perform any form of data stage-in or stage-out it will not manage local data caches on sites; and it will not enforce component program atomicity on data output. This can add substantial responsibility to component programs, in exchange for allowing arbitrary data storage and access methods to be plugged in to scripts.

  type file;

  app (external o) populateDatabase() {
    populationProgram;
  }

  app (file o) analyseDatabase(external i) {
    analysisProgram @o;
  }

  external database;
  file result <"results.txt">;

  database = populateDatabase();
  result = analyseDatabase(database);

Some external database is represented by the database variable. The populateDatabase procedure populates the database with some data, and the analyseDatabase procedure performs some subsequent analysis on that database. The declaration of database contains no mapping; and the procedures which use database do not reference them in any way; the description of database is entirely outside of the script. The single assignment and execution ordering rules will still apply though; populateDatabase will always be run before analyseDatabase.

Data processed by Swift is strongly typed. It may be take the form of values in memory or as out-of-core files on disk. Language constructs called mappers specify how each piece of data is stored.

imagefile f1<single_file_mapper;file="/home/yongzh/data/file1.bin">

binaryfile f1<"/home/yongzh/data/file1.bin">

The syntax of SwiftScript has a superficial resemblance to C and Java. For example, { and } characters are used to enclose blocks of statements.

A SwiftScript program consists of a number of statements. Statements may declare types, procedures and variables, assign values to variables, and express operations over arrays.

  typename variablename (<mapping> | = initialValue ) ;

  variable = value;

(type3 out1, type4 out2) myproc (type1 in1, type2 in2)

(binaryfile bf) myproc1 (int i, string s="foo")

binaryfile mybf = myproc1(1);

binaryfile mybf = myproc1 (1, s="bar");

app (binaryfile bf) myproc (int i, string s="foo") {
	myapp i s @filename(bf);
}			

(type2 b) foo_bar (type1 a) {
	type3 c;
	c = foo(a);    // c holds the result of foo
	b = bar(c);    // c is an input to bar
}

check_order (file a[]) {
	foreach f in a {
		compute(f);
	}
}

foreach controlvariable (,index) in expression {
    statements
}

if(predicate) {
    statements
} else {
    statements
}

switch(controlExpression) {
    case n1:
        statements2
    case n2:
        statements2
    [...]
    default:
        statements
}

iterate var {
	statements;
} until (terminationExpression);

The following infix operators are available for use in SwiftScript expressions.

At the top level of a SwiftScript program, the global modified may be added to a declaration so that it is visible throughout the program, rather than only at the top level of the program. This allows global constants (of any type) to be defined. (since Swift 0.10)

The import directive can be used to import definitions from another SwiftScript file. (since Swift 0.10)

For example, a SwiftScript program might contain this:

import defs;
file f;
			

which would import the content of defs.swift in the current directory:

type file;
			

Imported files are read from the current working directory.

There is no requirement that a module is imported only once. If a module is imported multiple times, for example in different files, then Swift will only process the imports once.

Imports may contain anything that is valid in a SwiftScript program, including code that causes remote execution.

The single_file_mapper maps a single physical file to a dataset.

    Swift variable ------------------->  Filename

       f                                 myfile

       f[0]                              INVALID

       f.bar                             INVALID

		

Example:

	file f <single_file_mapper;file="plot_outfile_param">;

There is a simplified syntax for this mapper:

	file f <"plot_outfile_param">;

The simple_mapper maps a file or a list of files into an array by prefix, suffix, and pattern. If more than one file is matched, each of the file names will be mapped as a subelement of the dataset.

Examples:

	type file;
	file f <simple_mapper;prefix="foo", suffix=".txt">;
			

The above maps all filenames that start with foo and have an extension .txt into file f.

    Swift variable ------------------->  Filename

       f                                 foo.txt

		

type messagefile;

(messagefile t) greeting(string m) {.
    app {
        echo m stdout=@filename(t);
    }
}

messagefile outfile <simple_mapper;prefix="foo",suffix=".txt">;

outfile = greeting("hi");
	

This will output the string 'hi' to the file foo.txt.

The simple_mapper can be used to map arrays. It will map the array index into the filename between the prefix and suffix.

type messagefile;

(messagefile t) greeting(string m) { 
    app {
        echo m stdout=@filename(t);
    }
}

messagefile outfile[] <simple_mapper;prefix="baz",suffix=".txt">;

outfile[0] = greeting("hello");
outfile[1] = greeting("middle");
outfile[2] = greeting("goodbye");

    Swift variable ------------------->  Filename

       outfile[0]                        baz0000.txt
       outfile[1]                        baz0001.txt
       outfile[2]                        baz0002.txt

		

simple_mapper can be used to map structures. It will map the name of the structure member into the filename, between the prefix and the suffix.

type messagefile;

type mystruct {
  messagefile left;
  messagefile right;
};

(messagefile t) greeting(string m) { 
    app {
        echo m stdout=@filename(t);
    }
}

mystruct out <simple_mapper;prefix="qux",suffix=".txt">;

out.left = greeting("hello");
out.right = greeting("goodbye");
	

This will output the string "hello" into the file qux.left.txt and the string "goodbye" into the file qux.right.txt.

    Swift variable ------------------->  Filename

       out.left                          quxleft.txt
       out.right                         quxright.txt

		

concurrent_mapper is almost the same as the simple mapper, except that it is used to map an output file, and the filename generated will contain an extract sequence that is unique. This mapper is the default mapper for variables when no mapper is specified.

Example:

	file f1;
	file f2 <concurrent_mapper;prefix="foo", suffix=".txt">;
			

The above example would use concurrent mapper for f1 and f2, and generate f2 filename with prefix "foo" and extension ".txt"

filesys_mapper is similar to the simple mapper, but maps a file or a list of files to an array. Each of the filename is mapped as an element in the array. The order of files in the resulting array is not defined.

TODO: note on difference between location as a relative vs absolute path wrt staging to remote location - as mihael said: It's because you specify that location in the mapper. Try location="." instead of location="/sandbox/..."

Example:

	file texts[] <filesys_mapper;prefix="foo", suffix=".txt">;
			

The above example would map all filenames that start with "foo" and have an extension ".txt" into the array texts. For example, if the specified directory contains files: foo1.txt, footest.txt, foo__1.txt, then the mapping might be:

    Swift variable ------------------->  Filename

       texts[0]                          footest.txt
       texts[1]                          foo1.txt
       texts[2]                          foo__1.txt

		

The fixed_array_mapper maps from a string that contains a list of filenames into a file array.

Example:

	file texts[] <fixed_array_mapper;files="file1.txt, fileB.txt, file3.txt">;
			

would cause a mapping like this:

    Swift variable ------------------->  Filename

       texts[0]                          file1.txt
       texts[1]                          fileB.txt
       texts[2]                          file3.txt

		

The array_mapper maps from an array of strings into a file

Example:

string s[] = [ "a.txt", "b.txt", "c.txt" ];

file f[] <array_mapper;files=s>;
		

This will establish the mapping:

    Swift variable ------------------->  Filename

       f[0]                              a.txt
       f[1]                              b.txt
       f[2]                              c.txt

		

The regexp_mapper transforms one file name to another using regular expression matching.

Example:

  string s = "picture.gif";
  file f <regexp_mapper;
    source=s,
    match="(.*)gif",
    transform="\\1jpg">; 

This example transforms a string ending gif into one ending jpg and maps that to a file.

    Swift variable ------------------->  Filename

       f                                    picture.jpg
		

The csv_mapper maps the content of a CSV (comma-separated value) file into an array of structures. The dataset type needs to be correctly defined to conform to the column names in the file. For instance, if the file contains columns: name age GPA then the type needs to have member elements like this:

  type student {
    file name;
    file age;
    file GPA;
  }

If the file does not contain a header with column info, then the column names are assumed as column1, column2, etc.

Example:

	student stus[] <csv_mapper;file="stu_list.txt">;
			

The above example would read a list of student info from file "stu_list.txt" and map them into a student array. By default, the file should contain a header line specifying the names of the columns. If stu_list.txt contains the following:

name,age,gpa
101-name.txt, 101-age.txt, 101-gpa.txt
name55.txt, age55.txt, age55.txt
q, r, s

then some of the mappings produced by this example would be:

    Swift variable ------------------->  Filename

       stus[0].name                         101-name.txt
       stus[0].age                          101-age.txt
       stus[0].gpa                          101-gpa.txt
       stus[1].name                         name55.txt
       stus[1].age                          age55.txt
       stus[1].gpa                          gpa55.txt
       stus[2].name                         q
       stus[2].age                          r
       stus[2].gpa                          s

		

The external mapper, ext maps based on the output of a supplied Unix executable.

The output of the executable should consist of two columns of data, separated by a space. The first column should be the path of the mapped variable, in SwiftScript syntax (for example [2] means the 2nd element of an array) or the symbol $ to represent the root of the mapped variable.

Example: With the following in mapper.sh,

#!/bin/bash
echo "[2] qux"
echo "[0] foo"
echo "[1] bar"
			

then a mapping statement:

	student stus[] <ext;exec="mapper.sh">;
			

would map

    Swift variable ------------------->  Filename

       stus[0]                              foo
       stus[1]                              bar
       stus[2]                              qux

		

The swift command is the main command line tool for executing SwiftScript programs.

The swift-osg-ress-site-catalog command generates a site catalog based on OSG's ReSS information system (since Swift 0.9)

Usage: swift-osg-ress-site-catalog [options]

swift-plot-log generates summaries of Swift run log files.

Usage: swift-plot-log [logfile] [targets]

When no targets are specified, swift-plog-log will generate an HTML report for the run. When targets are specified, only those named targets will be generated.

This section describes how an app procedure invocation is translated into a (remote) unix process execution. It does not describe the mechanisms by which Swift performs that translation; that is described in the next section.

In this section, this example SwiftScript program is used for reference:

 type file;

 app (file o) count(file i) {
   wc @i stdout=@o;
 }

 file q <"input.txt">;
 file r <"output.txt">;

The executable for wc will be looked up in tc.data.

This unix executable will then be executed in some application procedure workspace. This means:

Each application procedure workspace will have an application workspace directory. (TODO: can collapse terms //application procedure workspace// and //application workspace directory// ?

This application workspace directory will not be shared with any other application procedure execution attempt; all application procedure execution attempts will run with distinct application procedure workspaces. (for the avoidance of doubt: If a SwiftScript procedure invocation is subject to multiple application procedure execution attempts (due to Swift-level restarts, retries or replication) then each of those application procedure execution attempts will be made in a different application procedure workspace. )

The application workspace directory will be a directory on a POSIX filesystem accessible throughout the application execution by the application executable.

Before the application executable is executed:

During/after the application executable execution, the following must be true:

Things to not assume:

This section describes the implementation of the semantics described in the previous section.

Swift executes application procedures on one or more sites.

Each site consists of:

There is no assumption that the site shared file system for one site is accessible from another site.

For each workflow run, on each site that is used by that run, a run directory is created in the site working directory, by the Swift client side.

In that run directory are placed several subdirectories:

Application execution looks like this:

For each application procedure call:

The Swift client side selects a site; copies the input files for that procedure call to the site shared file cache if they are not already in the cache, using the file transfer mechanism; and then invokes the wrapper script on that site using the execution mechanism.

The wrapper script creates the application workspace directory; places the input files for that job into the application workspace directory using either cp or ln -s (depending on a configuration option); executes the application unix executable; copies output files from the application workspace directory to the site shared directory using cp; creates a status file under the status/ directory; and exits, returning control to the Swift client side. Logs created during the execution of the wrapper script are stored under the info/ directory.

The Swift client side then checks for the presence of and deletes a status file indicating success; and copies files from the site shared directory to the appropriate client side location.

The job directory is created (in the default mode) under the jobs/ directory. However, it can be created under an arbitrary other path, which allows it to be created on a different file system (such as a worker node local file system in the case that the worker node has a local file system).

The execution layer causes an application program (in the form of a unix executable) to be executed either locally or remotely.

The two main choices are local unix execution and execution through GRAM. Other options are available, and user provided code can also be plugged in.

The kickstart utility can be used to capture environmental information at execution time to aid in debugging and provenance capture.

Step i: text to XML intermediate form parser/processor. parser written in ANTLR - see resources/VDL.g. The XML Schema Definition (XSD) for the intermediate language is in resources/XDTM.xsd.

Step ii: XML intermediate form to Karajan workflow. Karajan.java - reads the XML intermediate form. compiles to karajan workflow language - for example, expressions are converted from SwiftScript syntax into Karajan syntax, and function invocations become karajan function invocations with various modifications to parameters to accomodate return parameters and dataset handling.

Some Swift functionality is provided in the form of Karajan libraries that are used at runtime by the Karajan workflows that the Swift compiler generates.

Takes a command line parameter name as a string parameter and an optional default value and returns the value of that string parameter from the command line. If no default value is specified and the command line parameter is missing, an error is generated. If a default value is specified and the command line parameter is missing, @arg will return the default value.

Command line parameters recognized by @arg begin with exactly one hyphen and need to be positioned after the script name.

For example:

trace(@arg("myparam"));
trace(@arg("optionalparam", "defaultvalue"));
			

$ swift arg.swift -myparam=hello
Swift v0.3-dev r1674 (modified locally)

RunID: 20080220-1548-ylc4pmda
SwiftScript trace: defaultvalue
SwiftScript trace: hello
			

@extractint(file) will read the specified file, parse an integer from the file contents and return that integer.

@filename(v) will return a string containing the filename(s) for the file(s) mapped to the variable v. When more than one filename is returned, the filenames will be space separated inside a single string return value.

@filenames(v) will return multiple values (!) containing the filename(s) for the file(s) mapped to the variable v. (compare to @filename)

@regexp(input,pattern,replacement) will apply regular expression substitution using the Java java.util.regexp API. For example:

string v =  @regexp("abcdefghi", "c(def)g","monkey");

will assign the value "abmonkeyhi" to the variable v.

@strcat(a,b,c,d,...) will return a string containing all of the strings passed as parameters joined into a single string. There may be any number of parameters.

The + operator concatenates two strings: @strcat(a,b) is the same as a + b

@strcut(input,pattern) will match the regular expression in the pattern parameter against the supplied input string and return the section that matches the first matching parenthesised group.

For example:

string t = "my name is John and i like puppies.";
string name = @strcut(t, "my name is ([^ ]*) ");
string out = @strcat("Your name is ",name);
trace(out);
			

will output the message: Your name is John.

@strsplit(input,pattern) will split the input string based on separators that match the given pattern and return a string array. (since Swift 0.9)

Example:

string t = "my name is John and i like puppies.";
string words[] = @strsplit(t, "\\s");
foreach word in words {
	trace(word);
}
			

will output one word of the sentence on each line (though not necessarily in order, due to the fact that foreach iterations execute in parallel).

@toint(input) will parse its input string into an integer. This can be used with @arg to pass input parameters to a SwiftScript program as integers.

readData will read data from a specified file.

The format of the input file is controlled by the type of the return value.

For scalar return types, such as int, the specified file should contain a single value of that type.

For arrays of scalars, the specified file should contain one value per line.

For structs of scalars, the file should contain two rows. The first row should be structure member names separated by whitespace. The second row should be the corresponding values for each structure member, separated by whitespace, in the same order as the header row.

For arrays of structs, the file should contain a heading row listing structure member names separated by whitespace. There should be one row for each element of the array, with structure member elements listed in the same order as the header row and separated by whitespace. (since Swift 0.4)

readdata2 will read data from a specified file, like readdata, but using a different file format more closely related to that used by the ext mapper.

Input files should list, one per line, a path into a Swift structure, and the value for that position in the structure:

rows[0].columns[0] = 0                                                          
rows[0].columns[1] = 2                                                          
rows[0].columns[2] = 4                                                          
rows[1].columns[0] = 1                                                          
rows[1].columns[1] = 3                                                          
rows[1].columns[2] = 5 
				

which can be read into a structure defined like this:

type vector {                                                                   
        int columns[];                                                          
}                                                                               
                                                                                
type matrix {                                                                   
        vector rows[];                                                          
}                                                                               
                                                                                
matrix m;                                                                       
                                                                                
m = readData2("readData2.in");    
				

(since Swift 0.7)

trace will log its parameters. By default these will appear on both stdout and in the run log file. Some formatting occurs to produce the log message. The particular output format should not be relied upon. (since Swift 0.4)

writeData will write out data structures in the format described for readData

maxSubmitRate limits the maximum rate of job submission, in jobs per second. For example:

<profile namespace="karajan" key="maxSubmitRate">0.2</profile>

will limit job submission to 0.2 jobs per second (or equivalently, one job every five seconds).

jobThrottle allows the job throttle factor (see Swift property throttle.score.job.factor) to be set per site.

initialScore allows the initial score for rate limiting and site selection to be set to a value other than 0.

delayBase controls how much a site will be delayed when it performs poorly. With each reduction in a sites score by 1, the delay between execution attempts will increase by a factor of delayBase.

status.mode allows the status.mode property to be set per-site instead of for an entire run. See the Swift configuration properties section for more information. (since Swift 0.8)

storagesize limits the amount of space that will be used on the remote site for temporary files. When more than that amount of space is used, the remote temporary file cache will be cleared using the algorithm specified in the caching.algorithm property.

wrapperInterpreter The wrapper interpreter indicates the command (executable) to be used to run the Swift wrapper script. The default is "/bin/bash" on Unix sites and "cscript.exe" on Windows sites.

wrapperInterpreterOptions Allows specifying additional options to the executable used to run the Swift wrapper. The defaults are no options on Unix sites and "//Nologo" on Windows sites.

wrapperScript Specifies the name of the wrapper script to be used on a site. The defaults are "_swiftwrap" on Unix sites and "_swiftwrap.vbs" on Windows sites. If you specify a custom wrapper script, it must be present in the "libexec" directory of the Swift installation.

cleanupCommand Indicates the command to be run at the end of a Swift run to clean up the run directories on a remote site. Defaults are "/bin/rm" on Unix sites and "cmd.exe" on Windows sites

cleanupCommandOptions Specifies the options to be passed to the cleanup command above. The options are passed in the argument list to the cleanup command. After the options, the last argument is the directory to be deleted. The default on Unix sites is "-rf". The default on Windows sites is ["/C", "del", "/Q"].

maxwalltime specifies a walltime limit for each job, in minutes.

The following formats are recognized:

When replication is enabled (see replication), then walltime will also be enforced at the Swift client side: when a job has been active for more than twice the maxwalltime, Swift will kill the job and regard it as failed.

When clustering is used, maxwalltime will be used to select which jobs will be clustered together. More information on this is available in the clustering section.

When coasters as used, maxwalltime influences the default coaster worker maxwalltime, and which jobs will be sent to which workers. More information on this is available in the coasters section.

queue is used by the PBS, GRAM2 and GRAM4 providers. This profile entry specifies which queue jobs will be submitted to. The valid queue names are site-specific.

host_types specifies the types of host that are permissible for a job to run on. The valid values are site-specific. This profile entry is used by the GRAM2 and GRAM4 providers.

condor_requirements allows a requirements string to be specified when Condor is used as an LRM behind GRAM2. Example: <profile namespace="globus" key="condor_requirements">Arch == "X86_64" || Arch="INTEL"</profile>

slots When using coasters, this parameter specifies the maximum number of jobs/blocks that the coaster scheduler will have running at any given time. The default is 20.

workersPerNode This parameter determines how many coaster workers are started one each compute node. The default value is 1.

nodeGranularity When allocating a coaster worker block, this parameter restricts the number of nodes in a block to a multiple of this value. The total number of workers will then be a multiple of workersPerNode * nodeGranularity. The default value is 1.

allocationStepSize Each time the coaster block scheduler computes a schedule, it will attempt to allocate a number of slots from the number of available slots (limited using the above slots profile). This parameter specifies the maximum fraction of slots that are allocated in one schedule. Default is 0.1.

lowOverallocation Overallocation is a function of the walltime of a job which determines how long (time-wise) a worker job will be. For example, if a number of 10s jobs are submitted to the coaster service, and the overallocation for 10s jobs is 10, the coaster scheduler will attempt to start worker jobs that have a walltime of 100s. The overallocation is controlled by manipulating the end-points of an overallocation function. The low endpoint, specified by this parameter, is the overallocation for a 1s job. The high endpoint is the overallocation for a (theoretical) job of infinite length. The overallocation for job sizes in the [1s, +inf) interval is determined using an exponential decay function: overallocation(walltime) = walltime * (lowOverallocation - highOverallocation) * exp(-walltime * overallocationDecayFactor) + highOverallocation The default value of lowOverallocation is 10.

highOverallocation The high overallocation endpoint (as described above). Default: 1

overallocationDecayFactor The decay factor for the overallocation curve. Default 0.001 (1e-3).

spread When a large number of jobs is submitted to the a coaster service, the work is divided into blocks. This parameter allows a rough control of the relative sizes of those blocks. A value of 0 indicates that all work should be divided equally between the blocks (and blocks will therefore have equal sizes). A value of 1 indicates the largest possible spread. The existence of the spread parameter is based on the assumption that smaller overall jobs will generally spend less time in the queue than larger jobs. By submitting blocks of different sizes, submitted jobs may be finished quicker by smaller blocks. Default: 0.9.

reserve Reserve time is a time in the allocation of a worker that sits at the end of the worker time and is useable only for critical operations. For example, a job will not be submitted to a worker if it overlaps its reserve time, but a job that (due to inaccurate walltime specification) runs into the reserve time will not be killed (note that once the worker exceeds its walltime, the queuing system will kill the job anyway). Default 10 (s).

maxnodes Determines the maximum number of nodes that can be allocated in one coaster block. Default: unlimited.

maxtime Indicates the maximum walltime that a coaster block can have. Default: unlimited.

remoteMonitorEnabled If set to "true", the client side will get a Swing window showing, graphically, the state of the coaster scheduler (blocks, jobs, etc.). Default: false

Profile keys set in the env namespace will be set in the unix environment of the executed job. Some environment variables influence the worker-side behaviour of Swift:

PATHPREFIX - set in env namespace profiles. This path is prefixed onto the start of the PATH when jobs are executed. It can be more useful than setting the PATH environment variable directly, because setting PATH will cause the execution site's default path to be lost.

SWIFT_JOBDIR_PATH - set in env namespace profiles. If set, then Swift will use the path specified here as a worker-node local temporary directory to copy input files to before running a job. If unset, Swift will keep input files on the site-shared filesystem. In some cases, copying to a worker-node local directory can be much faster than having applications access the site-shared filesystem directly.

SWIFT_EXTRA_INFO - set in env namespace profiles. If set, then Swift will execute the command specified in SWIFT_EXTRA_INFO on execution sites immediately before each application execution, and will record the stdout of that command in the wrapper info log file for that job. This is intended to allow software version and other arbitrary information about the remote site to be gathered and returned to the submit side. (since Swift 0.9)

Each pool element must have a handle attribute, giving a symbolic name for the site. This can be any name, but must correspond to entries for that site in the transformation catalog.

Optionally, the gridlaunch attribute can be used to specify the path to kickstart on the site.

Each pool must specify a file transfer method, an execution method and a remote working directory. Optionally, profile settings can be specified.

Transfer methods are specified with either the <gridftp> element or the <filesystem> element.

To use gridftp or local filesystem copy, use the <gridftp> element:

<gridftp  url="gsiftp://evitable.ci.uchicago.edu" />

The url attribute may specify a GridFTP server, using the gsiftp URI scheme; or it may specify that filesystem copying will be used (which assumes that the site has access to the same filesystem as the submitting machine) using the URI local://localhost.

Filesystem access using scp (the SSH copy protocol) can be specified using the <filesystem> element:

<filesystem url="www11.i2u2.org" provider="ssh"/>

For additional ssh configuration information, see the ssh execution provider documentation below.

Filesystem access using CoG coasters can be also be specified using the <filesystem> element. More detail about configuring that can be found in the CoG coasters section.

Execution methods may be specified either with the <jobmanager> or <execution> element.

The <jobmanager> element can be used to specify execution through GRAM2. For example,

    <jobmanager universe="vanilla" url="evitable.ci.uchicago.edu/jobmanager-fork" major="2" />

The universe attribute should always be set to vanilla. The url attribute should specify the name of the GRAM2 gatekeeper host, and the name of the jobmanager to use. The major attribute should always be set to 2.

The <execution> element can be used to specify execution through other execution providers:

To use GRAM4, specify the gt4 provider. For example:

<execution provider="gt4" jobmanager="PBS" url="tg-grid.uc.teragrid.org" />

The url attribute should specify the GRAM4 submission site. The jobmanager attribute should specify which GRAM4 jobmanager will be used.

For local execution, the local provider should be used, like this:

<execution provider="local" url="none" />

For PBS execution, the pbs provider should be used:

<execution provider="pbs" url="none" />

The GLOBUS::queue profile key can be used to specify which PBS queue jobs will be submitted to.

For execution through a local Condor installation, the condor provider should be used. This provider can run jobs either in the default vanilla universe, or can use Condor-G to run jobs on remote sites.

When running locally, only the <execution> element needs to be specified:

<execution provider="condor" url="none" />

When running with Condor-G, it is necessary to specify the Condor grid universe and the contact string for the remote site. For example:

 <execution provider="condor" />
 <profile namespace="globus" key="jobType">grid</profile>
 <profile namespace="globus" key="gridResource">gt2 belhaven-1.renci.org/jobmanager-fork</profile>

For execution through SSH, the ssh provider should be used:

<execution url="www11.i2u2.org" provider="ssh"/>

with configuration made in ~/.ssh/auth.defaults with the string 'www11.i2u2.org' changed to the appropriate host name:

www11.i2u2.org.type=key
www11.i2u2.org.username=hategan
www11.i2u2.org.key=/home/mike/.ssh/i2u2portal
www11.i2u2.org.passphrase=XXXX

For execution using the CoG Coaster mechanism, the coaster provider should be used:

<execution provider="coaster" url="tg-grid.uc.teragrid.org"
    jobmanager="gt2:gt2:pbs" />

More details about configuration of coasters can be found in the section on coasters.

The workdirectory element specifies where on the site files can be stored.

<workdirectory>/home/benc</workdirectory>

This file must be accessible through the transfer mechanism specified in the <gridftp> element and also mounted on all worker nodes that will be used for execution. A shared cluster scratch filesystem is appropriate for this.

Profile keys can be specified using the <profile> element. For example:

<profile namespace="globus" key="queue">fast</profile>

If an application procedure execution fails, Swift will attempt that execution again repeatedly until it succeeds, up until the limit defined in the execution.retries configuration property.

Site selection will occur for retried jobs in the same way that it happens for new jobs. Retried jobs may run on the same site or may run on a different site.

If the retry limit execution.retries is reached for an application procedure, then that application procedure will fail. This will cause the entire run to fail - either immediately (if the lazy.errors property is false) or after all other possible work has been attempted (if the lazy.errors property is true).

If a run fails, Swift can resume the program from the point of failure. When a run fails, a restart log file will be left behind in a file named using the unique job ID and a .rlog extension. This restart log can then be passed to a subsequent Swift invocation using the -resume parameter. Swift will resume execution, avoiding execution of invocations that have previously completed successfully. The SwiftScript source file and input data files should not be modified between runs.

Every run creates a restart log file with a named composed of the file name of the workflow being executed, an invocation ID, a numeric ID, and the .rlog extension. For example, example.swift, when executed, could produce the following restart log file: example-ht0adgi315l61.0.rlog. Normally, if the run completes successfully, the restart log file is deleted. If however the workflow fails, swift can use the restart log file to continue execution from a point before the failure occurred. In order to restart from a restart log file, the -resume logfile argument can be used after the SwiftScript program file name. Example:

$ swift -resume example-ht0adgi315l61.0.rlog example.swift.

When an execution job has been waiting in a site queue for a certain period of time, Swift can resubmit replicas of that job (up to the limit defined in the replication.limit configuration property). When any of those jobs moves from queued to active state, all of the other replicas will be cancelled.

This is intended to deal with situations where some sites have a substantially longer (sometimes effectively infinite) queue time than other sites. Selecting those slower sites can cause a very large delay in overall run time.

Replication can be enabled by setting the replication.enabled configuration property to true. The maximum number of replicas that will be submitted for a job is controlled by the replication.limit configuration property.

When replication is enabled, Swift will also enforce the maxwalltime profile setting for jobs as documented in the profiles section.

If you have a UChicago Computation Institute account, run this command in your submit directory after each run. It will copy all your logs and kickstart records into a directory at the CI for reporting, usage tracking, support and debugging.

rsync --ignore-existing *.log *.d login.ci.uchicago.edu:/disks/ci-gpfs/swift/swift-logs/ --verbose

TeraGrid users with no default project or with several project allocations can specify a project allocation using a profile key in the site catalog entry for a TeraGrid site:

<profile namespace="globus" key="project">TG-CCR080002N</profile>

More information on the TeraGrid allocations process can be found here.

Here is an example of running a simple MPI program.

In SwiftScript, we make an invocation that does not look any different from any other invocation. In the below code, we do not have any input files, and have two output files on stdout and stderr:

type file;

(file o, file e) p() { 
    app {
        mpi stdout=@filename(o) stderr=@filename(e);
    }
}

file mpiout <"mpi.out">;
file mpierr <"mpi.err">;

(mpiout, mpierr) = p();

Now we define how 'mpi' will run in tc.data:

tguc    mpi             /home/benc/mpi/mpi.sh   INSTALLED       INTEL32::LINUX GLOBUS::host_xcount=3

mpi.sh is a wrapper script that launches the MPI program. It must be installed on the remote site:

#!/bin/bash
mpirun -np 3 -machinefile $PBS_NODEFILE /home/benc/mpi/a.out 

Because of the way that Swift runs its server side code, provider-specific MPI modes (such as GRAM jobType=mpi) should not be used. Instead, the mpirun command should be explicitly invoked.

Since 10/11/09, the development version of Swift has the ability to run on a Windows machine, as well as the ability to submit jobs to a Windows site (provided that an appropriate provider is used).

In order to launch Swift on Windows, use the provided batch file (swift.bat). In certain cases, when a large number of jar libraries are present in the Swift lib directory and depending on the exact location of the Swift installation, the classpath environment variable that the Swift batch launcher tries to create may be larger than what Windows can handle. In such a case, either install Swift in a directory closer to the root of the disk (say, c:\swift) or remove non-essential jar files from the Swift lib directory.

Due to the large differences between Windows and Unix environments, Swift must use environment specific tools to achieve some of its goals. In particular, each Swift executable is launched using a wrapper script. This script is a Bourne Shell script. On Windows machines, which have no Bourne Shell interpreter installed by default, the Windows Scripting Host is used instead, and the wrapper script is written in VBScript. Similarly, when cleaning up after a run, the "/bin/rm" command available in typical Unix environments must be replaced by the "del" shell command.

It is important to note that in order to select the proper set of tools to use, Swift must know when a site runs under Windows. To inform Swift of this, specify the "sysinfo" attribute for the "pool" element in the site catalog. For example:

	<pool handle="localhost" sysinfo="INTEL32::WINDOWS">
	...
	</pool>