This manual provides reference material for Swift: the SwiftScript language and the Swift runtime system. For introductory material, consult the Swift tutorial.

Swift is a data-oriented coarse grained scripting language that supports dataset typing and mapping, dataset iteration, conditional branching, and sub-workflow composition.

Swift programs (or workflows) are written in a language called SwiftScript.

SwiftScript programs are dataflow oriented - they are primarily concerned with processing (possibly large) data files, by invoking programs to do that processing. Swift handles execution of such programs on remote sites by choosing sites, handling the staging of input and output files to and from the chosen sites and remote execution of program code.

2. The SwiftScript Language

2.1. Data model

Data processed by Swift is strongly typed. It may be take the form of values in memory or as out-of-core files on disk. Language constructs called mappers specify how each piece of data is stored.

2.1.1. Data

Data is represented in Swift by DSHandles (Dataset handles).

In Swift, a DSHandle can represent data in one of three forms:

an in-memory value, such as a string or integer.
a data file (on local disk or stored elsewhere on the internet). When a DSHandle represents an on-disk data file, the name of that file is provided by a mapper.
a container of other DSHandles - either an array or a defined type. Such a DSHandles contains subordinate DSHandles; these may be nested to arbitrary depth.

The above three are mutually exclusive - for example, a data set that is mapped to a file cannot have a value that can be used in a SwiftScript expression; and a data item that is a value cannot be passed into an application executable as a data file using the @filename function.

2.1.2. Mappers

When a DSHandle represents a data file (or container of datafiles), it is associated with a mapper. The mapper is used to identify which files belong to that DSHandle.

A dataset's physical representation is declared by a mapping descriptor, which defines how each element in the dataset's logical schema is stored in, and fetched from, physical structures such as directories, files, and remote servers.

Mappers are parameterized to take into account properties such as varying dataset location. In order to access a dataset, we need to know three things: its type, its mapping, and the value(s) of any parameter(s) associated with the mapping descriptor. For example, if we want to describe a dataset, of type imagefile, and whose physical representation is a file called "file1.bin" located at "/home/yongzh/data/", then the dataset might be declared as follows:

imagefile f1<single_file_mapper;file="/home/yongzh/data/file1.bin">

The above example declares a dataset called f1, which uses a single file mapper to map a file from a specific location.

SwiftScript has a simplified syntax for this case, since single_file_mapper is frequently used:

binaryfile f1<"/home/yongzh/data/file1.bin">

Swift comes with a number of mappers that handle common mapping patterns. These are documented in the mappers section of this guide.

2.1.3. Execution order based on data dependencies

Procedures in swift are by default executed in parallel. If five separate procedures are invoked, Swift will attempt to run them all at once.

The main exception to this is when one procedure produces a dataset as an output and another procedure uses that dataset as an input. In that case, the second procedure will be executed after the first procedure has produced the intermediate dataset.

2.1.4. Type System

The SwiftScript type system consists of a number of simple types, marker types for files, arrays, and complex types composed of these types.

The simple types are: string, float, int and boolean.

Complex types are specified using the type keyword. The syntax is similar to struct in C or class in Java. For example, the below example declares a complex type with two members, a string called name and an integer called age.

type person {
	string name;
	int age;
}

When referring to files on disk, the internal structure of the file is irrelevant; but it is still useful to declare types for those files so that Swift can perform type-checking on SwiftScript programs. In this case, a marker type can be declared, like this:

type binaryfile;

2.2. Variables

Variables in SwiftScript are declared to be of a specific type. Assignments to those variables must be data of that type. SwiftScript variables are single-assignment - a value may be assigned to a variable at most once. This assignment can happen at declaration time or later on in execution. When an attempt to read from a variable that has not yet been assigned is made, the code performing the read is suspended until that variable has been written to. This forms the basis for Swift's ability to parallelise execution - all code will execute in parallel unless there are variables shared between the code that cause sequencing.

2.2.1. Variable Declarations

Variable declaration statements declare new variables. They can optionally assign a value to them or map those variables to on-disk files.

Declaration statements have the general form:

  typename variablename (<mapping> | = initialValue ) ;

The format of the mapping expression is defined in the Mappers section. initialValue may be either an expression or a procedure call that returns a single value.

Variables can also be declared in a multivalued-procedure statement, described in another section.

2.2.2. Assignment Statements

Assignment statements assign values to previously declared variables. Assignments may only be made to variables that have not already been assigned. Assignment statements have the general form:

  variable = value;

where value can be either an expression or a procedure call that returns a single value.

Variables can also be assigned in a multivalued-procedure statement, described in another section.

2.3. Procedures

Datasets are operated on by procedures, which take input in the form of mapped variables, perform computations, and produce typed data as output that is again mapped to variables.

There are two kinds of procedure: An atomic procedure, which describes how an external program can be executed; and compound procedures which consist of a sequence of SwiftScript statements.

A procedure declaration defines the name of a procedure and its input and output parameters. SwiftScript procedures can take multiple inputs and produce multiple outputs. Inputs are specified to the right of the function name where outputs to the left. For instance:

(type3 out1, type4 out2) myproc (type1 in1, type2 in2)

The above example declares a procedure called myproc, which has two inputs in1 (of type type1) and in2 (of type type2) and two outputs out1 (of type type3) and out2 (of type type4).

A procedure input parameter can be an optional parameter in which case it must be declared with a default value. When we call a procedure, passing in the actual parameters, we allow both positional parameter and named parameter passing, provided that all optional parameters have to be declared after the required parameters and any optional parameter has to be bound using keyword parameter passing. So for instance if we declare a procedure myproc1:

(binaryfile bf) myproc1 (int i, string s="foo")

Then the procedure can be called like this

binaryfile mybf = myproc1(1);

or like this supplying the value for the optional parameter s:

binaryfile mybf = myproc1 (1, s="bar");

2.3.1. Atomic procedures

The body of an atomic procedure specifies how to invoke an external executable program or Web Service, and how logical data types are mapped to command line arguments. A complete specification for myproc1 can be:

(binaryfile bf) myproc1 (int i, string s="foo") {
	app {
		myapp1 i s @filename(bf);
	}
}

which specifies that myproc1 invokes an executable called myapp1, passing the values of i, s and the file name of bf as command line arguments. The @filename notation serves as a function denoting that the argument should be mapped as a file name, and since the notation is often required in invoking applications, a shorter syntax is defined where we can omit the filename part and use the @ sign only.

2.3.2. Compound procedures

A compound procedure contains a set of calls to other procedures. Shared variables in the body of a compound procedure specify data dependencies and thus the execution sequence of the procedure calls. For simple illustration, we define a compound procedure in below:

(type2 b) foo_bar (type1 a) {
	type3 c;
	c = foo(a);    // c holds the result of foo
	b = bar(c);    // c is an input to bar
}

2.4. Syntax, Statements

The syntax of SwiftScript has a superficial resemblance to C and Java. For example, { and } characters are used to enclose blocks of statements.

A SwiftScript program consists of a number of statements. Statements may declare types, procedures and variables, assign values to variables, and express operations over arrays.

2.5. Multivalued procedure invocation statements

Procedures can return more than one value. In such case, the previously mentioned declaration and assignment statements are insufficient. A multi-valued procedure invocation can be used. This has the general form:

'(' ((type)? variableName ( '=' binding ))+ ')' = procedureinvocation

Variables can be either declared (if a type is included) or assigned (if a type is not included). If no bindings are specified, then variables are assigned in the same order that they are specified in the procedure declaration. If bindings are specified, then variables are assigned to the named return parameter.

2.6. Control Constructs

SwiftScript provides if, switch, foreach, and while constructs, with syntax and semantics similar to comparable constructs in other high-level languages.

2.6.1. foreach

The foreach construct is used to apply a block of statements to each element in an array. For example:

check_order (file a[]) {
	foreach f in a {
		compute(f);
	}
}

foreach statements have the general form:

foreach controlvariable (,index) in expression {
    statements
}

The block of statements is evaluated once for each element in 'expression', with controlvariable set to the corresponding element and index set to the integer position in the array that is being iterated over.

2.6.2. if

The 'if' statement allows one of two blocks of statements to be executed, based on a boolean predicate. 'if' statements generally have the form:

if(predicate) {
    statements
} else {
    statements
}

where predicate is a boolean expression.

2.6.3. switch

Switch expressions allow one of a selection of blocks to be chosen based on the value of a numerical control expression. Switch statements take the general form:

switch(controlExpression) {
    case n1:
        statements2
    case n2:
        statements2
    [...]
    default:
        statements
}

The control expression is evaluated and the resulting numerical value used to select a corresponding case, and the statements belonging to that case are evaluated. If no case corresponds, then the statements belonging to the default block are evaluated.

Unlike C or Java switch statements, execution does not fall through to subsequent case blocks, and no break statement is necessary at the end of each block.

2.6.4. iterate

Iterate expressions allow a block of code to be evaluated repeatedly, with an integer parameter sweeping upwards from 0 until a termination condition holds.

The general form is:

iterate var {
	statements;
} until (terminationExpression);

with the variable var starting at 0 and increasing each iteration. That variable is in scope in the statements block and when evaluating the termination expression.

2.7. Operators

The following infix operators are available for use in SwiftScript expressions.

+ numeric addition; string concatenation

- numeric subtraction

* numeric multiplication

/ floating point division

%/ integer division

%% integer remainder-of-division

== != comparison and not-comparison

< > <= >=

&& || boolean and, or

! boolean not

3. Mappers

Mappers provide a mechanism to specify the layout of mapped datasets on disk. This is needed when Swift must access files to transfer them to remote sites for execution or to pass to applications.

Swift provides a number of mappers that are useful in common cases. This section details those standard mappers. For more complex cases, it is possible to write application-specific mappers in Java and use them within a SwiftScript program. For more information on writing an application specific mapper, see the tutorial module on mappers.

3.1. The single file mapper

Name: single_file_mapper

Description: A single file mapper maps a single physical file to a dataset.


    Swift variable ------------------->  Filename

       f                                 myfile

       f[0]                              INVALID

       f.bar                             INVALID

Parameter:
- file: The location of the physical file including path and file name.

Example:

	file f <single_file_mapper;file="plot_outfile_param">;

There is a simplified syntax for this mapper:

	file f <"plot_outfile_param">;

3.2. The simple mapper

Name: simple_mapper
Description: A mapper that maps a file or a list of files into an array by prefix, suffix, and pattern. If more than one file is matched, each of the file names will be mapped as a subelement of the dataset.
Parameters:
- location: A directory that the files are located.
- prefix: The prefix of the files
- suffix: The suffix of the files, for instance: ".txt"
- pattern: A UNIX glob style pattern, for instance: "*foo*" would match all file names that contain foo. When this mapper is used to specify output filenames, pattern is ignored.

Examples:

	type file;
	file f <simple_mapper;prefix="foo", suffix=".txt">;

The above maps all filenames that start with foo and have an extension .txt into file f.


    Swift variable ------------------->  Filename

       f                                 foo.txt

type messagefile;

(messagefile t) greeting(string m) {.
    app {
        echo m stdout=@filename(t);
    }
}

messagefile outfile <simple_mapper;prefix="foo",suffix=".txt">;

outfile = greeting("hi");

This will output the string 'hi' to the file foo.txt.

The simple_mapper can be used to map arrays. It will map the array index into the filename between the prefix and suffix.

type messagefile;

(messagefile t) greeting(string m) { 
    app {
        echo m stdout=@filename(t);
    }
}

messagefile outfile[] <simple_mapper;prefix="baz",suffix=".txt">;

outfile[0] = greeting("hello");
outfile[1] = greeting("middle");
outfile[2] = greeting("goodbye");


    Swift variable ------------------->  Filename

       outfile[0]                        baz0000.txt
       outfile[1]                        baz0001.txt
       outfile[2]                        baz0002.txt

simple_mapper can be used to map structures. It will map the name of the structure member into the filename, between the prefix and the suffix.

type messagefile;

type mystruct {
  messagefile left;
  messagefile right;
};

(messagefile t) greeting(string m) { 
    app {
        echo m stdout=@filename(t);
    }
}

mystruct out <simple_mapper;prefix="qux",suffix=".txt">;

out.left = greeting("hello");
out.right = greeting("goodbye");

This will output the string "hello" into the file qux.left.txt and the string "goodbye" into the file qux.right.txt.


    Swift variable ------------------->  Filename

       out.left                          quxleft.txt
       out.right                         quxright.txt

3.3. concurrent mapper

Name: concurrent_mapper
Description: Concurrent mapper is almost the same as the simple mapper, except that it is used to map an output file, and the filename generated will contain an extract sequence that is unique. This mapper is the default mapper for variables when no mapper is specified.
Parameters:
- location: A directory that the files are located.
- prefix: The prefix of the files
- suffix: The suffix of the files, for instance: ".txt"
- pattern: A UNIX glob style pattern, for instance: "*foo*" would match all file names that contain foo. When this mapper is used to specify output filenames, pattern is ignored.
Example:
```
	file f1;
	file f2 <concurrent_mapper;prefix="foo", suffix=".txt">;
			
```
The above example would use concurrent mapper for f1 and f2, and generate f2 filename with prefix "foo" and extension ".txt"

3.4. file system mapper

TODO: note on difference between location as a relative vs absolute path wrt staging to remote location - as mihael said: It's because you specify that location in the mapper. Try location="." instead of location="/sandbox/..."

Name: filesys_mapper
Description: This mapper is similar to the simple mapper, but maps a file or a list of files to an array. Each of the filename is mapped as an element in the array. The order of files in the resulting array is not defined.
Parameters:
- location: The directory where the files are located.
- prefix: The prefix of the files
- suffix: The suffix of the files, for instance: ".txt"
- pattern: A UNIX glob style pattern, for instance: "*foo*" would match all file names that contain foo.

Example:

	file texts[] <filesys_mapper;prefix="foo", suffix=".txt">;

The above example would map all filenames that start with "foo" and have an extension ".txt" into the array texts. For example, if the specified directory contains files: foo1.txt, footest.txt, foo__1.txt, then the mapping might be:


    Swift variable ------------------->  Filename

       texts[0]                          footest.txt
       texts[1]                          foo1.txt
       texts[2]                          foo__1.txt

3.5. fixed array mapper

Name: fixed_array_mapper
Description: This mapper maps from a string that contains a list of filenames into a file array.
Parameter:
- files: A string that contains a list of filenames, separated by space, comma or colon

Example:

	file texts[] <fixed_array_mapper;files="file1.txt, fileB.txt, file3.txt">;

would cause a mapping like this:


    Swift variable ------------------->  Filename

       texts[0]                          file1.txt
       texts[1]                          fileB.txt
       texts[2]                          file3.txt

3.6. array mapper

Name: array_mapper
Description: This mapper froms an array of strings into a file array.
Parameter:
- files: An array of strings containing one filename per element

Example:

string s[] = [ "a.txt", "b.txt", "c.txt" ];

file f[] <array_mapper;files=s>;

This will establish the mapping:


    Swift variable ------------------->  Filename

       f[0]                              a.txt
       f[1]                              b.txt
       f[2]                              c.txt

3.7. regular expression mapper

Name: regexp_mapper
Description: This mapper transforms one file name to another using regular expression matching.
Parameters:
- source: The source file name
- match: Regular expression pattern to match, use ( ) to match whatever regular expression is inside the parentheses, and indicate the start and end of a group; the contents of a group can be retrieved with the \number special sequence
- transform: The pattern of the file name to transform to, use \number to reference the group matched.

Example:

	string s = "picture.gif";
	file f <regexp_mapper;source=s,match="(.*)gif",transform="\1jpg">;

This example transforms a string "gif" into one ending with "jpg" and maps that to a file.


    Swift variable ------------------->  Filename

       f                                 picture.jpg

3.8. csv mapper

Name: csv_mapper
Description: This mapper maps the content of a CSV (comma-separated value) file into an array of structures. The dataset type needs to be correctly defined to conform to the column names in the file. For instance, if the file contains columns:
name age GPA
then the type needs to have the same member elements, say:
```
		type student {
			File  name;
			File  age;
			File  GPA;
		}
		
```
If the file does not contain a header with column info, then the column names are assumed as "column1", "column2", etc.
Parameters:
- file: The name of the CSV file to read mappings from.
- header: Whether the file has a line describing header info; default is true
- skip: The number of lines to skip at the beginning (after header line); default is 0.
- hdelim: Header field delimiter; default is the value of the "delim" parameter
- delim: Content field delimiters; defaults are space, tab and comma.

Example:

	student stus[] <csv_mapper;file="stu_list.txt">;

The above example would read a list of student info from file "stu_list.txt" and map them into a student array. By default, the file should contain a header line specifying the names of the columns. If stu_list.txt contains the following:

name,age,gpa
101-name.txt, 101-age.txt, 101-gpa.txt
name55.txt, age55.txt, age55.txt
q, r, s

then some of the mappings produced by this example would be:


    Swift variable ------------------->  Filename

       stus[0].name                         101-name.txt
       stus[0].age                          101-age.txt
       stus[0].gpa                          101-gpa.txt
       stus[1].name                         name55.txt
       stus[1].age                          age55.txt
       stus[1].gpa                          gpa55.txt
       stus[2].name                         q
       stus[2].age                          r
       stus[2].gpa                          s

3.9. external mapper

Name: ext
Description: This mapper maps based on the output of a supplied Unix executable.
Parameters:
- exec: The name of the executable (relative to the current directory, if an absolute path is not specified)
- Other parameters are passed to the executable prefixed by a - symbol.
The output of the executable should consist of two columns of data, separated by a space. The first column should be the path of the mapped variable, in SwiftScript syntax (for example [2] means the 2nd element of an array) or the symbol $ to represent the root of the mapped variable.

Example: With the following in mapper.sh,

#!/bin/bash
echo "[2] qux"
echo "[0] foo"
echo "[1] bar"

then a mapping statement:

	student stus[] <ext;exec="mapper.sh">;

would map


    Swift variable ------------------->  Filename

       stus[0]                              foo
       stus[1]                              bar
       stus[2]                              qux

4. Swift configuration properties

Various aspects of the behavior of the Swift Engine can be configured through properties. The Swift Engine recognizes a global, per installation properties file which can found in $SWIFT_HOME/etc/swift.properties and a user properties file which can be created by each user in ~/.swift/swift.properties. The Swift Engine will first load the global properties file. It will then try to load the user properties file. If a user properties file is found, individual properties explicitly set in that file will override the respective properties in the global properties file. Furthermore, some of the properties can be overridden directly using command line arguments to the swift command.

Swift properties are specified in the following format:

<name>=<value>

The value can contain variables which will be expanded when the properties file is read. Expansion is performed when the name of the variable is used inside the "standard" shell dereference construct: ${name}. The following variables can be used in the Swift configuration file:

Swift Configuration Variables

swift.home: Points to the Swift installation directory ($SWIFT_HOME).
user.name: The name of the current logged in user.
user.home: The user's home directory.

The following is a list of valid Swift properties:

Swift Properties

sites.file

Valid values: <file>

Default value: ${swift.home}/etc/sites.xml

Points to the location of the site catalog, which contains a list of all sites that Swift should use.

tc.file

Valid values: <file>

Default value: ${swift.home}/etc/tc.data

Points to the location of the transformation catalog file which contains information about installed applications. Details about the format of the transformation catalog can be found here.

ip.address

Valid values: <ipaddress>

Default value: N/A

The Globus GRAM service uses a callback mechanism to send notifications about the status of submitted jobs. The callback mechanism requires that the Swift client be reachable from the hosts the GRAM services are running on. Normally, Swift can detect the correct IP address of the client machine. However, in certain cases (such as the client machine having more than one network interface) the automatic detection mechanism is not reliable. In such cases, the IP address of the Swift client machine can be specified using this property. The value of this property must be a numeric address without quotes.

lazy.errors

Valid values: true, false

Default value: false

Swift can report application errors in two modes, depending on the value of this property. If set to false, Swift will report the first error encountered and immediately stop execution. If set to true, Swift will attempt to run as much as possible from a workflow before stopping execution and reporting all errors encountered.

When developing workflows, using the default value of false can make the workflow easier to debug. However in production runs, using true will allow more of a workflow to be run before Swift aborts execution.

caching.algorithm

Valid values: LRU

Default value: LRU

Swift caches files that are staged in on remote resources, and files that are produced remotely by applications, such that they can be re-used if needed without being transfered again. However, the amount of remote file system space to be used for caching can be limited using the swift:storagesize profile entry in the sites.xml file. Example:


<pool handle="example" sysinfo="INTEL32::LINUX">
	<gridftp url="gsiftp://example.org" storage="/scratch/swift" major="2" minor="4" patch="3"/>
	<jobmanager universe="vanilla" url="example.org/jobmanager-pbs" major="2" minor="4" patch="3"/>
	<workdirectory>/scratch/swift</workdirectory>
	<profile namespace="SWIFT" key="storagesize">20000000</profile>
</pool>

The decision of which files to keep in the cache and which files to remove is made considering the value of the caching.algorithm property. Currently, the only available value for this property is LRU, which would cause the least recently used files to be deleted first.

execution.retries

Valid values: positive integers

Default value: 2

The number of time a job will be retried if it fails (giving a maximum of 1 + execution.retries attempts at execution)

pgraph

Valid values: true, false, <file>

Default value: false

Swift can generate a Graphviz file representing the structure of the workflow it runs. If this property is set to true, Swift will save the provenance graph in a file named by concatenating the workflow name and the instance ID (e.g. helloworld-ht0adgi315l61.dot).

If set to false, no provenance graph will be generated. If a file name is used, then the provenance graph will be saved in the specified file.

The generated dot file can be rendered into a graphical form using Graphviz, for example with a command-line such as:

								swift -pgraph graph1.dot q1.swift
								dot -ograph.png -Tpng graph1.dot

pgraph.graph.options

Valid values: <string>

Default value: splines="compound", rankdir="TB"

This property specifies a Graphviz specific set of parameters for the graph.

pgraph.node.options

Valid values: <string>

Default value: color="seagreen", style="filled"

Used to specify a set of Graphviz specific properties for the nodes in the graph.

clustering.enabled

Valid values: true, false

Default value: false

Enables clustering.

clustering.queue.delay

Valid values: <int>

Default value: 4

This property indicates the interval, in seconds, at which the clustering queue is processed.

clustering.min.time

Valid values: <int>

Default value: 60

Indicates the threshold wall time for clustering, in seconds. Jobs that have a wall time smaller than the value of this property will be considered for clustering.

kickstart.enabled

Valid values: true, false, maybe

Default value: maybe

This option allows controlling of when Swift uses Kickstart. A value of false disables the use of Kickstart, while a value of true enables the use of Kickstart, in which case sites specified in the sites.xml file must have valid gridlaunch attributes. The maybe value will enable the use of Kickstart only on sites that have the gridlaunch attribute specified.

kickstart.always.transfer

Valid values: true, false

Default value: false

This property controls when output from Kickstart is transfered back to the submit site, if Kickstart is enabled. When set to false, Kickstart output is only transfered for jobs that fail. If set to true, Kickstart output is transfered after every job is completed or failed.

wrapperlog.always.transfer

Valid values: true, false

Default value: false

This property controls when output from the Swift remote wrapper is transfered back to the submit site. When set to false, wrapper logs are only transfered for jobs that fail. If set to true, wrapper logs are transfered after every job is completed or failed.

throttle.submit