by Marc Stiegler
Copyright (C) 2000, Marc Stiegler. All Rights Reserved.
Table of Contents
Java, Perl, Python, C++, TCL, and on and on. Don't we have enough languages already? How could we justify bringing yet another programming language into the world?
In fact, there is still a compelling reason for developing better programming languages: we keep on writing more complicated programs. And worse, we keep on writing more complicated systems of interdependent programs.
We have entered the age of globally distributed computing with a vengeance. Anyone who has cobbled together a major system with a hodgepodge of Web servers, Java, JSP, SQL, CGI, CORBA, RMI, XML, and Perl knows that this cannot be the toolset we will use in 20 years. The sooner we move up to the tools of 20 years hence, the better off we will be. Into this situation we introduce E:
These qualities cannot be achieved with traditional security approaches. Do not expect the next release of Java, Windows, or Linux to fix the problem: the flaws in these systems lie at the heart of their architectures, unfixable without breaking upward compatibility, as we shall discuss in the chapter on Secure Distributed Programming. Of course, there is nothing to prevent people from advertising that they are releasing a new, upward compatible, totally-secure version of a product. Just don't jump off the Brooklyn Bridge to buy it.
E wraps these strengths in a C/Java syntax to make it quickly comfortable for a large number of software engineers. It is built with objects at the core of its design, making it easy to write modular, readable, maintainable software using the strategies familiar from Java. It has the kind of powerful string handling that will be recognized and seized upon by the Perl programmer. For both better and for worse, E is a dynamically typed language like Smalltalk, not a statically typed language like Java. Users of Perl and Python will immediately assert this is an advantage; Java and C++ programmers will not be so sure. A discussion of the merits and demerits of static typing could fill a book the size of this one. Many of the most complex yet most reliable systems in the world today have been developed with dynamically typed languages. If you are a Java programmer, unshakably convinced of the perfect correctness of static typing, all we can do is urge you to try E first and form your conclusions later. We believe you will find the experience both pleasant and productive, as the long heritage of programmers from Scheme to Smalltalk to Perl and Python have found in the past.
E is not a panacea. The current implementation of E runs on top of the java virtual machine. As such, it is not a good language for low level machine manipulation. So do not try to use E for writing device drivers. And E's performance is quite unfavorable compared to raw C, so do not write the real time kernel of a missile guidance system with it, either.
Actually, most arguments over performance miss the most important points about software development. One of the few time-tested truths of software development is, "first get it to work, then get it to work fast". E is an excellent language for getting a system to work. Once it is working, you can gain the performance rewards by profiling to see which portions of the code actually affect performance, then rewriting those (typically small) portions in Java, or C, or assembler, depending on your needs. In fact, for many traditional compute-intensive activities, E is just as performance-efficient as anything else: when creating windows and buttons on the screen, you use the gui widgets from the underlying jvm. Since the jvm uses native drawing machinery, there is little distinction among languages when working the windows.
So perhaps E is in fact almost a panacea. But not quite.
This book is intended as an introductory text for practical E programming. Comparisons to Java are frequent, so some understanding of Java is desirable. If you wish to write software with point-and-click user interfaces, familiarity with either the Java Swing classes or the IBM Standard Widget Toolkit (SWT) is practically required.
This book is not a language specification. This book is an introduction to get you started and productive. If you encounter some surprising behavior not explained here, please join the e-lang discussion group and ask there: many helpful people can be found. If you need a precise specification, look to the reference materials posted by the language author and keeper, Mark Miller, at the ERights web site. You can join the e-lang discussion group at the same site.
When a very early version of E and E in a Walnut were presented to a programmer chat group, this was one of the comments:
"I think the key to hooking someone is to make them read the whole walnut cause it just looks like another scripting language until you get into the security and distribution then its like fireworks."
Though E is a powerful language with which to write single-cpu programs, the main power of E becomes evident only after you move into distributed programming. It would be tempting to introduce the distributed computing features first...except that you can't really do any meaningful computing without the basic data, flow, function, and object structures. So this book introduces "Ordinary Programming" Part I before getting into the serious distributed computing facilities. However, since E was designed in the C/Java syntax tradition, an experienced programmer can probably glean enough from the Quick Reference Card to skip directly to Part II on Distributed Computing. If you are short of time and have the requisite background, we recommend that strategy. Go back and read Part I when you are convinced that E's power for distributed programming meets your needs.
If you have read this far, you will probably want to retrieve a current version of E from www.erights.org. Follow the installation directions in the distribution.
You will find the rune interactive interpreter by going into the E directory and typing
java -jar e.jar --rune
Rune will be your friend for experimentation as you learn E. Indeed, you may find, as others have, that rune is useful even when developing ordinary Java: with this scratchpad you can quickly explore what the API for Java really does when you use it, quickly and easily, before you put Java code into the middle of a program that is difficult to test.
Crucial note: At the time of this writing, E is in transition to a return-keyword based syntax for returning values from methods and functions. The following examples will only work if you put the following pragmas at the top of E source files:
# E sample
pragma.syntax("0.9")
Using Rune: One-line commands start with a question mark prompt and return the result on the next line in a comment:
? 2 + 2 # value: 4
Commands that cannot be completed in a single line will receive close-angle-bracket continuations when you press carriage return at the end of the first line; the whole set of lines from the question mark down to the end of the command (the closing brace) will be evaluated when you close the statement:
? if (3 < 5) {
> println("is less")
> }
is lessAnother one of the wonderful little E features that helps enormously in rune is the help(obj) function. Type "help(interp)" into rune, and you will get a complete listing of all the methods you can call on the interp object. You will be surprised at first glance by the number of methods there; these methods are explained throughout the course of the book, starting at the end of the discussion of E objects.
We will present examples both as straight E source and as rune source, depending on how complex the example is.
The Trace Log will be one of your best friends when you step outside of rune and start writing E programs. When you install E, one of the items in the "eprops.txt" file is a specification of the directory where trace information should be written when a program fails during execution. Find your etrace directory(on Windows95, c:\windows\temp\etrace), and go there when you need debugging information. This is an important resource--more than one early adopter of E has kicked himself for forgetting that it is there, then spending several hours trying to understand a problem that was simple and obvious if you just looked at the log.
In case you skipped the introduction, this is a last reminder that the fireworks start with Distributed Computing, and you can go there now, or continue to read about normal, ordinary computing in E, starting with Hello World.
We will show Hello World as both an E program and an rune script. Rune first: just type
? println("Hello World")when you hit the Enter key, you will get
Hello World ?
To run an E program, put "println("Hello World")" into a text file. Make sure the file extension is ".e" (to run Swing-based apps, use extension ".e-awt", and to use SWT-based apps, use ".e-swt"). Type
java -jar e.jar --rune textfilename.e
into a command shell (a DOS window under Windows, or a Cygwin or Unix shell, such as bash). The greeting will be printed to your display.
Now that you have started rune and run Hello World, it would be a good time to look at your trace log folder for the first time to make sure you know where to look. If you are on Windows95, the trace log will default to c:\windows\temp\etrace. The etrace folder location can always be found in the eprops.txt file in the E installation directory (under Unix, you will have created eprops.txt yourself from eprops-template.txt). You should, after launching rune, find one file in the trace folder, which is the trace log created for the rune session. It will probably be empty, unless you have gone wild experimenting with Hello World.
Here are some of the basics of the language:
# E sample
# Comment on this piece of code
def a := 3
var b := a + 2
b += 1
if (a < b) {
println("a is less than b")
} else {
println("Wow, the arithmetic logic unit in this processor is confused")
}Variable declarations in E are made with the var statement. Variables that are only assigned a value once at creation (i.e., constants, or variables declared final) are created with the def statement. In E as in Java, "+=" is shorthand for adding the righthand value to the lefthand variable.
Single-line comments have a "#" at the beginning, and terminate with the end of line. The /**...*/ comment style is used only for writing javadoc-style E comments, discussed later. link here
Assignment uses the ":=" operator. The single equal sign "=" is never legal in E, use ":=" for assignment and "==" for testing equality. The function "println" prints to the console. The "if" statement looks identical to its Java equivalent, but the braces are required.
What is the end-of-statement delineator in E? In Java, you terminate a statement with a semi-colon. In E, the end-of-line is also the end-of-statement unless there is an open operation at the end of the line. In the example, the "if" statement's first line ends with an open brace; E knows the next line must be a continuation line. Some quick examples
This works ? def a := 1 + 2 + 3 + 4 # value: 10 And this works ? def b := 1 + 2 + > 3 + 4 # value: 10 But this does not work, because the first line can and does evaluate without a continuation, so the second line is a syntax error: ? def c := 1 + 2 ? + 3 + 4 # example syntax error: # + 3 + 4 # ^
The end-of-line statement termination of E makes it easy to use E for command lines, as in rune.
If your organization uses a line continuation convention that puts the operator at the beginning of the second line, you can use the explicit continuation character "\" as here:
This also works ? def c := 1 + 2 \ > + 3 + 4 # value: 10
The basic types in E are int, float, string, char, and boolean. All integer arithmetic is unlimited precision, as if all integers were BigIntegers.
Floats are represented as 64-bit IEEE floating point numbers. The operators +, -, * have their traditional meanings for integers and floats. The normal division operator "/" always gives you a floating point result. The floor divide operator "//" always gives you an integer, truncated towards negative infinity. So (-3.5 // 1) == -4.
E has 2 modulo operations: "%", like the Java modulo operator, returns the remainder of division that truncates towards zero. E also supplies "%%", which returns a remainder of division that truncates towards negative infinity.
Operator precedence is generally the same as in Java. In a few cases E will throw a syntax error and require the use of parentheses.
"+" when used with strings is a concatenation operator as in Java. It automatically coerces other types on the right-hand if the left-hand operand is a string:
# E sample def x := 3 def printString := "Value of x is: " + x
You can also use quasi-literals, which enable the easy processing of complex strings as described in detail later; here is a very simple example:
# E sample def printString := `Value of x is: $x`
wherein the back-ticks denote a quasi-literal, and the dollar sign denotes a variable whose value is to be embedded in the string.
&& and || and ! have their traditional meanings for booleans; true and false are boolean constants.
Strings are enclosed in double quotes. Characters are enclosed in single quotes, and the backslash acts as an escape character as in Java: '\n' is the newline character, and '\\' is the backslash character. Strings in E are so similar to strings in Java, it is easy to conclude they are identical when they are not. The protocol for E strings is detailed in the E javadoc.
== and != are the boolean tests for equality and inequality respectively. When the equality test is used between appropriately designated link to selfless here transparent immutables, such as integers, the values are compared to see if the values are equal; for other objects the references are compared to see if both the left and right sides of the operator refer to the same object. Chars, booleans, integers, and floating point numbers are all compared by value, as in Java. In addition, Strings, ConstLists, and ConstMaps link here are also compared by value, which makes it different from, and more natural than, Java.
Other transparent immutables (notably ConstLists and ConstMaps) will be introduced later. Additional useful features of transparent immutables are discussed under Distributed Computing.
There are some special rules about the behavior of the basic operators because of E's distributed security. These rules are described in the Under the Covers section later in this chapter.
We have already seen the if/then/else structure. Other traditional structures include:
One structure that is more powerful than its Java counterpart is the for loop.
# E sample
for i in 1..3 {
println(i)
}
for j in ["a", 1, true] {println(j)}
In this simple example, i becomes 1, 2, 3 in succession. In the second, j becomes each of the elements of the list.
The for loop operates not only with number ranges, but also with lists, maps (i.e. hashtables), text files, directories, and other structures discussed later in this book. The expanded version of the for loop that is needed to get both keys and values out of maps is:
# E syntax
for key => value in theMap {
println(`Key: $key Value: $value`)
}
# You can get the index and the value from a list at the same time the same way
for i => each in ["a", "b"] {
println(`Index: $i Value: $each`)
}
You can create your own data structures over which the for loop can iterate. An example of such a structure, and a brief explanation of the iterate(function) method you need to implement, can be found in the Library Packages: emakers section later in this chapter, where we build a simple queue object.
Flow control structures actually return values. For example, the if-else returns the last value in the executed clause:
# E sample
def a := 3
def b := 4
def max := if (a > b) {a} else {b}This behavior is most useful when used with the when-catch construct described in the chapter on Distributed Computing.
The break statement, when used in a for or a while loop, can be followed by an expression, in which case the loop returns the value of that expression.
(Note: the following patch of code is used by updoc.e, the E testing tool, to enable execution of all the upcoming code that depends on Swing)
?? in new vat awtVat.e-awt
? pragma.syntax("0.9")
As noted earlier, if the file is to use the Swing gui toolkit, it must have a suffix of ".e-awt". If the file is to use the SWT gui toolkit, it must have a suffix of ".e-swt". If the file will run headless, it should be placed in a file with suffix ".e".
A basic function looks like this:
# E sample
def addNumbers(a,b) {
return a + b
}
# Now use the function
def answer := addNumbers(3,4)You can nest the definitions of functions and objects inside other functions and objects, giving you functionality comparable to that of inner classes in Java. Nested functions and objects play a crucial role in E, notably in the construction of objects as described shortly.
A parameterless function must still have an open/close paren pair. Calls to parameterless functions must also include the parens.
Functions can of course call themselves recursively, as in
# E sample
def factorial(n) {
if (n == 0) {
return 1
} else {
return n * factorial(n-1)
}
}
E guards perform many of the functions usually thought of as type checking, though they are so flexible, they also work as concise assertions. Guards can be placed on variables, parameters, and return values.
Guards are not checked during compilation. They are checked during execution, and will throw exceptions if the value cannot be coerced to pass the guard. Guards play a key role in protecting the security properties when working with untrusted code, as discussed in Secure Distributed Computing.
The available guards include the items below. Some of them are typical types (String, int). Others are used most often in distributed programming, and are explained later in the book. A detailed explanation of all the things you can do with guards is postponed to the Additional Features chapter.put in link
import weakpointermaker makevat 2 args 1)kind of vat, string"headless, awt, swt", 2)name of vat for debugging import seedvatauthor authorize it to get a seedvat function with uri getter seedvat takes 2 args: 1)vat 2)string that is contents of a dot-e file except all evaluated at once. seedvat function returns a promise for the result of evaluating that expression. since it evaluates in foreighn vat, it is a far reference.typical pattern: only do importing and construction in the expression string
Here are some quick examples of guards being declared:
# E sample
# guarding a variable
def title :String := "abc"
# guarding a parameter, and a return value. Note the guard on the
# return value is part of the def line for the function.
def reciprocal(x :float64) :float64 {
return 1 / x
}
Different people use different strategies about how much type checking/guard information to include in their programs. In this book, the general style is to use guards sparingly, as might be typical in a rapid prototyping environment. One place where guards must be used with rigor is in the objects on the boundaries between trust realms, in security aware applications; see the Powerbox pattern in the Secure Distributed Programming section for an important example. link here
Objects, and object constructors, look considerably different in E than in Java or C++. We will start our exploration of objects with a simple singleton object.
Objects, functions, and variables are defined with the keyword "def"; all of these can be passed as arguments in parameter lists. Methods on an object, in contrast, are defined with the keyword "to":
# E sample
def origin {
to getX() {return 0}
to getY() {return 0}
}
# Now invoke the methods
def myY := origin.getY()
Like functions, methods require a parenthesis pair even if there are no arguments. (But, Python programmers beware, methods are not functions. Methods are just the public hooks to the object that receive messages; functions are standalone objects).
When invoking the method, the object name and the method called are separated by a dot.
The "class" concept in Java is used to achieve multiple goals. In E, these goals are factored out in a different way. For example, Java classes supply a place to put constructors, which have a special syntax unique to constructors. In E, objects are constructed by ordinary functions.
# E sample
# Point constructor
def makePoint(x,y) {
def point {
to getX() {return x}
to getY() {return y}
to makeOffsetPoint(offsetX, offsetY) {
return makePoint(x + offsetX, y + offsetY)
}
to makeOffsetPoint(offset) {
return makePoint(x + offset, y + offset)
}
}
return point
}
# Create a point
def origin := makePoint(0,0)
# get the y value of the origin
def y := origin.getY()Inside the function makePoint, we define a point and return it. As demonstrated by the makeOffsetPoint method, the function (makePoint) can be referenced from within its own body. Also note that you can overload method names (two versions of makeOffsetPoint) as long as they can be distinguished by the number of parameters they take.
The (x, y) passed into the function are not ephemeral parameters that go out of existence when the function exits. Rather, they are true variables (implicitly declared with "def" ), and they persist as long as any of the objects that use them persist. Since the point uses these variables, x and y will exist as long as the point exists. This saves us the often tedious business in Java of copying the arguments from the parameter list into instance variables: x and y already are instance variables.
We refer to an object-making function such as makePoint as a "Maker". Let us look at a more serious example, with additional instance variables:
# E sample
def makeCar(var name) {
var x := 0
var y := 0
def car {
to moveTo(newX,newY) {
x := newX
y := newY
}
to getX() {return x}
to getY() {return y}
to setName(newName) {name := newName}
to getName() {return name}
}
return car
}
# Now use the makeCar function to make a car, which we will move and print
def sportsCar := makeCar("Ferrari")
sportsCar.moveTo(10,20)
println(`The car ${sportsCar.getName()} is at X location ${sportsCar.getX()}`)
Inside the Maker, we create the instance variables for the object being made (x and y in this example), then we create the object (car). Note that the variable "name", passed into the function, is explicitly declared with "var", so that it can be altered later; in this case, it is reassigned in the setName() method.
Sometimes in the body of an object you wish to refer to the object itself. A keyword like "this" is not required. The name given to the object is in scope in the body of the object, so just use it:
# E sample
def makeCar(name) {
var x := 0
var y := 0
def car {
to moveDistance(newX,newY) {car.moveTo(x + newX, y + newY)}
# ....define other methods including moveTo as above ....
}
return car
}What if you need to reference the object during the object's construction, i.e., during the creation of the instance variables that precedes the definition of the object itself? In the below example, we give the car a weatherReportRadio that is supposed to alert the car to changing weather conditions. This radio requires, as a parameter during its construction, the car it will be alerting. So the radio needs the car during radio construction, and the car needs the radio during car construction.
# E sample
def makeRadio(car) {
# define radios
}
def makeCar(name) {
var x := 0
var y := 0
# using def with no assignment
def car
def myWeatherRadio := makeRadio(car)
bind car {
to receiveWeatherAlert(myWeatherRadio) {
# ....process the weather report....
}
to getX() {return x}
to getY() {return y}
# ....list the rest of the car methods....
}
return car
}Here, we do a "def" of the car with no assignment, then we use the car, then we do a "bind" of the car which binds a value to the car. This looks and behaves like a "forward reference" from C. Under the covers, the statement "def car" creates a promise for the car, and the bind statement fulfills the promise. We discuss promises in greater depth in the Distributed Computing chapter, where they play a key role.
Before we can talk about multiple constructors and the static-method-like behavior in E, it is time to reveal the truth about E functions. They are in fact simple objects with a single method, the "run" method, that is invoked by default if no other method is explicitly designated. For example, the following square function
# E sample
def square(x) {return x*x}is really syntactic shorthand for
# E sample
def square {
to run(x) {return x*x}
}In the second one, the run() method is explicitly defined. Using this explicit form in a Maker function, you can define multiple constructors, discriminated by the number of parameters they receive. Similarly, by adding methods to the Maker other than "run" methods, you get other "static methods". In the example below, we have a queue Maker with 2 constructors and a non-constructor method. The one-argument constructor requires an initial capacity; the no-argument constructor supplies an initial capacity of 10.
# E sample
def makeQueue {
to run(initialCapacity) {
# ....create a queue object with the specified initial capacity....
}
to run() {return makeQueue(makeQueue.getDEFAULT_CAPACITY())}
to getDEFAULT_CAPACITY() {return 10}
}
# Now use both constructors
def queue1 := makeQueue(100)
def queue2 := makeQueue()
println(`default capacity is: ${makeQueue.getDEFAULT_CAPACITY()}`)
Note also that one can use methods such as getDEFAULT_CAPACITY() to achieve the same effect as Java achieves with public static final variables.
E enables polymorphism through method name matching. In this example, we move both a car and an airplane to a new location using a single common piece of code:
# E sample
def makeCar(name) {
def car {
to moveTo(x, y) {
# move the car
}
# other methods
}
return car
}
def makeJet(name) {
def jet {
to moveTo(x, y) {
# move the jet, very different code from the code for car
}
# other methods
}
return jet
}
def vehicles := [makeCar("car"), makeJet("jet")]
for each in vehicles {
each.moveTo(3,4)
} An object can refer calls to itself to another object (a "super-object", built by the "super-constructor", faintly similar to the way one uses superclasses in Java) using the extends keyword: think of better example
? def makePoint(x, y) {
> def point {
> to getX() {return x}
> to getY() {return y}
> to getMagnitude() {return (x*x + y*y)**0.5}
> }
> return point
> }
# value: <makePoint>
? def makePoint3D(x,y,z) {
> def point3D extends makePoint(x,y) {
> to getZ() {return z}
> to getMagnitude() {
> def xy := super.getMagnitude()
> return (z*z + xy*xy)**0.5
> }
> }
> return point3D
> }
# value: <makePoint3D>
? def place := makePoint3D(1,2,3)
# value: <point3D>
? def x := place.getX()
# value: 1
? def z := place.getZ()
# value: 3
Here makePoint3D acts as an extension of makePoint. Inside the extends clause, we run the makePoint constructor to create the Point upon which the Point3D will be built. This specially constructed point is referred to in the point3D definition as "super"; you can see super being used in the getMagnitude method.
Using extends as described above follows the delegation pattern. Delegation is a simple pattern of object composition in which an object says, "if another object calls me with a method that I don't have defined here, just route the message to this other object, and send the result back to the caller." Delegation is similar to inheritance, but it is conceptually simpler.
Some experts consider inheritance to be a dangerous feature. A discussion of the pros and cons of inheritance is beyond the scope of this book. However, none of the full-blown examples in this book actually use full inheritance. Delegation, via the simple extends keyword plus polymorphism serve most of the purposes for which inheritance was intended, and is used everywhere here.
Having said that, there are times and places where full inheritance is the right answer. The extends keyword is also used, with one additional convention, to support such full inheritance. In inheritance, the "self" to which a super-object refers is the sub-object of which it is a part, so that the super-object can use methods in the sub-object:
# E sample
def makeVehicle(self) {
def vehicle {
to milesTillEmpty() {
return self.milesPerGallon() * self.getFuelRemaining()
}
}
return vehicle
}
def makeCar() {
var fuelRemaining := 20
def car extends makeVehicle(car) {
to milesPerGallon() {return 19}
to getFuelRemaining() {return fuelRemaining}
}
return car
}
def makeJet() {
var fuelRemaining := 2000
def jet extends makeVehicle(jet) {
to milesPerGallon() {return 2}
to getFuelRemaining() {return fuelRemaining}
}
return jet
}
def car := makeCar()
println(`The car can go ${car.milesTillEmpty()} miles.`)
As seen in the example, the super-object constructor specifies a parameter, "self", which can be used to refer to the sub-object. The sub-object includes itself in the the call to the super-object constructor, thus becoming "self".
Delegation with extends does a straight pass-through of the method call. It is possible to intercept collections of calls before delegating them, using match. In this example, we count the number of calls made to a car:
# E sample
def makeCalledCountCar(name) {
def myCar := makeCar(name)
var myCallCount := 0
def car {
to getCallCount() {return myCallCount}
match [verb,args] {
myCallCount += 1
E.call(myCar,verb,args)
}
}
return car
} In "match[verb,args]", the verb is a string specifying the name of the method being called, and the args is a list of arguments. If the car is called with a method that is not defined in a "to" statement, the name of the method and the arguments are bound to the variables inside the square brackets. We then increment the call count, and finally forward the message using the E.call function. E.call(...) takes as arguments the target object for the action, the name of the method to call, and the list of arguments for the method to invoke. You can manually use E.call() like this:
# E syntax
def car := makeCar("mercedes")
E.call(car, "moveTo", [3,4])
# which is equivalent to
car.moveTo(3,4)In this example, the elements for the argument list are placed in square brackets to form an E ConstList, described later in this chapter.
The match construct is useful for making "adapters" when interfacing to Java. Here we build a singleton object that can be used anywhere a java.awt.event.MouseListener would be used; the match clause is used to absorb and discard events in which we are not interested:
# E sample
def mouseListener {
to mouseClicked(event) {
# process the event
}
match [verb,args] {}
}This general purpose matching is also useful in security applications when building facets and revocable capabilities, as described in the chapter on Secure Distributed Computing.
E supports the construction of javadoc-style comments that can be postprocessed into HTML documentation. For this purpose, the "/** ... */" comment format has been specially reserved; comments of this style can only appear in front of a function/object definition (a "def" statement), or in front of an object method definition (a "to" statement):
# E sample
/**
* Add 2 numbers together.
* <p>
* Currently works with any objects that support the "plus" method
*
* @param a the first number.
* @param b the second number
* @return the result of adding.
*/
def adder(a, b) {return a + b}For more information on how to generate Edoc HTML from your E programs, see the Advanced Topics.make link here
We have already discussed the fact that functions are really objects with a single method, "run". Functions-as-objects have one practical consequence that would be surprising in the absence of understanding. You cannot create two functions with different numbers of parameters and have them overloaded:
# E syntax
def times2(a) {return a * 2}
def compute(a,b) {
def times2(c,d) {return (c + d) * 2}
# ....do computation...
# The following line will throw an exception
def answer := times2(a)
return answer
}
This would throw an exception because the inner definition of the function-like object times2 completely shadows the outer times2.
Not only functions, but also built-in constants like integers and floats, are objects. Operators such as "+" are really shorthands for message passing: 3.add(4) is identical to 3 + 4. This is why the operation
"The number is " + 1
works but
1 + " is the number"
does not. A string knows how to handle a concatenate message with a number as the argument, but an integer is clueless what to do with an add of a string.
Since "+" is really just shorthand for "add", you can construct objects that work with the "+" operator just by implementing an "add(argument)" method in the object.
There are a number of messages to which all normal objects respond, known as Miranda Methods. One example of such a method is the printOn(textWriter) method, which defines how an object will write itself out as by default. Other Miranda methods will be described in other sections of the book. The full list of Miranda methods can be found in the Appendix. You can also see them in rune by typing help(def obj{}) duplicate of next section, fix
One important note about Miranda methods: they are not forwarded by the extends structure, and they are not intercepted by the match[verb,args] structure. They are always directly interpreted by the object that receives them.
Another often convenient way of finding the list of methods associated with an object, be it a Java object or an E object or an object delivered as part of the E runtime, is to use the help(object) function of E, often used in rune.
? def add(a, b) {
> return a + b
> }
# value: <add>
? help(add)
# value: an org.erights.e.elang.evm.EImplByProxy
# interface "__main$add" {
#
# to run(:any, :any) :any
# }
#
?
In this very simple example you see the "run" method listed here, which is the implicit method for a function. The rest of the methods shown are Miranda methods. For a more sophisticated object than a simple add function, each method in the object would be listed as well.
E file objects are created with the <file:name> expression (which produces a "tamed" Java File Object, i.e., a File object that follows capability security discipline). If you can hard-code the name, you usually do not need to use quotes (unless there is a space or another invalid url character). If the name is contained in a variable or is retrieved with a function call, you must XXX place a space between the colon and the representation of the name. If the word "file" is replaced with a single letter like "c", the letter is assumed to be the drive letter on a Windows machine:
# E sample #File objects for hardwired files: def file1 := <file:myFile.txt> def file2 := <file:/home/marcs/myFile.txt> #Using a variable for a file name: def filePath := "c:\\docs\\myFile.txt" def file3 := <file>[filePath] #Using a single character to specify a Windows drive def file4 := <c:/docs/myFile.txt> def file5 := <c:\docs\myFile.txt>
Note that in the file4 example, we used slashes, not backslashes, to separate directories on a Windows drive. In E, the slash always works as a directory separator no matter the underlying operating system.
When constructing the filePath string we used double backslashes because in strings the backslash must be escaped by a backslash. A double backslash was not required in the hardwired file (file5).
File objects can represent directories as well as simple files. Files inside the directory can be accessed by indexing using square brackets:
# E syntax def dir := <file:/home/marcs/dataDir/> def file6 := dir["myFile.txt"]
Both text files and directories work in conjunction with the "for" loop. With a directory, the loop iterates through the files in the directory. With a text file, the loop iterates through the lines of the file; each line-object is terminated with a newline ('\n') character. The lines in the loop are terminated with newline regardless of the underlying operating system, regardless of the operating system's end-of-line designation.
Files respond to the following messages (see the Edoc for a complete list).
stdout and stderr are both TextWriters in the E environment.
write the example
In the next section, we will learn how to interface to Java, which will allow you to use most of the data structures supplied by the underlying jvm. E does offer a few data structures of its own, however, which are not only convenient, but which have special properties useful in distributed programming.
There are three built-in data structures in E, Lists, Maps, and Sets. Each of these structures comes in two flavors, Flex (editable) and Const (immutable). The ConstList is the simplest:
# E sample
def list := ["a",2,true]
#This is true: list[0] == "a"
#This is true: list[2] == true
def newList := list + ["last"]
#This is true: newList[3] == "last"
for each in newList {println(`Element: $each`)}
for i => each in newList {println(`Element number $i is $each`)}
def listSize := newList.size()
# get subrange starting at 0, running up to element 2, but not including element 2
def subrange := list(0,2)
#This is true: subrange == ["a",2]Lists are numerically indexed starting from 0, use the "+" operator to create a new list which is the concatenation of lists. Lists work with the for loop. The "size" method returns the number of elements. The E String type is a kind of ConstList in which all the elements are characters. So, for example, the mechanism for taking a subrange shown above gives you a substring if the ConstList is a String.
You can make a FlexList from a ConstList with the diverge method:
# E sample
def newList := []
def editable := newList.diverge()
editable.push(100)
if (editable.pop() == 100) {
println("Yeah, push and pop make a list work like a stack")
}
editable[0] := "b"
def frozen := editable.snapshot()
All the methods that work on a ConstList also work on a FlexList. In addition, there are editing operations for FlexLists, including push/pop (which allows the FlexList to behave like a stack) and element replacement that uses Java array-style syntax. You can get a ConstList from a FlexList with the snapshot method. Just as Java Strings are like ConstLists in which all elements are characters, Java StringBuffers are like FlexLists in which all elements are characters.
Maps are composed of key/value pairs. In the following example, in the ConstMap table, "a" is a key, and 1 is a value:
# E sample
def table := ["a" => 1, 2 => "b"]
# This is true: table["a"] == 1
# This is true: table[2] == "b"
def emptyTable := [].asMap()
def keyTable := ["a",1,true].asKeys()
# This is true: keyTable[true] == null
# This is true: keyTable.maps("a") == true
for key => value in table {println(`Key: $key Value: $value`)}
for each in table {println(`Value: $each`)}
def mapSize := table.size()Elements in a map are retrieved using an array-like syntax, using the key as the index in the square brackets. Maps can be made from lists using asMap and asKeys; when you use asKeys, the values are nulls. The for loop works nicely with the map structures.
FlexMaps can be made from ConstMaps with the diverge method:
# E sample def editableMap := table.diverge() editableMap["c"] := 3 editableMap["a"] := "replacement" editableMap.removeKey(2) def frozenMap := editableMap.snapshot()
Values in a FlexMap can be added and replaced with array-like syntax. Key-value pairs can be removed with removeKey(key). The snapshot method creates a ConstMap from a FlexMap.
Sets are quite similar, with ["a","b"].asSet() producing a ConstSet with 2 elements. Documentation for all the operations on Const and Flex lists, maps, and sets can be reached directly from the Javadoc For E index.
A String is actually a ConstList of characters. You can create a FlexList of elements of a specific type by specifying the type in the diverge/snapshot method:
# E sample def aString := [].diverge(char)
This aString will not accept elements which are not of type char; the variable aString has many of the characteristics of a StringBuffer from Java.
One feature of E that Java/C programmers will find surprising but useful is the ability to define multiple variables by using a list:
# E sample def [a,b] := [2,3] #a holds the value 2 #b holds the value 3
While this is an amusing way to initialize a random bunch of variables at once, the structure is most valuable when creating groups of variables that have no meaning in the absence of one another. The method Ref.promise(), described in Distributed Computing, is an important user of this pattern.
One of the biggest differences between Flex and Const structures is the test for equality. Const structures are compared by value, and as such are considered equal if their contents are identical. The test for equality is, in other words, the same as the test for equality between Java integers.
Two Flex structures are considered equal if and only if they are the same structure, i.e., both of the variables being tested are references to a single object. This test for equality is, in other words, the same as the test for equality between Java StringBuffers.
We can import a class from the underlying Java virtual machine with the import statement, and speak to the Java object much as we would speak to an E object: replace vector as example
# E sample
#create a single vector with a direct call to Java
def myVector := <unsafe:java.util.makeVector>()
myVector.addElement("abc")
#create a makeVector function which can be called repeatedly to make more vectors
def makeVector := <unsafe:java.util.makeVector>
def vector2 := makeVector()
# create a shorthand for the java.util package, that gives quick access to all the
# classes in java.util
def <util> := <unsafe:java.util.*>
def vector3 := <util:makeVector>()
In the example, we showed 3 ways of getting a Java object, which are roughly comparable to the different styles of using "import" in Java. If you just want one vector, you can get it by directly calling the vector constructor. If you plan to make many vectors, you can import the class into a variable. And if you plan to use many classes in a particular package, you can create your own representative for the package that you can use instead of "unsafe:". fixNote the suffix "__uriGetter" on the util variable. This is a special suffix that allows the variable to be used in statements of the form <uri:name>.
We have now seen 3 of these uriGetters, file:, unsafe:, and the programmer-defined util:. Four other uriGetters are also predefined, swing: (for accessing javax.swing) and awt: (for accessing java.awt), resource: (for read-only access to resource files such as images, sounds, and static texts), and import:, described next.
The import: uriGetter is similar to unsafe:, except it only imports those parts of the Java API that have been audited for capability security and found to convey no authority (see the Appendix for a definition of "authority").put in link, think of better example. Remove vector from the book Since the Vector class conveys no authority, you could also get a vector with import:
def vector4 := <unsafe:java.util.makeVector>()
As discussed later with Library Packages and Secure Mobile Code, emakers cannot use unsafe: unless the unsafe__uriGetter is explicitly handed to them. So import: is often useful in these security-aware situations. A complete list of safe versus unsafe classes, and the small percentage of Java API methods which are suppressed for security reasons, are listed in the Appendix.link
resource: mention it is safe, can be used in emakers
getting static public constants from Java: must put parens after the name as if it was a method returning a value, not a variable.
As noted earlier, E does not have an expression to directly represent public static final variables. To use such static finals from a Java class, put parentheses at the end of the variable name, making it syntactically look like a function call, prefix it with "get", uppercase the first letter, and E will get the value for you:
# E sample <unsafe:java.util.makeCalendar>.getYEAR()
Speaking to Java objects is usually as easy as shown above, where we simply sent myVector an addElement(object) message as if it were an E object. A problem arises with some Java objects if they have overloaded methods for which the only distinguishing mark is the static typing of the parameters in which one of the methods specifies a type that is a superclass of the type in the other method--a rare but not unheard-of situation. coercion causes same problemIn that case, we need to revert to a more descriptive messaging convention.
We introduced the E.call() function earlier for communicating with E objects. It can also be used to call Java objects and to specify the signature of an overloaded method with the types of the parameters. In this example, we see that the print function for the System.out object is overloaded for both String and Object, with String being a subclass of Object, so the full signature is required:
? def out := <unsafe:java.lang.makeSystem>.getOut()
# value: <a PrintStream>
? out.print("abc")
# example problem: <IllegalArgumentException: ...>
? E.call(out, "print(String)", ["abc"])
? def myLabel := E.call(<swing:makeJLabel>, "run(String)",["label text"])
If you encounter one of these special cases that requires multiple parameters, the signatures must be laid out with exact spelling and exact spacing, notably with exactly one space between the comma and the next signature:
E.call(javaObject, "method(String, int)", ["abc", 23])
It is possible to make Java arrays of Java primitive types, as in this example:
# E sample def makeFlexList := <elib:tables.makeFlexList> def byteArray := makeFlexList.fromType(<type:byte>, 300) byteArray.setSize(300) byteArray[0] := 3
Here we have created a 300-element editable array of bytes. The individual elements can be accessed with square bracket notation.
XXX Now that this example has been corrected, does it still serve its purpose? Also, it's not technically correct to say this makes a Java array. It makes an E FlexList, which, on E-on-Java, currently happens to be implemented by wrapping a Java array. However, if your program depends on this fact, it is probably broken.
As described earlier, you can use the match[verb,args] construct to aid in writing adapters and other objects that must meet a Java interface specification. E will automatically manipulate the resulting E objects and the java virtual machine so that Java understands that the E object meets the interface specification. It is not possible to make subclasses of Java classes in E. However, because the developers of the Java API have wisely moved aggressively to make sure that interfaces, rather than subclasses, are core to Java object interoperability, there is virtually no Java interfacing goal that cannot be achieved with this machinery.
A final note: almost all of the objects and methods in the Java API are available in E for version 1.0 through the unsafe_uriGetter; unfortunately very few have been audited sufficiently to be included with the import__uriGetter. In the future, more of the API will be accessible through the import__uriGetter; however, some parts of the Java API will be altered and/or suppressed even in unsafe__uriGetter to achieve a more secure yet easier to use computing framework. This is discussed further in the chapter on Mobile Code. You can see the list of suppressed methods and unsafe classes in the Javadoc.
E supports quasi-parsers. A quasi-parser allows one to compress large numbers of operations into a succinct notation (a quasi-literal) in a specific problem domain. Writing your own quasi-parsers (which can be done in E itself, as the JPanel quasiparser described below was written) is beyond the scope of this book. However, E comes with several quasi-parsers built in: a simple quasi-parser, a regular expression quasi-parser, a JPanel quasi-parser for Swing, and a swtGrid quasi-parser for SWT.
The default quasi-parser is a text manipulating parser capable of extracting data from strings and constructing new strings. In its simplest form, it is a clean and simple way of constructing strings for printing:
# E sample
def a := 3
def answerString := `The value a is $a, and a times two is ${2 * a}.`
println(answerString)Here we use a simple quasi-parser to build an output string in a fashion similar to the printf statement in C. Quasi literals are enclosed in back-ticks. A dollar sign denotes the beginning of a description of a value to be inserted at the dollar sign location. If the dollar sign is immediately followed by a variable name, the value of that variable is used. If the dollar sign is followed by an open brace, everything up to the close brace is evaluated to compute the value to be substituted (so you could put "$" inside the braces to put a dollar sign in the string, as well as doing arithmetic as in the above example). Quasi-literals can span multiple lines, in which case the carriage return is part of the structure.
A more sophisticated use of simple quasi-literals is for pattern matching. Here we parse a sentence:
# E sample
def line := "By the rude bridge that arched the flood"
if (line =~ `@word1 @{word2} rude @remainder`) {
println(`text matches, word1 = $word1`)
}The string on the left of =~ is compared to the quasi literal on the right, evaluating true if the string can be parsed in conformance with the quasi literal. The "@" asserts that any text can match this part of the string, and the variable declared after the "@" contains that text at the end of the evaluation. The variable "word2" is enclosed in braces to offset it from the word "rude" immediately following it, which would look like part of the variable name without the offset.
In this example, the minimal string that can get a match would be space-space-"rude ", in which case the data extracted for variables word1, word2, and remainder would all be zero-length strings. As it is, at the end of the evaluation, word1=="By", word2=="the", and remainder == "bridge that arched the flood".
A single quasi-literal can contain dollar signs as well as "@"'s, in which case the results of the dollar sign evaluations will be included in the matching conditions.
Quasi-literals can almost always be treated as strings. They accept almost all of the string protocol (technically, the quasi-literals are of type "twine"). The one place where they cannot be treated as strings is in comparisons to actual strings. To compare a quasi-literal to a string, use the "bare" method: this is now wrong! all strings seem to now be twines
def equalStrings := "abc".bare() == `abc`.bare()
The regular expression quasi-parser gives E much of the CGI scripting power that Perl and Python share. Since E runs on top of a jvm with all the startup time such jvms entail, using E for CGI scripts per se is not recommended. However, if one uses the distributed computing power of E to run CGI-like E programs as services for a Web server, one can achieve the same effect, and receive a number of bonuses unavailable in Perl and Python. The example Web server written in E, shown at the end of the book, was designed with just this thought in mind.
The JPanel quasi-parser processes visually understandable strings into complex window panel layouts for gui applications. It is a most remarkable and useful example of quasi-parsers in action, giving the developer a rather WYSIWYG presentation of his windows. Thus the E programmer has no need to resort to the typical inflexible IDE-specific drawing tools that produce code no one can read and absolutely no one can edit. Under the covers, the JPanel uses the GridBagLayout manager to compose the panel, giving it a flexibility comparable to the TK layout system from TCL (which actually inspired the JPanel). Unlike the typical visual layout editors in Java IDEs, the JPanel system is able to define a broad range of resizable windows simply and intuitively.
# E syntax # define the panels explanation,label, field, okButton, cancelButton, logo # pad1, pad2, which are label fields before # this example begins def composedPanel := JPanel`$explanation.Y > > $label $field > $okButton $cancelButton $pad1.X V $pad2 $logo`
In this layout, the explanation (presumably a textArea) is at the top of the composedPanel, with the label and field left-to-right underneath, and the okButton to the left of the cancelButton/logo area arranged top-to-bottom. This is a layout for a 3-wide, 4-high grid of cells, though some the panes fill multiple cells, and the rules for which cells grow are sophisticated, as described next:
The ".Y" says that the explanation should soak up any extra vertical space. The ".X" says the field should soak up any extra horizontal space. The two ">" symbols say that the explanation should span all three of the columns of this pane. The field fills two horizontal cells as denoted by the ">" which follows it. The "V" in the lower lefthand corner says that the okButton should fill two vertical cells.
When this pane is part of a resizable window, enlarging vertically makes the explanation larger. Enlarging the window horizontally enlarges the field but not the label. Both the cancel button and the okButton should remain the same size regardless of resizing since the extra space is being soaked up elsewhere.
If several elements have ".Y", the extra vertical space is divided evenly among them; similarly fields with ".X" share extra horizontal space.
The space characters used to separate the elements of this layout have no meaning to the quasi-parser; we have used the space to create a visual representation of the layout that makes it easy to see the layout even though this is just code.
It is not possible to lay out all possible compositions with a single JPanel, but JPanels can dramatically reduce the nesting of panels compared to Java applications, while making the layout visually clear in the code itself. And of course, you can always place JPanel-laid-out panels inside of other JPanel layouts. The result is tremendously more compact, easier to understand, and easier to maintain than the result of nesting large numbers of Swing Box layouts.
A similar quasi-parser, the swtGrid, is included for layout of SWT panels. The main difference, as shown in later example code, is that the swtGrid requires a first entry that is the parent panel for the laid out components improve this discussion
The E library package system is based on emakers. An emaker is an E source file named with ".emaker" as the suffix.
When an emaker is imported into an application, the code is executed at the time of import, and the result of that execution is returned to the importing program.
# E sample
def makeQueue() {
var head := null
var tail := null
def makeCarrier(obj) :near {
var next := null
def carrier {
to attach(nextCarrier) {next := nextCarrier}
to get() {return obj}
to next() {return next}
}
}
def queue {
to push(obj) {
if (head == null) {
head := makeCarrier(obj)
tail := head
} else {
tail.attach(makeCarrier(obj))
tail := tail.next()
}
}
to pop() {
def current := head
head := head.next()
return current.get()
}
to iterate(func) {
var current := head
var i := 0
while (current != null) {
func(i, current.get())
current := current.next()
i += 1
}
}
}
return queue
}
Side Note: This queue does have one interesting characteristic: the iterate method is the feature required to work correctly with E's for-loop. Iterate receives a function that expects 2 arguments: a key and an object. In this queue, the key value is filled with the index of the value in the queue. Iterate walks the entire data structure, invoking the function once for each element.
Place this queuemaker in a file named makeQueue.emaker, and put the file in the classpath, either with the -cp command line argument to Java, or by placing the file in a subdirectory under lib/ext in the java directory system. You can use Java-style package specifications: as in Java, you can put the file in a jar in lib/ext. In the below example, we place the file under lib/ext/emakers/com/yourdomain/e/queueMaker.emaker (in your jre directory), and import:
# E syntax
def makeQueue := <import:com.yourdomain.e.makeQueue>
def queue := makeQueue()
queue.push(1)
queue.push(2)
for each in queue {println(each)}There are several ways of making E aware of the availability of an emaker, corresponding to the several ways Java can become aware of a class. You can place it in a zip or a jar and place it in the jvm's ext directory. Or you can place it in a subfolder in the jvm's ext directory. Or you can place it, or a jar that contains it, on the classpath when you start up the jvm.
Emakers have an important security property: they come to life contained in a world with no authority. Such a world is even more restrictive than the Java sandbox used for applets. However, this world is more flexible than the sandbox because authority can be granted and revoked during operation using capabilities, as described in the chapter on Secure Distributed Computing.
The makeQueue function example is a nice first example of emakers in action because makeQueue and the queues it makes need no authority. For emakers that require authority, by convention we use the Authorizer pattern. Here is an infoWindowMakerAuthor: the infoWindowMaker needs the authority to create windows, so we wrap it in an Authorizer:
# E syntax
#emaker in file lib/ext/com/skyhunter/infoWindowMakerAuthor.emaker
def infoWindowMakerAuthor(frameMaker) {
def infoWindowMaker(infoText) {
def frame := frameMaker("Information")
def label := <swing:makeJLabel>(infoText)
frame.getContentPane().add(label)
frame.pack()
frame.show()
return frame
}
return infoWindowMaker
}
#.....in using program....
def makeInfoWindow := <import:com.skyhunter.infoWindowMakerAuthor> (<swing:makeJFrame>)
def infoBox := makeInfoWindow("First Bit Of Info")
def detailedInfoBox := makeInfoWindow("Second Bit Of Info")
By convention, authors follow the pattern of functions, using an implied run() method to return the Maker that is being wrapped.
The meticulous reader may have noticed that the JFrame was a capability that needed to be authorized, whereas the JLabel was not. The vast bulk of the Java API is non-capability-transferring, and can just be acquired using import:. Commonly-used exceptions are the JFrame, everything have to do with files in java.io, and the URL object in java.net. For a complete listing of suppressed methods (which are inaccessible in E though documented in Java), safe classes (which can be imported directly by emakers), and unsafe classes (which can only be imported using the unsafe__uriGetter, which is not directly available to emakers), see the JavaDoc for E.
Emakers can also just be obsolete functions, in which case the naming convention uses "Func" to distinguish it from a Maker:
# E syntax
#emaker in file lib/ext/org/erights/computeSquareFunc.emaker
def computeSquareFunc(number) {
return number * number
}
#...in using program...
def computeSquare := <import:org.erights.computeSquareFunc>
def squareOfThree := computeSquare(3)There are also functions that require capabilities before they can become operational, in which case they are by convention wrapped in Authorizers, for example, analyzeOutlineFuncAuthor.
The good news is that, because emakers come to life without authority, they can always be imported with import:, so any emaker can bring in any other emaker (though you won't be able to run the authorizer without sufficient authority). The bad news is that, if you are writing standalone E applications in which the emakers are fully trustworthy (or at least as trustworthy as the program that calls them), it can seem a substantial nuisance at first to be granting lists of authorities just to break your program into packages. If you are writing pure E applications that use emakers of your own construction, emakers which you trust totally with the full authority of the system, and you have no qualms about flouting the wisdom of architectural purists around the world, you can take a shortcut by simply granting the unsafe__uriGetter. For example:
# E sample
def powerfulObjectMakerAuthor(unsafe__uriGetter) {
def file := <unsafe:java.io.File>("name")
def swing__uriGetter := <unsafe:javax.swing.*>
def frame := <swing:makeJFrame>("powerful object window title")
def powerfulObjectMaker() {
#....define the objectMaker and the object it makes...
}
return powerfulObjectMaker()
}
Note that in this example, one of the things we created from our unsafe__uriGetter was our own swing__uriGetter. Emakers, when they come to life, do have a swing__uriGetter of their own; however, their built-in swing: accesses only the safe subset of swing, much as import: accesses only the safe subset of unsafe:. In the example we create our own swing: that overrides the default swing:, which enables us access unsafe classes such as the JFrame.
For even more information on the security properties of emakers, see the chapter on Secure Mobile Code.
Below is a simple game of racetrack: 3 cars are set on the track, to race between walls around curves to the finish line. Each driver can choose to accelerate or decelerate his car by +/- 1 space/turn during each turn. Do not build up too much velocity, you won't be able to slow down!
The example has just about everything in it: JPanels, objects, functions, Java API calls, it's all there. We will come back to this example later in the book, to make the track distributed and secure. Then we shall invite Satan for a little competition, with souls on the line.
# E sample
#!/usr/bin/env rune
# Copyright 2002 Combex, Inc. under the terms of the MIT X license
# found at http://www.opensource.org/licenses/mit-license.html
pragma.syntax("0.9")
def traceln(text) { stderr.println(text) }
def attachAction(component,target,verb) {
def listener {
to actionPerformed(event) {
E.call(target, verb, [])
}
}
component.addActionListener(listener)
}
def newButton(labelText, verb, target) {
def button := <swing:makeJButton>(labelText)
button.setBackground(<awt:makeSystemColor>.getControl())
attachAction(button,target,verb)
return button
}
def abs(number) {return if (number >= 0) {number} else {-number}}
def makeCoord(x,y) {
def coord {
to getX() {return x}
to getY() {return y}
to printOn(writer) {writer.print(`coord: $x,$y`)}
to samePlace(coord2) :boolean {
return x == coord2.getX() && y == coord2.getY()
}
/**
* The "add" method is the underlying function for the "+" operator.
* Here, by writing an "add" method, we make coordinates work with "+"
*/
to add(coord2) {return makeCoord(x + coord2.getX(),y + coord2.getY())}
/**
* The "subtract" method is the underlying function for the "-" operator
*/
to subtract(coord2) {return makeCoord(x - coord2.getX(),y - coord2.getY())}
}
return coord
}
def makeInstrumentPanel(car) :near {
def makeIndicator(speed,positiveText,negativeText):pbc {
var answer := ""
var direction := positiveText
if (speed < 0) {direction := negativeText}
for i in 1..abs(speed) {answer := answer + direction}
if (speed == 0) {answer := "0"}
return answer
}
def makeXIndicator(speed) {return makeIndicator(speed,">","<")}
def makeYIndicator(speed) {return makeIndicator(speed,"^\n","V\n")}
def frame := <swing:makeJFrame>(`Car ${car.getName()} Instrument Panel`)
def lbl(text) {return <swing:makeJLabel>(text)}
def xLabel := lbl("Horizontal Speed:")
def xIndicator := <swing:makeJTextArea>()
xIndicator.setText("0")
def yLabel := <swing:makeJTextArea>("V \ne\nr\nt\ni\nc\na\nl")
yLabel.setBackground(<awt:makeSystemColor>.getControl())
def yIndicator := <swing:makeJTextArea>()
yIndicator.setText("0")
def statusPane := lbl("Running...")
def instrumentPanel
def btn(name,action) {return newButton(name,action,instrumentPanel)}
def submitButton := btn("Submit","submit")
var acceleration := makeCoord(0,0)
def realPane :=JPanel`
${lbl("")} $xLabel > > >
V $xIndicator > > >
$yLabel.Y $yIndicator ${btn("\\","upLeft")} ${btn("^","up")} ${btn("/","upRight")}
V V ${btn("<","left")} ${btn("0","zero")} ${btn(">","right")}
V V ${btn("/","downLeft")} ${btn("V","down")} ${btn("\\","downRight")}
V V $submitButton.X > >
$statusPane > > > >`
frame.setDefaultCloseOperation(<swing:makeWindowConstants>.getDO_NOTHING_ON_CLOSE())
frame.getContentPane().add(realPane)
frame.pack()
frame.show()
bind instrumentPanel {
to submit() {
submitButton.setEnabled(false)
car.accelerate(acceleration)
}
to prepareForNextTurn() {
xIndicator.setText(makeXIndicator(car.getVelocity().getX()))
yIndicator.setText(makeYIndicator(-(car.getVelocity().getY())))
acceleration := makeCoord(0,0)
submitButton.setEnabled(true)
# Note, public static transferFocus on awt Component is not Java API, added in E environment
<awt:makeComponent>.transferFocus([frame.getContentPane()], statusPane)
}
to setStatus(status) {statusPane.setText(status)}
to upLeft() {acceleration := makeCoord(-1,-1)}
to up() {acceleration := makeCoord(0,-1)}
to upRight() {acceleration := makeCoord(1,-1)}
to left() {acceleration := makeCoord(-1,0)}
to zero() {acceleration := makeCoord(0,0)}
to right() {acceleration := makeCoord(1,0)}
to downLeft() {acceleration := makeCoord(-1,1)}
to down() {acceleration := makeCoord(0,1)}
to downRight() {acceleration := makeCoord(1,1)}
}
}
def makeCar(name,startLocation,raceMap) {
var location := startLocation
var acceleration := makeCoord(0,0)
var velocity := makeCoord(0,0)
var hasCrashed := false
var hasFinished := false
def instrumentPanel
def sign(x) {
return if (x > 0) {
1
} else if (x < 0) {
-1
} else {0}
}
def accelReactors := [].asMap().diverge()
/**
* Compute the path the car will take from the location at the
* beginning of this turn to the end; return the result
* as a list of coords
*/
def computeIntermediateLocations(start,finish) {
def locations := [].diverge()
def slope := (finish.getY() - start.getY()) / (finish.getX() - start.getX())
def computeRemainingLocations(current) {
var nextX := current.getX()
var nextY := current.getY()
var distToGo := 0
#if the car is traveling faster in the x direction than
#in the y direction, increment x position by one along
#the path and compute the new y
if (slope < 1.0 && slope > -1.0) {
distToGo := finish.getX() - current.getX()
nextX += sign(distToGo)
def distTraveled := nextX - start.getX()
nextY := start.getY() + ((slope * distTraveled) + 0.5) //1
#if the car is traveling faster in the y direction than
#in the x direction, increment y position by one along
#the path and compute new x
} else {
distToGo := finish.getY() - current.getY()
nextY += sign(distToGo)
def distTraveled := nextY - start.getY()
nextX := start.getX() + ((distTraveled/slope) + 0.5) //1
}
def next := makeCoord(nextX,nextY)
locations.push(next)
if (!(next.samePlace(finish))) {
computeRemainingLocations(next)
}
}
computeRemainingLocations(start)
return locations
}
def car {
to accelerate(accel) {
traceln(`accelerating car $name`)
acceleration := accel
for each in accelReactors {
each.reactToAccel(car)
}
}
to move(){
traceln("into move")
velocity += acceleration
def newLocation := location + velocity
traceln("got newlocation")
def path := computeIntermediateLocations(location,newLocation)
location := newLocation
traceln("assigned location")
hasCrashed := hasCrashed || raceMap.causesCrash(path)
hasFinished := hasFinished || raceMap.causesFinish(path)
traceln("got crash finish")
if (hasCrashed) {
instrumentPanel.setStatus("Crashed")
} else if (hasFinished) {instrumentPanel.setStatus("Finished")}
traceln("out of move")
}
to getLocation() {return location}
to getVelocity() {return velocity}
to hasCrashed() {return hasCrashed}
to hasFinished() {return hasFinished}
to getName() {return name}
to prepareForNextTurn() {instrumentPanel.prepareForNextTurn()}
to addAccelReactor(reactor) {accelReactors[reactor] := reactor}
to removeAccelReactor(reactor) {accelReactors.remove(reactor)}
}
bind instrumentPanel := makeInstrumentPanel(car)
return car
}
def makeTrackViewer(initialTextMap) {
def frame := <swing:makeJFrame>("Track View")
def mapPane := <swing:makeJTextArea>(initialTextMap)
def statusPane := <swing:makeJLabel>(" ")
def realPane :=
JPanel`$mapPane.Y
$statusPane`
frame.getContentPane().add(realPane)
def windowListener {
to windowClosing(event) {
interp.continueAtTop()
}
match [verb,args] {}
}
frame.addWindowListener(windowListener)
frame.pack()
frame.show()
def trackViewer {
to refresh(textMap) {mapPane.setText(textMap)}
to showStatus(status) {statusPane.setText(status)}
}
return trackViewer
}
def makeRaceMap() {
def baseMap := [
"..........W...............",
"..........W...........FFFF",
"......W...WW..............",
"......W....W..............",
"......W....WWW............",
"......W........W..........",
"......W.....W.............",
"......W.....W.............",
"......W...................",
"......W..................."]
def isWall(coord) :boolean {return baseMap[coord.getY()] [coord.getX()] == 'W' }
def isFinish(coord) :boolean {return baseMap[coord.getY()] [coord.getX()] == 'F'}
def pointCrash(coord) :boolean {
var result := false
if (coord.getX() < 0 || coord.getY() < 0 ||
coord.getX() >= baseMap[0].size() || coord.getY() >= baseMap.size()) {
result := true
} else if (isWall(coord)) {
result := true
}
return result
}
def raceMap {
to getAsTextWithCars(cars) {
def embedCarsInLine(index,line) {
def inBounds(xLoc) :boolean {return xLoc >= 0 && xLoc < line.size()}
var result := line
for each in cars {
if (each.getLocation().getY() == index &&
inBounds(each.getLocation().getX())) {
def editable := result.diverge(char)
editable[each.getLocation().getX()] := (each.getName())[0]
result := editable.snapshot()
}
}
return result
}
var result := ""
for i => each in baseMap {
result := result + embedCarsInLine(i,each) + "\n"
}
return result
}
to causesCrash(path) :boolean {
var result := false
for each in path {
if (pointCrash(each)) {
result := true
break()
}
}
return result
}
to causesFinish(path) :boolean {
var result := false
for each in path {
if (pointCrash(each)) {
break()
} else if (isFinish(each)) {
result := true
break()
}
}
return result
}
}
return raceMap
}
/**
* create the cars, place them in a flex map to be used as a set
*/
def makeCars(raceMap) {
def carList := [
makeCar("1",makeCoord(1,9),raceMap),
makeCar("2",makeCoord(2,9),raceMap),
makeCar("3",makeCoord(3,9),raceMap)]
def carSet := [].asMap().diverge()
for each in carList {carSet[each] := each}
return carSet
}
/**
* @author Marc Stiegler
*/
def makeRaceTrack() {
def raceMap := makeRaceMap()
def cars := makeCars(raceMap)
var carsReadyToMove := [].asMap().diverge()
def mapViewer := makeTrackViewer(raceMap.getAsTextWithCars(cars))
def raceTrack {
to reactToAccel(car) {
traceln("racetrack reacting to accel")
carsReadyToMove[car] := car
if (carsReadyToMove.size() >= cars.size()) {
raceTrack.completeNextTurn()
}
}
to completeNextTurn() {
def winners := [].diverge()
for each in cars {
each.move()
if (each.hasCrashed()) {
cars.removeKey(each)
} else if (each.hasFinished()) {
winners.push(each)
}
}
mapViewer.refresh(raceMap.getAsTextWithCars(cars) )
if (winners.size() == 1) {
mapViewer.showStatus(`Car ${winners[0].getName()} has won!`)
} else if (winners.size() > 1) {
mapViewer.showStatus("It's a tie!")
} else if (cars.size() == 0) {
mapViewer.showStatus("Everyone's dead!")
} else {raceTrack.prepareForNextTurn()}
}
to prepareForNextTurn() {
traceln("into prepare for next turn")
carsReadyToMove := [].asMap().diverge()
for each in cars {
each.prepareForNextTurn()
}
}
}
for each in cars {each.addAccelReactor(raceTrack)}
return raceTrack
}
makeRaceTrack()
# In actual code, the following line would not be commented out
# interp.blockAtTop()
Multiple threads form the basis of conventional models of concurrent programming. Remarkably, human beings engage in concurrent distributed computing every day even though people are generally single threaded (there are exceptions: Walter Cronkite routinely listened to one broadcast with one ear, a different broadcast with the other, while simultaneously taking notes on another topic and speaking authoritatively to an audience of millions. But most people, including this author, live single-threaded lives).
How do we simple creatures pull off the feat of distributed computing without multiple threads? Why do we not see groups of people, deadlocked in frozen tableau around the coffepot, one person holding the sugar waiting for milk, the other holding milk waiting for sugar?
The answer is, we use a sophisticated computational mechanism known as a nonblocking promise.
Let us look at a conventional human distributed computation. Alice, the CEO of Evernet Computing, needs a new version of the budget including R&D numbers from the VP of Engineering, Bob. Alice calls Bob: "Could you get me those numbers?"
Bob jots Alice's request on his to-do list. "Sure thing, Alice, I promise I'll get them for you after I solve this engineering problem."
Bob has handed Alice a promise for the answer. He has not handed her the answer. But neither Bob nor Alice sits on their hands, blocked, waiting for the resolution.
Rather, Bob continues to work his current problem. And Alice goes to Carol, the CFO: "Carol, when Bob gets those numbers, plug 'em into the spreadsheet and give me the new budget,okay?"
Carol: "No problem." Carol writes Alice's request on her own to-do list, but does not put it either first or last in the list. Rather, she puts it in the conditional part of the list, to be done when the condition is met--in this case, when Bob fulfills his promise.
Conceptually, Alice has handed to Carol a copy of Bob's promise for numbers, and Carol has handed to Alice a promise for a new integrated spreadsheet. Once again, no one waits around, blocked. Carol ambles down the hall for a contract negotiation, Alice goes back to preparing for the IPO.
When Bob finishes his calculations, he signals that his promise has been fulfilled; when Carol receives the signal, she uses Bob's fulfilled promise to fulfill her own promise; when Carol fulfills her promise, Alice gets her spreadsheet. A sophisticated distributed computation has been completed so simply that no one realizes an advanced degree in computer science should have been required.
In this simple example, we see why human beings never get trapped in the thread-based deadlock situations described endlessly in books on concurrent programming. People don't deadlock because they live in a concurrent world managed with a promise-based architecture. And so it is with E.
This chapter on distributed computing is the longest and most challenging part of the book. The Promise Architecture of E is very different from the threads, synchronized methods, guarded objects, and RMI of Java. One might expect the Security chapter and its Capability paradigm to be equally new and challenging. But in fact, the implementation of security is embedded so deeply in E's infrastructure that much of the effort merely involves realizing the power inherent in constructs we have already learned. Security with E is more a matter of architecture than of coding.
Distributed computation, however, still requires gritty effort. We start simply, with the eventually operator.
All distributed computing in E starts with the eventually operator, "<-":
# E syntax
car <- moveTo(2,3)
println("car will eventually move to 2,3. But not yet.")The first statement can be read as, "car, eventually, please moveTo(2,3)". As soon as this eventual send has been made to the car, the program immediately moves on to the next sequential statement. The program does not wait for the car to move. Indeed, as discussed further under Game Turns below, it is guaranteed that the following statement (println in this case) will execute before the car moves, even if the car is in fact running on the same computer as this code. The request to move the car has been entered on a to-do list (actually sent to the appropriate event queue, for those already familiar with event loops).
Since the program does not wait around for the eventual send, an eventual send is very different from a traditional object-oriented method call (referred to here as an immediate call to distinguish it from an eventual send; the statement "car.moveTo(2,3)" is an immediate call). In general, you do not know, and cannot know, exactly when the car will move; indeed, if the car is on a remote computer, and the communication link is lost, the car may never move at all (and creates a broken promise that you can catch and process, described later).
This brings us to the most interesting feature of an eventual send: just as you do not know when the operation will complete, you also do not know, and do not need to know, which computer in your distributed system is executing the car's code. The car could be a local object running on the same machine with the program... or it could be on a computer a thousand miles away across the Internet. Regardless, E keeps track of the car's Universal Resource Identifier (URI) and delivers the message for you. Moreover, if the car is indeed remote, E sets up a secure communication link between the program and the car; and since the URI for the car includes an unguessable random string of characters, no one can send a message to the car or extract information from the car except someone who has explicitly and intentionally received a reference to it from someone who already has the reference. Thus a computation running on five computers scattered across five continents, all publicly accessible by the whole world of the Web, can be as secure as a computation running on a box locked in your basement.
Another very interesting feature of the eventual send: because the program continues on to the next statement immediately, without waiting for the eventual send to finish, deadlock can never occur.
Finally, note that the eventual send invoked an ordinary object method ("moveTo(x,y)") in the car. The programmer who created the makeCar constructor defines the cars to have ordinary methods, with ordinary returns of values. He does not know and does not care whether objects that invoke those methods use calls or sends, and neither knows nor cares whether those invoking objects are local or remote.
When you make an eventual send to an object (referred to hereafter simply as a send, which contrasts with a call to a local object that waits for the action to complete), even though the action may not occur for a long time, you immediately get back a promise for the result of the action:
# E syntax
def carVow := makeCar <- ("Mercedes")
carVow <- moveTo(2,3)In this example, we have sent the makeCar function the default "run" message. Eventually the carMaker will create the new car on the same computer where the carMaker resides; in the meantime, we get back a promise for the car.
Once we've got a promise, pipelining comes into the picture: We can immediately start making eventual sends to the promise just as if it were indeed the car. Enormous queues of operations can be shipped across the poor-latency comm network while waiting for the car to be created, confident that those operations will be executed as soon as possible.
But don't try to make an immediate call on the promise. Don't try it even if the car maker (and therefore the car it creates) actually live on the same computer with the program. To make immediate calls on the promised object, you must set up an action to occur when the promise resolves, using a when-catch construct:
# E syntax
def temperatureVow := carVow <- getEngineTemperature()
when (temperatureVow) -> {
println(`The temperature of the car engine is: $temperatureVow`)
} catch prob {
println(`Could not get engine temperature: $prob`)
}
println("execution of the when-catch waits for resolution of the promise,")
println("but the program moves on immediately to these printlns")
We can read the when-catch statement as, "when the promise for a temperature becomes done, and therefore the temperature is locally available, perform the main action block... but if something goes wrong, catch the problem in variable prob and perform the problem block". In this example, we have requested the engine temperature from the carVow. Eventually the carVow resolves to a car and receives the request for engine temperature; then eventually the temperatureVow resolves into an actual temperature. The when-catch construct waits for the temperature to resolve into an actual (integer) temperature, but only the when-catch construct waits (i.e., the when catch will execute later, out of order, when the promise resolves). The program itself does not wait: rather, it proceeds on, with methodical determination, to the next statement following the when-catch.
Inside the when-catch statement, we say that the promise has been resolved. A resolved promise is either fulfilled or broken. If the promise is fulfilled, the main body of the when-catch is activated. If the promise is broken, the catch clause is activated.
Notice that after temperatureVow resolves to temperature, the when clause treats temperature as a local object. This is always safe to do when using vows. A vow is a reference that always resolves to a local object. In the example, variable names suffixed with "vow" remind us and warn others that this reference is a vow. We can reliably make eventual sends on the vow and immediate calls on what the vow resolves to, but we cannot reliably make immediate calls to the vow itself. A more detailed discussion of these types of warnings can be found in the Naming Conventions section later.
Not all references can guarantee to resolve to a local object. A receiver is a reference that can resolve to a local or remote object. Since we cannot ascertain beforehand whether it is local or remote, we must treat it as a remote object, i.e., we must interact with it only using eventual sends. We can use either the suffix Rcvr on the variable name as a reminder, or we can use the rcvr guard when defining the variable as a reminder. If you use Rcvr as a suffix, you will be more reliably reminded not to use immediate calls, but it does produce long variable names.
# E syntax
def car :rcvr := makeCarRcvr <- ("Mercedes")
car <- moveTo(2,3)Under the covers, the "done(value)" part of the when-catch is actually the definition of a function. You can make up your own names for "done", such as "finished". Indeed, if a single object uses more than one when-catch, each "done" function must have a distinct name, such as done1, done2, etc.
The function nature of the "done" element of the when-catch is useful for making when-catch create promises for you, as discussed later.
When-catch, like try-catch, has an optional finally clause that always executes:
# E syntax
when (tempVow) -> {
#...use tempVow
} catch prob {
#.... report problem
} finally {
#....log event
}
If you need to resolve several references before making a computation, use a multiway when-catch, in which you specify several arguments in the structure:
# E syntax
def maxTempVow := vehicleSpecsRcvr <- getMaxEngineTemperature(carType)
def engineTempVow := carRcvr <- getEngineTemperature()
when (engineTempVow,maxTempVow) -> {
if (engineTempVow > maxTempVow) {println("Car overheating!")}
} catch e {println(`Lost car or vehicleSpecs: $e`)}
In this when-catch, since all the arguments are vows, the corresponding parameters (engineTemp and maxTemp, to the right of the "->") are simply local objects (integers in this case). How do we know that these integers will be local, not remote, even though we retrieved them from Rcvrs? Because integers are passed by construction, as discussed next.
In the above example, the temperature is an integer and is guaranteed to resolve to a local integer object upon which you can make immediate calls, even if the car is remote. Transparent immutable objects, such as integers, floating point numbers, booleans, strings, and the E data structures ConstLists and ConstMaps, are always passed by copy(which is the most common form of pass by construction), so you always get a local copy of the object on resolution if one of these is returned by a method call or send. This local copy can of course accept immediate calls.
Mutable objects, like the car in this example, reside on the machine upon which they were constructed. Such an object is passed by proxy. If such an object is on a different machine from the one on which your piece of the system is running, even when the promise resolves it will only resolve into a far reference (as opposed to a near reference for a local object). Far references accept eventual sends, but cannot accept immediate calls.
So in general, if you created the car using an eventual send to a possibly remote object (a receiver), you would probably have to interact with the car using eventual sends forever, since only such sends are guaranteed to work with remote objects.
# E syntax
def carRcvr := carMakerRcvr <- ("Mercedes")
carRcvr <- moveTo(2,3)
carRcvr <- moveTo(5,6)
carRcvr <- moveTo(7,3)
def fuelVow := carRcvr <- fuelRemaining()The above example moves the carRcvr around repeatedly and then asks for the amount of fuel remaining. This displays another important property of eventual sends: if object A sends several messages via a single reference to object B, it is guaranteed that those messages will arrive and be processed in the order of sending. This is only a partial ordering, however. From B's point of view, there are no guarantees that the messages from A will not be interspersed with messages from C, D, etc. Also, from A's point of view, there are no guarantees that, if A sends messages to both B and C, the message to B will arrive before (or after) the message to C, regardless of the order in which A initiates the messages. Furthermore, if A happens to have 2 different references to B (such as 2 promises that both will eventually resolve to B), the guarantee only applies to messages being sent down one reference.
Despite those uncertainties, however, in the example the partial ordering is sufficient to guarantee that fuelPromise will resolve to the quantity of fuel remaining after all three of the moveTo() operations have occurred. It also guarantees that those moveTo()s will have been performed in the specified sequence.
On first introduction, it is hard to fully appreciate the relationship between partial ordering and when-catch delayed activation. Here is a more sophisticated example.
# E syntax
def cars := [car1,car2,car3]
for each in cars {
def nameVow := each <- getName()
def moveVow := each <- moveTo(3,4)
when (nameVow, moveVow) -> {
println(`Car $nameVow has arrived`)
} catch e {println("A car did not make it")}
}
println ("Cars have been told to move, but no print of any arrivals has yet occurred")
In this example, getName and moveTo messages are sent to all the cars in the list in rapid-fire succession, never waiting for any answers to get back. The "Cars have been told to move" message at the bottom is guaranteed to print before any "has arrived" messages print. We cannot predict which of the 3 cars will arrive first, second, or third. It all depends on where they are running, how slow the connections are, how loaded the processors are. We cannot even predict which will receive the moveTo message first: though the program is guaranteed to fire off the message to car1 first, the processor upon which car1 executes might be several thousand miles away across the Internet, across a dozen bottlenecked routers. And car2 could be running on a computer two feet away, a short hop down an Ethernet connection. In that case car2 would probably be first to receive its message, and be first to actually arrive, and be the first to return its resolution so the arrival message could print--all three of those events being independent, and the first car to do one is not necessarily the first car to do the next.
Meanwhile, it is guaranteed that each car will get the getName message before it gets the moveTo message. It is not guaranteed that the nameVow will resolve to an answer before the moveVow has resolved, so you must use a multi-vow when-catch to use the name inside the when body with confidence. And if the moveTo promise is broken, which activates the catch clause, we do not know the resolution of the name--it could have fulfilled before the moveTo promise broke, or the name promise could be waiting (and on the verge of being broken or fulfilled) as well. We could use the E isBroken() function described later if we wanted to try to print the name of the car in the catch clause in those situations where the name was fulfilled though the moveTo was broken.
Also it is worth noting that the moveTo method, when used in an eventual send, returns a promise even though the moveTo method is of type "void" (i.e., "does not return a value"). This promise, if fulfilled, will always become the uninteresting value of null...but as shown in this example, sometimes it is valuable to know that fulfillment has occurred, or that fulfillment has failed, even if you do not care what the fulfilled value may be.
Sometimes you want to ensure that car1 arrives before car2, and car2 before car3. In a simple problem you could use nested when-catches, described later. In this example, where you don't necessarily know how many cars there are, you would use the recursive when-while pattern or the sendValve utility, both described at the end of the chapter.
In the examples up to this point we have been using a number of naming conventions. Here are some common conventions.
car -- an object that you can treat as a near object, using immediate calls or eventual sends as you wish.
carVow -- a car that will be near, available for immediate calls, once you resolve the vow inside a when-catch clause.
carRcvr -- a car that must be treated as a remote object, spoken to only with eventual sends, never with immediate calls, because although it might sometimes be a near car, it can also be remote, and only eventual sends are safe. As noted earlier, another way of denoting that a car may be remote is to use the ":rcvr" guard.
getCar(licensePlate) -- a function or a method that returns a local car (local to the object implementing getCar)
getCarVow(licensePlate) -- a function or a method that returns a vow for a car, not the car itself. If you are calling this method on a local object, once you resolve the vow, the car is local.
getCarRcvr (licensePlate) -- This method returns a car which the author of the method has reason to believe may be a remote car. Consequently, it is a warning that you should only interact with the returned object with eventual sends.
getCarVow(licensePlateVow, titlePapersRcvr) -- this method explicitly names the parameters licensePlateVow and titlePapersRcvr. Whereas the suffixes Vow and Rcvr are warnings when used in a method or variable name (warnings to avoid the use of immediate calls), when used as parameter suffixes they are commitments. In this example, the developer of the getCarVow method is making the commitment to users of the method that he will not make immediate calls on the licensePlate until he has resolved the plate in a when-catch block. And he is making the commitment that he will only interact with the titlePapers using eventual sends. In general, if at least one of the parameters for a method is either a Vow or a Rcvr, the method itself cannot return a near object: if the method must wait on eventual sends to gather the info to return an object, it cannot reasonably return the object immediately. No doubt, however, someone reading this will be the first person to invent a situation that breaks this general rule. We congratulate you in absentia.
makeCar (name) -- a carMaker that is local, can be immediately called, and makes cars that can be immediately called.
makeCarVow (name) -- an object maker like a carMaker, but which is dependent on construction that may only resolve eventually, so it can only send you a promise for the car.
makeCarRcvr(name) -- a local carMaker that may make the car on a remote system, and therefore the cars should be treated as Rcvrs.
carMakerRcvr(name) -- a carMaker that may itself be on a remote system, that should only be spoken to with eventual sends. Rather than having a very wordy term for remote Makers that make remote cars (which are local to the Maker), since such a Maker is almost always going to create cars that you must treat as Rcvrs, we use carMakerRcvr to also designate makers that we might be tempted to name "carRcvrMakerRcvr". This name, carMakerRcvr, is a first and only example you will see here of the more generalized naming convention used when constructors and authorizers and rcvrs are interrelated in a complex way. Rather than depending on a prefix that might refer to any chunk of the name, we stack up the descriptions as suffixes. Multiple levels of constructors and authorizers play an important role in achieving E's security goals, so while these complicated structures are uncommon in most of the code in a system, there are likely to be a few parts in a secure system where such naming conventions play a role.
As noted above, you can send eventual messages to both promises and far references. So if you plan to treat the fulfilled object as a far reference, using only eventual sends, there is usually no reason to use a when-catch to wait for resolution. Just go ahead and start sending messages, uncaring of whether the promise has yet been fulfilled.
There are four reasons to use a when-catch construct to resolve the promise for a far object.
Here is a test for equality:
# E syntax
def polePositionCarRcvr := raceTrack <- getPolePositionCar()
when (polePositionCarRcvr) -> {
if (polePositionCarRcvr == myCar) {
println("My car is in pole position")
}
} catch prob {}
Note that the catch clause is empty. Because remote references across a network are unreliable, you will usually want to handle the problem that triggered the catch clause. However, if you have many pending actions with a single remote object, all the catch clauses for all those actions will start up if the connection fails, so you may want to disregard most of the catch clauses and focus on handling the larger problem (loss of connection) in one place. Often in this situation you will want to set up a whenBroken message, described later.
But in this example, there is no compelling special action to take if you can't find out whether your car is in pole position, so no action is taken.
If you write a function or method that must get values from a far object before returning an answer, your function should return a promise for the answer, and fulfill that answer later. You can create your own promises using Ref.promise() in E:
# E sample
def calcMilesBeforeEmptyVow(carRcvr) {
def [milesBeforeEmptyPromise, milesBeforeEmptyResolver] := Ref.promise()
def fuelRemainingVow := carRcvr <- getFuelRemaining()
def mpgVow := carRcvr <- getEfficiency()
when (fuelRemainingVow, mpgVow) -> {
milesBeforeEmptyResolver.resolve(mpgVow * fuelRemainingVow)
} catch prob {
milesBeforeEmptyResolver.smash(`Car Lost: $prob`)
}
return milesBeforeEmptyPromise
}
#....Meanwhile, somewhere else in the program....
def myCar {
to getFuelRemaining() {return 5}
to getEfficiency() {return 2}
}
def milesVow := calcMilesBeforeEmptyVow(myCar)
when (milesVow) -> {
println(`miles before empty = $milesVow`)
} catch e {println(`Error: $e`)}
This example highlights several different features of promises. The basic idea of the example is that the function calcMilesBeforeEmptyVow(carRcvr) will eventually compute the number of miles the car can still travel before it is out of fuel; later in the program, when this computation is resolved, the program will print the value. However, before we can do the computation, we must first get both the fuelRemaining and the milesPerGallon values from the remote car.
The promise and the resolver for that promise are created as a pair using Ref.promise(). They are "normal" variables in the sense that they can be passed as arguments, returned from methods as answers, and sent eventual messages. Resolvers truly are ordinary objects. There are 3 restrictions on the promises: they cannot accept immediate calls, they cannot be tested for equality, and they cannot be used as keys in hash tables.
In this example, the function returns the milesBeforeEmptyPromise to the caller just as it would return any other kind of value. To cause the promise to resolve, the function calls the resolver (or sends to the resolver if the resolver may be remote to your program) with the "resolve(value)" method. To break the promise (which produces a problem that will cause the catch clause of a when-catch to execute), call the resolver with the "smash(problem)" method.
It is possible to chain resolutions: you can resolve a promise by handing the resolver another promise (though if you hand the resolver the promise which it is intended to resolve, E will detect the problem and automatically break the promise--a promise resolved to itself can never be fulfilled). If you resolve a promise with another promise, the original promise is not yet considered resolved, i.e., the when-catch body will not be activated by such a resolution. The when-catch body will only be activated when the entire promise chain is resolved, and there is an actual reference to an actual object available.
As noted in chapter 1 under the Secret Lives of Flow Control Structures, control structures return values. When used as described up to this point, the return value of the when--catch expression has been ignored. In fact, the when-catch expression returns a promise for the value that the done function will return. If the done function does not return a value, then it will always resolve to null. However, if the done(value) function of the when-catch returns a value, the when-catch returns a promise that will be fulfilled by the last value in the executed clause. Rewriting the calcMilesBeforeEmptyVow using this feature, we get: not right, not right, fix
# E sample
def calcMilesBeforeEmptyVow(carRcvr) {
def fuelRemainingVow := carRcvr <- getFuelRemaining()
def mpgVow := carRcvr <- getEfficiency()
return when (fuelRemainingVow, mpgVow) -> {
mpgVow * fuelRemainingVow
} catch prob {
throw(`Car Lost: $prob`)
}
}
Here the when-catch creates the promise which is returned by the function, and resolution is automatic at the end of the when-catch execution. What does the promise resolve to during a catch and/or a finally?
When chaining promises, what happens if one of the promises in the middle or the far end breaks? The problem that breaks the far promise is propagated down all the promises dependent on it. As a consequence, broken promises work their way through the system in a similar way to exceptions being caught in try/catch blocks.
Having worked our way through near, far, broken, and promised references, let us put all the transitions among these together in a single place. The following diagram shows all the kinds of references E distinguishes between, and it shows the two possible transitions:
The 2 transitions are:
That is the complete collection of transitions.
Very early in the book, we presented a list of E types, including any, integer, float64, etc. In that list were "rcvr", "vow", "pbc" and "near". "pbc" means that the type of the value is required to be "pass by construction". Transparent immutables are generally pbc, and can be passed through this type declaration. Strings, integers, floats, booleans, ConstMaps and ConstLists are all pbc. Objects that are pbc are copied when they are sent, so that even if the object originated on a remote computer, when you get it you can use immediate calls on it. Also, since pbc objects do not embody any code, pbc objects passed across a trust boundary can be used with far fewer security concerns: the pbc object may contain bad data, but it cannot play any tricky games in its execution.
"Near" means that the value must be local. Promises and far references will not pass the near guard. Anything that passes a near guard can be sent messages with immediate calls.
At this time, "rcvr" and "vow" have the same algorithmic meaning as "any", but can be used as hints to the programmer about how to interact with the object (a vow should be referenced only eventually until it is resolved; a rcvr should always be referenced only eventually).
As discussed earlier, we can guarantee the ordering of messages sent from one object to another object. But when a program sends messages to several objects, on different machines, the order in which answers will be received cannot be predicted. Suppose you need to perform a computation involving two far answers, and the algorithm depends on the result from the first answer. You can nest when-catch clauses to await the outcome of the dependency:
# E syntax
def fileRcvr := fileCacheRcvr <- getFile()
when (def exists := fileRcvr <- exists()) -> {
if (exists) {
when (def text := fileRcvr <- getText()) -> {
println(`File has text: $text`)
} catch p2 {println(`Problem with existing file: $p2`)}
} else {println("File does not exist")}
} catch prob {println(`Problem reaching file: $prob`)}
Here, we want to print the text of a farFile, but we want to be sure the file exists first. So we ask the file if it exists, and only after we determine that it does indeed exist do we proceed to get the text.
As noted earlier, the "done(variable)" clause of the when-catch construct is actually the declaration of a function and a series of parameters. The reasons for this are beyond the scope of this book, but the operational consequence is that you must use different function names if you have multiple when-catch clauses in the same scope.
Nested when-catch clauses put the "done" functions into the same scope. In the example here, we gave the inner when-catch "done" function the name "done2" to prevent name collision. We also gave the error-containing variables in the catch clauses different names, e and e2.
Up to this point we have carefully dodged a critical question. How does a program acquire its very first reference to an object on a different computer? In all our examples up to this point, we have always started out with a reference to at least one remote object, and we retrieved other remote objects by asking that object for other objects: for example, we asked a remote carMakerRcvr for a new (remote) carRcvr, and were able to immediately work with the car just like any other remote object. How did we get the reference to the carMakerRcvr in the first place?
In E, the reference to an object can be encoded as a Universal Resource Identifier string, known as a uri (the familiar url of the Web is a type of uri). This uri string can be passed around in many fashions; one good secure way of passing a uri is to save it as a text file, encrypt and sign it with PGP, and send it in email. If you wish to run a seriously secure distributed E system, encrypting the uris is crucial: indeed, the passing of the uris from machine to machine is the main security issue that E cannot address for you (and is a problem shared by all security systems, a problem that will diminish with time as secure systems like PGP email are deployed). Other ways uris have been passed in operational E systems have been to send the uri over an ssh connection, and (less securely) by reading the uri off over a telephone! If you are using E on a local area network and have no security concerns, but are using E simply because it is simpler, safer, and more maintainable for distributed computing, the uris can be stored in files on a shared file system and read directly by the programs on different computers as they start up.
The functions makeURI(object) and objFromURI(uri) detailed in the E Quick Reference Card perform the basic transformations you need to hook up objects on multiple computers. These routines use sturdy refs and liverefs in their computations. A sturdyRef is an object that contains an enduring description of where an object can be found; sturdyRefs and URIs are simple transformations of one another. LiveRefs are actual connections to objects that disappear any time a connection goes down. LiveRefs carry the actual traffic in E communication. When you request a remote object from another remote object (as in farCarMaker <- ("mercedes")), what you actually get is a liveRef. If you want to continue to connect with that particular car across multiple sessions, you will need to explicitly get a sturdyRef from the car (and the car will have to have an explicit method for granting the sturdyRef. The sturdyref function is an important capability, not one handed out by default).
Each program that expects to work with remote objects needs to invoke the command
# E sample introducer.onTheAir()
before starting any remote connections, including the making or using of uris. An example of these functions working together can be found in eChat.
E programs never block. They never just sit and wait around for results. However, most programs will toss out some windows, or set up some services, and then desire to wait for someone or some thing to use them. Often, the first version of a program written by a new E programmer will set everything up, execute the last line of the program, and then shut down. It is quite disconcerting to watch your windows briefly light up and then disappear, along with the java virtual machine underlying the entire program.
The command
interp.blockAtTop()
causes the E interpreter to wait for requests to come in. The command
interp.continueAtTop()
causes the E interpreter to stop waiting. These commands are used in the eChat sample at the end of this chapter: the last line of the program initiates blockAtTop, and continueAtTop is invoked when the user closes his chat window.
If a method accepts a promise or far reference as a parameter, the returned value almost certainly cannot be a local object. Why is this? If the computation of the returned object depends on something eventual, the computation will have to wait for an answer, which means the best the object can do for immediately returning something is to return a promise. So a method like getCar(licensePlateVow) has almost certainly violated the naming convention, one way or the other.
As a consequence of this and other characteristics of promises, beginning E programmers often experience a moment of breathlessness soon after they first try to use them. This moment occurs when the programmer erroneously concludes that, once you let a promise into your system, the program has tumbled down a slippery slope from which there is no recovery. It feels as if you will never be able to have a real value in your hands again; you'll spend the rest of your life making eventual sends, coupled with endless when-catch constructs just to get to the next step.
The breathlessness can metastasize into a feeling of having lost control of the software: everything is off computing somewhere else and all you have is a bunch of promises. Since E never (really, never!) blocks, you can never tell things, "just stop for a moment while I catch my breath and gather up all the values" :-)
In fact, there are several patterns and tools for getting control of things again. The basic multi-promise when-catch, and the promiseAllResolved and Summoner patterns at the end of this chapter are just a few examples of structures specifically designed to reduce the frequency with which you need when-catch. They allow you to get quickly back into local computation after a distributed action. You really can catch your breath.
And once the moment of breathlessness passes, you will feel freedom, for you will never have to deal with thread synchronization again; you will never lie awake at night wondering if an unexplored race condition will cause deadlock a few days after the production load level hits your system. How early can I put this paragraph easily? AHK proposes right after Promises.
Each vat has a separate event loop, each game turn corresponds to the processing of a single event to conclusion with no interruptions. There are frontend vats, which all share the swing event queue for queuing messages to their event loops, and there are backend queues, each of which has its own separate event queue and runs on a separate thread. Frontend vats have the advantage that they can interact directly with user interface widgets. The initial vat created by default at the start of an E program is a frontend vat. You might want multiple frontend vats if you want to shut down whole subsystems in one fell swoop: you can tell a vat to shut down, and it will terminate all its connections to the outside world, making it available for garbage collection. Have markm review this and tell me what is wrong with this description.
creating a second frontend vat on a single jvm. creating a backend vat on a jvm
We started this chapter with a description of a human interaction, with Alice as the CEO of Evernet Computing in search of a new spreadsheet for her IPO preparation. What would that distributed problem look like in code? Though we present only a skeleton of this human operation here, it is perhaps nonetheless informative. It might also be informative, after one has read how to do the example in E, to think about how it would be done with threads, in Java or C++. That, however, is left as an exercise for the reader.
# E sample
#---- Bob Code
def bobRcvr {
to handleExplosionInNetworkServerRoom() {
# handle the explosion
}
to getCostNumbers() :pbc {
return ["R&D" => 10000]
}
}
#------------ Carol Code
def carolRcvr {
to attendMeeting(room, time) {
# attend the meeting
}
to integrateSpreadsheetVow(RDNumbersVow) {
def sheetVow := when (RDNumbersVow) -> {
["Marketing" => 50000] | RDNumbersVow
} catch prob {
println("No numbers!")
prob
}
return sheetVow
}
}
#--------- Alice Code
def aliceRcvr {
to prepareIPOVow() {
def RDNumbersVow := bobRcvr <- getCostNumbers()
def spreadsheet := carolRcvr <- integrateSpreadsheetVow(RDNumbersVow)
return when (spreadsheet) -> {
println(`Appendix: Budget Spreadsheet: $\n$spreadsheet`)
} catch prob {
aliceRcvr <- fireSomebody([bobRcvr, carolRcvr])
}
# do other IPO preparations
}
to fireSomebody(candidateRcvrs) {
# get rid of culprit
}
}
bobRcvr <- handleExplosionInNetworkServerRoom()
carolRcvr <- attendMeeting(1,2)
aliceRcvr <- prepareIPOVow()
In order to focus on the distributed computation features of this chat tool, we have gone to extreme lengths to make the user interface machinery a very short piece of code. Hence, this should properly be called "minChat", because the user interface is so brutally minimal. Even with such minimization, however, the user interface still requires more lines of code than the actual distributed computation!
The chatController at the bottom of the example is the heart and soul of the computation.#?? in new vat minChatVat.e-swt
## E sample
#!/usr/bin/env rune
#eChat with minimalist user interface
pragma.syntax("0.9")
def <widget> := <swt:widgets.*>
def SWT := <swt:SWT>
introducer.onTheAir()
# return the object represented by the URI
def getObjectFromURI(uri) {return introducer.sturdyFromURI(uri).getRcvr()}
def makeURIFromObject(obj) :String {
# This implementation assumes a non-persistent single incarnation
def [sr, _, _] := identityMgr.makeKnown(obj)
#XXX not a uri if bracketed, bug, markm?
def bracketed := introducer.sturdyToURI(sr)
if (bracketed =~ `<@uri>`) {return uri}
return bracketed
}
def chatController
def chatArea
def chatUI {
to show() {
def frame := <widget:Shell>(currentDisplay)
frame.setText("eChat"); frame.setBounds(30, 30, 600, 300)
def winDisposeListener {to widgetDisposed(event) {interp.continueAtTop()}}
frame.addDisposeListener(winDisposeListener)
bind chatArea := <widget:Text>(frame,
(SWT.getMULTI() | SWT.getWRAP()) | (SWT.getBORDER() | SWT.getV_SCROLL()))
def commandLine := <widget:Text>(frame, SWT.getSINGLE() | SWT.getBORDER())
def enterKeyListener {
to keyPressed (event) {
if (event.getKeyCode() == 27) {
if (commandLine.getText() =~ `@command @argument`) {
commandLine.setText("")
switch (command) {
match == "save" {chatController.save(<file: argument>)}
match == "load" {chatController.load(<file: argument>)}
match == "send" {chatController.send(argument)}
match _ {commandLine.setText("Error. Try save load, or send")}
}
} else {commandLine.setText("Error. Try save, load, or send")}
}
}
match [verb, args] {}
}
commandLine.addKeyListener(enterKeyListener)
def label := <widget:Label>(frame, SWT.getLEFT())
label.setText("save, load, send. No quotes for path. Use Escape to start operation. ")
swtGrid`$frame: $chatArea.X.Y
$label.X
$commandLine.X`
frame.open()
}
to showMessage(initiator, text) {chatArea.append(`$initiator: $text $\n`)}
}
def friend
bind chatController {
to send(message) {
when (friend<-receive(message)) -> {
chatUI.showMessage("self", message)
} catch prob {chatUI.showMessage("system", "connection lost")}
}
to receive(message) {chatUI.showMessage("friend", message)}
to receiveFriend(friendRcvr) {
bind friend := friendRcvr
chatUI.showMessage("system", "friend has arrived")
}
to save(file) {file.setText(makeURIFromObject(chatController))}
to load(file) {
bind friend := getObjectFromURI(file.getText())
friend <- receiveFriend(chatController)
}
}
chatUI.show()
# In actual code, the following line would not be commented out
# interp.blockAtTop()
There are only 5 methods in making chat happen: sending a message to the friend, receiving a message from the friend, being informed that the friend has found you (the receivedFriend method), saving the URI where your friend can find you, and loading the URI that describes where you can find your friend.
A quick run through can be done by putting the source in a file (file extension ".e-swt", this example uses SWT for user interface) and launching the file twice to get 2 chat windows. In one window, type "save ~/me.minchat" and press the Escape key. This creates a file (in your home directory) that contains a cap: URI describing where your chat session can be found (a chat session that can only be found by someone to whom this file has been given--the location cannot be found or guessed by anyone else). In the other window, type "load ~/me.minchat" and press Escape. In the first window you should see a message pop up that the "friend has arrived". Now in either window type "send hi y'all", and you will see the message appear in both windows.
In minChat , every time you start up the program, it comes to life with a new uri that must be sent to the friend with whom you wish to chat; the old uri from the previous session is useless. MinChat would be far more useful if you could create a minChat session with Bob, and continue that session even after you have rebooted your computer. We look at building such a persistent chat in the chapter on Persistent Secure Distributed Computing--after first looking at minChat here through the eyes of a security reviewer (or a cybercracker) in the chapter on Secure Distributed Computing.
Suppose you have a computation that cannot be performed until you have received answers from multiple remote objects. Your first thought would be to use a multi-vow when-catch. But further suppose you do not know beforehand how many vows there are (i.e., suppose you have an arbitrary list of promises). In this case neither the multi-vow when-catch construct nor the nested when-catch pattern will work. For this situation, you can use the resolveAllVow utility described here.
The resolveAllVow utility has another distinction compared to a multi-vow when-catch: whereas the multi-vow activates the catch clause as soon as it hits a broken promise, resolveAllVow sends no resolution until everything in the list is resolved one way or the other. So, even if several of the promises were broken, with resolveAllVow you can be sure that the rest have been fulfilled when the catch clause is activated.
In this example, we use resolveAllVow to sum a list of contributions. We do not know beforehand how many contributions there are (perhaps thousands), and we want to sum them up even if some of the requests for donations return broken promises.
E isBroken(obj) does not work. what does?
# E sample
# Given a List of promises, when all the promises have been resolved,
# resolve the returned vow, either fulfilling it with the list of fulfillments (if
# all the promises were fulfilled) or by smashing the returned promise
# with one of the broken promises (if at least one promise was broken).
def resolveAllVow(promises) {
def [resultVow, resolver] := Ref.promise()
var count := promises.size()
var resolution := promises
var noneBroken := true
if (count == 0) {resolver.resolve(promises)}
for promise in promises {
when (promise) -> {
# success processed in finally clause
} catch prob {
resolution := prob
noneBroken := false
} finally {
count -= 1
if (count == 0) {
if (noneBroken) {
resolver.resolve(resolution)
} else {resolver.smash(resolution)}
}
}
}
return resultVow
}
#now use the promiseAllResolved to sum contributions, where the
#number of contributions is unknown prior to execution
def printTotalContributions(amountsList) {
var total := 0
for each in amountsList {
if (!(Ref.isBroken(each))) {total += each}
}
println(`Total contributions are: $total`)
}
# scaffold contributors: real contributors would be remote Rcvrs
def makeContributer(donation) {
return def contributor { to getDonation() :int {return donation} }
}
def contributorRcvrsList := [makeContributer(5), makeContributer(6)]
def amountsVows := [].diverge()
for each in contributorRcvrsList {
amountsVows.push(each <- getDonation())
}
when (def amounts := resolveAllVow(amountsVows)) -> {
printTotalContributions(amounts)
} catch prob {
#due to the nature of resolveAllVows, in this catch clause
#we are guaranteed everything in amountsVows is resolved to
#either an amount or a broken reference
printTotalContributions(amountsVows)
}
The basic behavior of resolveAllVow is this: when initiated, the function takes a count of how many promises there are, then immediately spins off when-catches for all of them. Each time a promise resolves, the count of outstanding promises is decremented. When the counter drops to zero, implying all the promises have been resolved, the single promise initially returned by resolveAllVow is finally resolved.
Java's standard dialog boxes, found in JOptionPane, are seriously flawed for use in a distributed E system: those convenience dialogs are fully blocking. As a consequence, if your computer is part of a distributed computation, and you go home for the night, if a Java dialog box pops up, your machine is effectively offline until you return in the morning. Included in the E distribution is a dialog box tool, the dialogVowMaker:
code here
As discussed earlier, putting a when-catch inside a for loop does not give you any guarantees on which when-catch will activate first: it only guarantees which message is sent first, which is mostly uninteresting. For general-purpose ensured sequencing of resolution, you must use recursion.
Earlier, we had an example in which we told a list of cars to moveTo the same location. Suppose we wished to ensure that the cars arrived in the order in which they appear in the list:
# E sample
def moveAllCarsInSequence(carRcvrs,toX,toY) {
var carI := 0
def moveRemainingCars() {
if (carI < carRcvrs.size()) {
def nextCarRcvr := carRcvrs[carI]
carI += 1
def nameVow := nextCarRcvr <- getName()
when (nameVow,nextCarRcvr <- moveTo(toX,toY)) -> {
println(nameVow + " arrived, next car about to start moving")
} catch e {
println(`car died: $e`)
} finally {moveRemainingCars()}
}
}
moveRemainingCars()
}
# scaffold carmaker
def makeCarRcvr(name) {
def carRcvr {
to getName() :String {return name}
to moveTo(x, y) {
#move the car
}
}
return carRcvr
}
moveAllCarsInSequence([makeCarRcvr("car1"),makeCarRcvr("car2")],3,4)
Inside the moveAllCarsInSequence function, we declare the recursive moveRemainingCars function, then simply invoke moveRemainingCars. The function moveRemainingCars tells the next car to move; once that movement has resolved, moveRemainingCars invokes itself once again.
In this version of moveAllCarsInSequence, if a car is lost (i.e., its promise to move is broken), the recursion continues. If we wanted to abort after a problem occurred, we would simply delete the call to moveRemainingCars() from the finally clause, and place it in the when clause.
As we have seen, it is possible to spin off large numbers of eventual sends in the blink of an eye. In fact, tossing off vast numbers of such sends can sometimes consume a lot of resources for no good purpose. In the eDesk program at the end of the book, it is possible for the user to request the transfer of whole directories full of files. It would be possible for eDesk to initiate transfer of all of those files simultaneously. But the processing bottleneck is probably bandwidth, so initiating them all at once won't get the whole transfer done any faster, and meanwhile each individual file transfer consumes buffer space once initiated. Starting all the transfers at once could actually slow down the transfer if enough buffers thrash virtual memory.
What you want is a valve, which can be opened and closed depending on the situation. The program can still set up all the operations in a single swoop, but the valve will constrain the number of such operations that are actually active at a given moment. The sendValve below performs this function.
#sendValve. If you have numerous eventual sends to initiate,
# but initiating them all at once would consume vast resources
# and/or would actually slow down processing, queue the actions
# through a valve.
#An ActionTrio is the list of [obj,"verb",[arguments]]
# that can be used in an E send()
#The actions are guaranteed to be initiated in the sequence in
# which they are placed in the valve, though of course there is
# no guarantee as to which will terminate first (unless you have
# special knowledge of such sequencing outside the valve).
#The numSimultaneousAllowed is the number of actions that can
# be run concurrently through this valve.
# E sample
def makeSendValve (numSimultaneousAllowed) {
var actionQueue := []
var numRunning := 0
def startNext() {
if (!(actionQueue.size()==0)) {
def [actionTrio, resolver] := actionQueue[0]
actionQueue := actionQueue(1, actionQueue.size())
numRunning += 1
def completeVow := E.send(actionTrio[0],actionTrio[1],actionTrio[2])
resolver.resolve(completeVow)
when (completeVow) -> {
} catch prob {
} finally {
numRunning -= 1
startNext()
}
}
}
def valve {
to makeActionVow(actionTrio) {
def [completeVow, resolver] := Ref.promise()
actionQueue := actionQueue.with([actionTrio, resolver])
if (numRunning < numSimultaneousAllowed) {
startNext()
}
return completeVow
}
}
return valve
}
The sendValve utility, if used with numSimultaneousAllowed==1, can also perform the send sequencing function described earlier. However, the earlier general send sequencing pattern is still useful. For example, if the moveAllCarsInSequence operation should abort upon a encountering a broken resolution, this version of sendValve would not give the correct result.
It is possible to have a network of machines in which the machines do not all have direct access to one another. Suppose in the network with machine A, B, and C, A and C cannot form a direct connection because of network topology, but B can reach everything. If A needs a capability on C, we can put a transparent forwarder on B such that A sends its messages to the forwarder on B:
# E sample
def makeTransparentForwarder(representedRcvr) {
def forwarder {
match [verb,args] {E.send(representedRcvr,verb,args)}
}
return forwarder
}Create the forwarder on B, and hand the reference to the forwarder to the appropriate object on A. An example can be found in the eDesk example at the end of the book, in which a forwarder is used if the system detects that two of the file servers are unable to directly connect. Note that the "delegate" keyword does not quite work here: delegate generates immediate calls, not eventual sends. So we had to revert to the more general purpose matching process.
Suppose you are periodically requesting a particular piece of information from a far object. Suppose the far object cannot supply the answer in a single game turn, i.e., it must ask other objects for information to get the answer, and so it sends you a promise rather than a result. It is possible for the resolutions of these promises to occur out-of-order, i.e., the resolution on the second request could get to you after the resolution of the third request has already occurred. In this situation, it will appear that the answer to the second request is newer and more up-to-date than the third request.
In this situation, to ensure you are getting only new information, not stale information, use the acceptOnlyMoreRecentVow utility.
def acceptOnlyMoreRecentVow
code here
This situation, while rare, actually arises in the Marketplace example in the Secure Distributed Computing chapter.
something about time, the race construct, timeout construct
The following is not really a pattern so much as it is an example of all the more novel (i.e., not-like-Java) elements of E playing together. The problem, taken from the eDesk program in the Examples at the end of the book, is this:
example
Though E is safe from deadlock, in which several objects compete for acquisition to a set of resources, there is a related dilemma that E programs can encounter. If two or more "when" clauses wait for each other to resolve before they can resolve themselves, this causes data lock. A complete discussion of data lock and why it is far less of a risk than deadlock is beyond the scope of this book. Briefly, both theory and current experience suggest that data locks are more difficult to create in the wild than deadlocks. Furthermore, when datalocks do occur, they are less likely to freeze up important subsystems on a grand scale: unlike the deadlock, they only choke a few "when" clauses, not a set of "critical sections" which have been named "critical" in thread programming for good reasons. And lastly, datalocks are consistent, reproducible, and therefore fixable, unlike the deadlocks that appear mysteriously when some incredibly arcane race condition occurs.
Here is a simple though foolish example of datalock. It is the E implementation of the English sentence, "This sentence is false."
def sentence := sentence <- not()
In this construction, the sentence will forever be an unresolved promise. As noted above, the failure of this to resolve will not impede the progress of the rest of the program in the least. And it is very reliable: the unresolvability of the promise will appear the first time you run the program, and the second, and in every debugging pass thereafter.
At the time of this writing, data lock has been seen only once in the wild. In that instance, a widely used object (call it WUO),contained one piece of mutable state and one piece of immutable state. The object that computed new versions of WUO's mutable state needed the immutable part of the state, and requested the current version of WUO from the owner of the object. Unfortunately, the owner knew that a new version was under construction, and was only sending out promises for WUO until the update was resolved. The solution in this case was to refactor WUO. This resulted in an architecture that was cleaner even by normal object design criteria, without regard to the distributed usage behavior. Insufficient data is yet available to assess how commonly this type of solution will suffice.
E uses capability based security to supply both strong security and broad flexibility without incurring performance penalties. Capabilities might be thought of as the programming equivalent of physical keys, as described with the following metaphors.
Principle of Least Authority (POLA)
When you buy a gallon of milk at the local 7-11, do you hand the cashier your wallet and say, "take what you want and give me the rest back?" Of course not. In handing the cashier exact change rather than your wallet, you are using the Principle of Least Authority, or POLA: you are giving the cashier no more authority than he needs. POLA is a simple, obvious, crucial best-practice for secure interactions. The only people who do not understand the importance of POLA are credit card companies (who really do tell you to give that far-off Internet site all your credit, and hand back what they don't want), and computer security gurus who tell you to use more passwords.
Suppose all security in the physical world were based on ID badges and ID readers. At your home you might put an ID reader on your door, another on your CD cabinet, and another on your gun vault. Suppose further you had to depend on 4-year-old children to fetch your CDs for you when you were at the office. How would you do it? You would hand your ID badge to the child, and the child could then go through the front door and get into the CD cabinet. Of course, the child with your ID badge could also go into the gun vault. Most of the children would most of the time go to the CD cabinet, but once in a while one would pick up a gun, with lamentable results.
In the real physical world, if you had to depend on children to fetch CDs, you would not use an ID badge. Instead you would use keys. You would give the child a key to the front door, and a key to the CD cabinet. You would not give the child a key to the gun vault.
All current popular operating systems that have any security at all use the ID badge system of security. NT, Linux, and Unix share this fundamental security flaw. None come anywhere close to enabling POLA. The programming languages we use are just as bad or worse. Java at least has a security model, but it too is based on the ID badge system--an ID badge system so difficult to understand that in practice no one uses anything except the default settings (sandbox-default with mostly-no-authority, or executing-app with total-authority).
The "children" are the applications we run. In blissful unawareness, we give our ID badges to the programs automatically when we start them. The CD cabinet is the data a particular application should work on. The gun vault is the sensitive data to which that particular application should absolutely not have access. The children that always run to get a gun are computer viruses like the Love Bug.
In computerese, ID badge readers are called "access control lists". Keys are called "capabilities". The basic idea of capability security is to bring the revolutionary concept of an ordinary door key to computing.
Let us look at an example in a computing context, of how keys/capabilities would change security.
Consider the Melissa virus, now ancient but still remembered in the form of each new generation of viruses that use the same strategy the Melissa used. Melissa comes to you as an email message attachment. When you open it, it reads your address book, then sends itself - using your email system, your email address, and your good reputation - to the people listed therein. You only had to make one easy-to-make mistake to cause this sequence: you had to run the executable file found as an attachment, sent (apparently) by someone you knew well and trusted fully.
Suppose your mail system was written in a capability-secure programming language. Suppose it responded to a double-click on an attachment by trying to run the attachment as an emaker. The attachment would have to request a capability for each special power it needed. So Melissa, upon starting up, would first find itself required to ask you, "Can I read your address book?" Since you received the message from a trusted friend, perhaps you would say yes - neither Melissa nor anything else can hurt you just by reading the file. But this would be an unusual request from an email message, and should reasonably set you on guard.
Next, Melissa would have to ask you, "Can I have a direct connection to the Internet?" At this point only the most naive user would fail to realize that this email message, no matter how strong the claim that it came from a friend, is up to no good purpose. You would say "No!"
And that would be the end of Melissa, all the recent similar viruses, and all the future similar viruses yet to come. No fuss, no muss. They would never rate a mention in the news. Further discussion of locally running untrusted code as in this example can be found later under Mobile Code.
Before we get to mobile code, we first discuss securing applications in a distributed context, i.e., protecting your distributed software system from both total strangers and from questionable participants even though different parts of your program run on different machines flung widely across the Internet (or across your Intranet, as the case may be). This is the immediate topic.
There are a couple of fundamental concepts that must be gotten right by a programming language in order to use capability discipline. We mention these here.
Pointer arithmetic is, to put it bluntly, a security catastrophe. Given pointer arithmetic, random modules of code can snoop vast spaces looking for interesting objects. C and C++ could never support a capability system. Java, Smalltalk, and Scheme, on the other hand, did get this part of capability discipline right.
In a capability language you can not reach inside an object for its instance variables. Java, Smalltalk, and Scheme pass this test as well.
In JavaScript, as a counterexample, all instance variables are public. This is occasionally convenient but shatters any security hopes you might have. JavaScript is a relatively safe language only because the language as a whole is so thoroughly crippled. We consider JavaScript safe, but not secure: We consider security to require not only safety but also the power to get your work done: POLA means having enough authority, as well as not having too much. Using this definition of security, the Java applet sandbox is mostly safe, but not at all secure. And a Java applet that has been allowed to run in a weaker security regime because the applet was "signed", is neither safe nor secure.
In a capability system, the only source of positive authority for an object should be the references that the object holds.
Java fails here, along with Smalltalk and Scheme. A famous example of the trouble that static mutable state can get you appeared in Java 1.0 (corrected in 1.1, an upward compatibility break so rarely used they could get away with it). The object System.out, to which people routinely sent print messages, was replaceable. A programmer in the middle of a bad hair day could easily replace this object, reading everything everyone else was doing, and preventing anyone else from reading their own outputs.
You can make everything else right, but if the APIs for a language were designed without consideration of security, the capability nature of the system is seriously flawed. Let us consider an example in Java. Suppose you had an analysis program that would present graphs based on the contents of an existing spreadsheet. The program only needs read access on the one spreadsheet file, it needs nothing else in your file system. In Java, then, we would grant the application an InputStream.
Unfortunately, an InputStream in Java leaks authority. In this example, you could "cast down" to a FileInputStream, from which you could get the File, from which you could get a WriteStream and a whole filepath, which would give you total access and control over the entire directory system.
To fix this problem for a single object in a single application, you could write your own InputStream class that doesn't leak. This strategy does not scale, however: requiring the programmer to write replacements for every object in the API to achieve security will result in few secure programs (just as requiring the programmer to write his own bitmap handlers and gui widgets will result in few point-and-click programs). To really fix this security problem in Java, you would have to rewrite the entire java.io package, wrecking backward compatibility as a side effect. Without that kind of massive, serious repair, it is always easier to create a breach than to create a secure interaction. With an infrastructure actively trying to harm you, what chance do you really have?
E has no pointer arithmetic. E has no mutable statics. E has an API carefully thought out to prevent capability leaks. This would make it a capability secure language for single-processor applications. But E goes a step further. It takes the concept of a secure, unforgeable object reference and extends it to distributed objects:
These aspects of E protocol can be understood better by looking at the URI that identifies an E object to other objects outside its own vat.
There is no such thing as a free lunch, but this does not rule out the possibility of lunches at bargain prices. When programming in E, you are automatically working in a capability secure environment. All references are secure references. All powers are accessible only through capabilities. Making an E program secure is largely a matter of thinking about the architecture before you code, and doing a security audit after you code. When designing and auditing you use a small number of principles for scrutinizing the design:
The Principle of Least Authority (known henceforth as POLA), which has been used by human beings instinctively for thousands of years, translates to computer programs fairly straightforwardly: never give an object more authority than it needs. In particular, if an object may be running on a remote untrusted machine, think very carefully about the minimum set of authorities it needs, and give it capabilities only on facets (described later) that ensure it gets no more. A simple word processor needs read and write access to the one file it is being used to edit, and it needs read-only access on all the system fonts. It does not need any access to anything else. Do not give it access to anything else. If it asks for something else, it is lying to you, and you should not trust it.
When developing software, remember that the person who controls the hardware always, at the end of the day, controls the software. Hence, if you send someone a piece of a distributed program to run on their own equipment, that person totally and utterly owns everything that resides on his machine. They can modify the code you gave them, or rewrite it from scratch yet make it look from the outside like it is identical. You must therefore think carefully about what features of your system you really trust on that remote machine. A key feature of E that enhances its reliability is that objects which are manufactured in a particular vat remain resident in that vat, so that the objects remain as reliable as the objectMakers used to produce them. Only transparent immutables (immutables that don't encapsulate anything) actually move across computational boundaries.
Many people have made the error of believing this principle of hardware ownership can be circumvented. At the time of this writing, the music recording industry is throwing away truly fabulous sums of money on schemes that assume they can somehow control data after it has arrived on a user's own hardware. Microsoft is busily developing Palladium (uh, I mean, NGSCB). Intel is busily developing TCP (uh, I think they changed the name to La Grande). Their fate has already been foretold in the fate of the popular game Diablo I: authoritative representations of data were allowed to reside on user computers, assuming that the object code was too difficult to understand to be hacked and that the Diablo client code would always behave as the designers intended. They were 99% correct, which in the computer security field means they were 100% wrong. Today, 99% of the people who hack Diablo I don't understand the object code. But somewhere some single individual figured it out and posted the result on the Web. Now your grandmother can sabotage shared Diablo I games as quickly and easily as the most accomplished hacker in history. For Diablo II, the developers had learned the lesson. Authoritative information is stored only on the server, giving them the beginnings of true security.
Not only does hardware own the software, so too does the underlying operating system. As we have stated repeatedly here, E enables the construction of extremely secure computing systems. However, E systems can still be attacked from below, by viruses granted authority by the operating system underneath the E world. Such attacks can only be prevented by completing the security regime, either with capability-based operating systems, or with computers on which only capability-secure programs are allowed. There is one open-source capability-based OS under development, at www.eros-os.org.
One form of attack that even E cannot defend against is denial of service (DOS). In a denial of service, someone simply swamps your system with requests, making your system unable to operate. Such attacks take many forms, but the key limitation in such attacks is this: such an attack can never steal your secret data or gain control of your sensitive operations. If DOS were the only kind of attack possible, the world would be a vastly better place. Indeed, if only DOS attacks were possible, even most modern DOS attacks would fail, because the serious DOS attacks require that the attacker first gain control of hundreds of computers that belong to other people, by using attacks far more invasive and damaging than DOS itself.
The minChat application presented earlier as the example for a distributed system was designed with no consideration at all for security issues. This is literally true: minChat is derived from eChat, which was written by the author as his first practice exercise to learn E. However, because eChat was written in a capability secure environment, because the author used a fairly clean modular architecture, and because clean architecture in a capability-secure infrastructure makes its own luck, there were no serious security breaches in eChat. And, as we shall see, there are no serious security breaches in minChat, though there are many interesting lessons we can learn from it.
First of all, in minChat as in all E programs, the communication is all encrypted and the objects are all unfindable and unguessable. Therefore no third party can enter the conversation to either eavesdrop or forge messages. The only source of security breach will be the person at the other end of the chat system, i.e., the person with whom you planned to chat. So, right off the bat, simply by using E you have eliminated outsider attacks. But insider attacks are still possible, and indeed constitute the most serious threat to most security systems anyway. You have to be able to achieve cooperation in the presence of limited trust. Cooperation despite limited trust is something we humans achieve every day, for example when paying 3 bucks for a gallon of milk at the local QwikMart. But it is notoriously difficult to get these transactions working correctly on computer systems when using conventional languages. So we will examine minChat very closely to ensure that the person we want to chat with does not also get any inappropriate powers, like the ability to delete all our files.
Now, let us begin with a quick review of minChat's code to see what parts of the system we need to examine to do a security review. As you may recall from the chapter on Ordinary Computing, emakers come into existence with no authority whatsoever, and receive authority only via the arguments that are passed into their objects. In larger programs, one can often ascertain that the emaker receives so little authority it can be disregarded from a security perspective: if it doesn't have enough authority, it just can't be a danger (as documented in the DarpaBrowser Final Report and Security Review).
MinChat is just 2 pages of code. It does not use emakers, so this does not help in our current situation. However, what does help is that the only object accessible to the outside world is the chatController, whose reference is sent to the friend (or, in this case, perhaps the enemy) with whom we wish to chat (hey, we need to talk to our enemies too, from time to time). What unpleasantness can our friend do on our computer through the chatController interface he receives? We will reproduce the crucial lines of code here for convenience:
to send(message) {
when (friend<-receive(message)) -> {
chatUI.showMessage("self", message)
} catch prob {chatUI.showMessage("system", "connection lost")}
}
to receive(message) {chatUI.showMessage("friend", message)}
to receiveFriend(friendRcvr) {
bind friend := friendRcvr
chatUI.showMessage("system", "friend has arrived")
}
to save(file) {file.setText(makeURIFromObject(chatController))}
to load(file) {
bind friend := getObjectFromURI(file.getText())
friend <- receiveFriend(chatController)
}
There are only 5 messages here. Let's go through what our friend could do with each one of them in sequence.
So, our friend can crash minChat, but can't do any serious harm: an attack whose only result is to cut off the target from the attacker's attacks hardly constitutes a winning gambit.
The upshot is, even the friend who has received the capability to talk to our chat tool doesn't get much traction. He can annoy us (by crashing minChat), but that is pretty much the only interesting thing he can do....or so it seems on first review.
Let's see if there is anything exotic an attacker could do by combining several of these unimportant attacks. First let us make the problem more concrete. Suppose Bob has a crush on Alice, who is dating Ted. Bob decides to make Alice think that Ted doesn't really like her. So he gives Alice his chat reference to Ted, and tells her that, when using this reference, Ted will think she is really Bob, and so Alice will be able to get Ted to "speak candidly" (to Bob) about her. At this point, Bob uses the "send" method on Ted's computer to send messages that Alice will read, messages that look to Alice like Ted is sending them! Bob sends cruel jokes about Alice, and Alice breaks up with Ted.
Hmmm....it looks pretty bad for Ted. Did we just find a really cool attack?
Well, sort of. First there is the risk that Ted will notice that his chat tool is, without any help from the owner sending offensive messages to Bob. Ted might then wonder what is going on. But more fundamentally, Bob has a much simpler scheme for messing with Alice's headspace, if Alice is open to this type of attack.
Far simpler for Bob would be to create 2 brand new chat capabilities, one that is Bob's FakeTed, and the other is Bob's FakeBob. He has Alice start up as FakeBob, and Bob himself starts up FakeTed. Now Ted is completely out of the loop, Bob can send any horrid thing to Alice he wants to send, without interference.
So is this a fundamental flaw with the entire E security approach? Not exactly. This attack can be made regardless of the technology being used -- Bob could have created 2 Yahoo Instant Messenger accounts and played the same game. Indeed, variations on this attack were used repeatedly by William Shakespeare throughout his career to create both high tragedy and low comedy. The fundamental problem comes when you trust a middleman as the sole validation of the authenticity of messages coming from another person. This is a human problem, not a technology problem. The best source of contact information for a person is the person him/herself; trust a third party at your own risk. Often the risk makes sense, but don't ever forget there is a risk there, whether you are using capability security or just a parchment and quill.
We have come full circle to the conclusion that minChat doesn't have any serious security breaches. Nonetheless, there are a couple of annoying things Bob may do to Ted. These annoyances can be traced directly to the presence of five messages in the chatController: there are only two messages out of the five that the friend is "supposed" to use. Even forgetting about security reviews for a moment, it is clear that, if the friend is only supposed to have two methods, he should not have five methods at his disposal. This make sense from a simple modular design point of view, it limits the number of mistakes you can make. For the security reviewers, this is an enormous win, because it reduces the number of methods to examine by 60%. This is a substantial gain, given how circuitous and complicated security analysis can become.
So how do we cut the number of methods available to the friend? By using a facet of the chatController that only has the appropriate methods:
# E syntax
# Place the following object above the binding of the chatController
def chatFacet {
to receive(message) {chatController.receive(message)}
to receiveFriend(friendRcvr) {chatController.receiveFriend(friendRcvr)}
}
# Now modify the chatController's save method to save the chatFacet, not the chatController:
to save(file) {file.setText(makeURIFromObject(chatFacet))}
# and similarly modify the load method to send the chatFacet not the chatController:
to load(file) {
bind friend := getObjectFromURI(file.getText())
friend <- receiveFriend(chatFacet)
}
This version of minChat is more secure (only a little more secure, though, since it was pretty secure already), and vastly easier to review.
There are still a number of interesting lessons we can learn here by considering some variations on minChat. The first variation is based on the old adage, "You can write FORTRAN in any language." This is true in E as well, but the implications can be grievous.
Let's go back the minChat without the chatFacet, and consider the following small change to the program. Suppose that the save method in the chatController, instead of receiving a file object from the chatUI and creating the uri string itself, received the file path and the uri string from chatUI:
# E syntax
to save(filePath :String, uri :String) { <file: filePath>.setText(uri)}
This is a fairly FORTRAN-ish way of accomplishing the goal: since FORTRAN doesn't have objects, it may seem perfectly obvious and reasonable to just send the path to the file and let the chatController figure out how to write to that file on its own.
However, in this situation, it is a disaster. Without the chatFacet, now the friend at the far end has a method he can call that allows him to write any data he wants, into any location the user can reach on his own computer:
# on the friend's machine
# put the code for the SubSevenServer Trojan in a string
def code :String := "@#$!...and so on, the character representations of the code bytes"
# The friend's friend is our poor user
# Put the trojan horse in the user's startup profile
friend <- save("~/startup/sub7.exe", code)
Now we're having some fun! Every time the user logs onto his computer, the friend gets full control of all his resources.
What went wrong here? There are several ways of slicing the pie and pinning different pieces of the code with blame. One we have already identified: we failed to follow POLA in allowing the friend to call the save method in the first place. But another one is just as crucial: we failed to follow POLA twice with this revised version of the save method that gets 2 strings.
We failed first by sending the save method an insufficient amount of authority (remember, Least Authority also means Adequate Authority). We did this by sending the chatController a string describing the file's path rather than just sending him the file itself. As a consequence, the chatController had to use special powers gotten from elsewhere to translate the description into a file object.
Once the chatController had to do this translation, it became vulnerable to the Confused Deputy Attack, a classic in security literature. The Confused Deputy attack is the aikido move of computer security: the attacker persuades the target to use the target's own authority against itself. Confused Deputy attacks are very hard to pull off as long as a single reference serves as both the designation of the object and the authority to use the object (as is always the case with the objects in a capability-based programming language). But when the authority to use the object is separated from the designation, as it is when a path string is being used to describe the file, you are in deep trouble.
Simply sending the file object rather than the file description solves this problem. The friend at the far end has no way to send an object that will be interpreted by the save method as a local file, and so the friend gets no traction whatsoever as long as the save method is expecting the object not the description. This leads to the general rule:
When you must read in the descriptions of objects rather than the objects themselves, translate them into capability secure objects at the earliest possible moment. When you must write out descriptions of objects rather than the objects themselves, postpone the translation of the object into a description until the last possible moment.
This rule is not merely good for secure, capability-oriented programming. Rather, it is good for all object-oriented programming. Manipulating the descriptions of objects is much less reliable, and much more error prone, than direct manipulation of the object itself--which is both why crackers like it, and object oriented designers avoid it. This exemplifies an interesting truth: capability oriented development is often just object oriented development taken seriously.
Working directly with the object, not a description, is a rule the author learned the hard way. A security review team ripped numerous holes in a single piece of the author's software, simply because the author foolishly worked with descriptions rather than with objects. The story is told in excruciating detail in the DarpaBrowser Security Review. If you are one of the rare and lucky individuals who are able to learn from other people's mistakes, rather than having to make the mistakes firsthand yourself first before you really get it, this is a magnificent lesson to learn secondhand.
In large software systems, most objects are constructed inside emakers, rather than in the main E program body. Consequently they come to life with severely restricted, POLA-oriented authority(i.e., only the authority we send to them when we construct them, which tends to be what they need but not anything else). Let us emulate this characteristic of large systems in our little minChat by making a paranoid version of the chatFacet:
This is grossly broken syntax for eval
# E syntax
def makeFacet := e`
def makeFacet(chatController) {
def chatFacet {
to receive(message) {chatController.receive(message)}
to receiveFriend(friendRcvr) {chatController.receiveFriend(friendRcvr)}
}
return chatFacet
}
`.eval(universalScope)
def chatFacet := makeFacet(chatController)
The universalScope eval method creates the makeFacet function in strict confinement, identical to the confinement imposed on emakers. Consequently, the only authority the chatFacet has is the authority handed in, namely, the authority to send messages to the chatController.
Now let's assume we are running a more complex program, with a more complex component, i.e., a component that is not so simple we can just look at it and see it doesn't do anything very exciting. Now let's further assume that by some extraordinary bit of legerdemaine the attacker succeeds in completely subverting the component (the chatFacet in our little example). How terrible is the disaster?
As we have already shown, there is no disaster. Even if the attacker gets direct access to the chatController, there still isn't anything interesting he can do. We have achieved defense in depth. A typical penetration of our system gives the attacker only a limited access to other objects, which in turn must be individually penetrated.
Compare this to the situation now typical in software engineering. Even with the most modern "secure" languages like Java and C#, an attack can acquire the full authority of the program by subverting even the most inconsequential object. With tools like access control lists and firewalls, we engage in "perimeter defense", which is more correctly described as "eggshell defense". It is like an eggshell for the following reason: while an eggshell may seem pretty tough when you tap on it, if you can get a single pinhole anywhere in the surface, you can suck out the entire yoke. No wonder cybercrackers laugh at our silly efforts to defend ourselves. We have thrown away most of our chances to defend ourselves before the battle even begins.
As discussed in the minChat security review, facets are objects that act as intermediaries between powerful objects and users that do not need (and should not be granted) its full power. We saw a facet in use in the audited version of the eChat program, where the chatFacet. was interposed between the chatController and the remote chat participant. Here is a little general-purpose facet maker:
# E sample
/**
* <param> target is the underlying powerful object
* <param> allowedMethods is a map. The key is a method name, the value
* is the set of allowed numbers of arguments. So ["receive" =>[0, 2].asSet()]
* would allow the receive method, with either 0 or 2 arguments, to be forwarded.
**/
def makeFacet(target, allowedMethods) {
def filteringFacet {
match [verb, args] {
if (allowedMethods.maps(verb) &&
allowedMethods[verb].contains(args.size())) {
return E.call(target, verb, args)
}
}
}
return filteringFacet
}
def chatController
def chatFacet := makeFacet(chatController, ["receive"=>[1].asSet(),
"receiveFriend"=>[1].asSet()])
Facets of this type can be retrofitted onto an existing system. We did this with very little effort for minChat with the chatFacet, but the technique works for far more complicated problems as well. The capability-secure windowing toolkits that E has placed on top of Swing and SWT uses facetization as the main tool.
Facets can be made much more sophisticated in their restrictions on access to their underlying object. You can make a facet that logs requests and sends email when certain methods are called. In an SEC-regulated stock exchange facet, you might wish to grant the capability to make a trade only from 9AM to 3PM excluding weekends and holidays.
One interesting example is the use-once facet, which allows the holder of the facet to use the facet only one time. For example, this version of a chatReceiver only allows a single message to be sent:
# E sample
def onceOnlyReceiver(baseChatController) {
var chatController := baseChatController
def onceOnlyReceiver {
to receive(text) {
chatController.receive(text)
chatController := null
}
}
}This version will throw an exception back to the sender of a second message.
It can be tempting in a facet to suppress a couple of powerful methods in the underlying powerful object and delegate the rest.
# E sample
def powerfulObject {
to doPowerfulOperation() {
#do powerful operation
}
to doWeak1() {}
to doWeak2() {}
to doWeak3() {}
#....
to doWeak99() {}
}
def badFacet extends powerfulObject {
to doPowerfulOperation() {
#do nothing, no forwarding for the powerful operation, but forward everything else
}
} Avoid this. For a facet to remain secure during maintenance, it should never just delegate by default. If a new method is added to a powerful object (in this example, suppose powerfulObject is updated with a new method, doPowerful2()), it should not be exposed through the facet by default: rather, the facet must by default not expose it.
This risk can be exhausting to avoid, but it is always dangerous to accept. The first version of the capability windowing toolkit on Swing suppressed the handful of dangerous methods in the Java version 1.3 of Swing rather than explicitly allowing the safe methods, of which there were thousands. Within 30 days, the entire system was broken: the Java 1.4 Beta included hundreds of new, security-breaking methods. The only saving grace was that we had always known that the first version, thrown together in haste on weekends, was just a proof-of-principle and would have to be replaced. We just hadn't appreciated how soon replacement would be required.
If you wish to give someone restricted access to an object with facets, it is quite likely you will want to revoke access at some point as well.
The simplest way of making a capability revocable is to use a transparent forwarder that includes a revocation method:
# E sample
def revocableCapabilityMaker(baseCapableObject) {
var capableObject := baseCapableObject
def forwarder {
to revoke() {capableObject := null}
match [verb, args] {E.call(capableObject, verb, args)}
}
return forwarder
}Note that, even though the forwarder is nominally delegating all method invocations (except revoke()) to the capableObject, we cannot use the extends keyword to capture the behavior. "Extends" creates an immutable reference, so the fact that the capableObject is a var wouldn't allow you to revoke the delegating behavior. Instead we use the match[verb, args] pattern.
Capability revocation, like facet forwarding, can be based on complex sets of conditions: revoke after a certain number of uses, revoke after a certain number of days. This example uses a simple manual revocation. Indeed, this version is too simple to work reliably during system maintenance and upgrade, and a more sophisticated pattern is generally recommended:
# E sample
def makeCapabilityRevokerPair(baseCapableObject) {
var capableObject := baseCapableObject
def revoker {
to revoke() {capableObject := null}
}
def forwarder {match[verb, args] {E.call(capableObject, verb, args)}}
return [forwarder, revoker]
}
def powerfulObject {
#...big powers
}
def [power, revoker] := makeCapabilityRevokerPair(powerfulObject)
In this pattern, the authority of the object and the authority to revoke are separated: you can hand the power to revoke to an object you would not trust with the authority itself. Also, the separate revoker can itself be made revocable.
A sealer/unsealer pair makes it possible to use untrusted intermediaries to pass objects safely. Use the sealer to make a sealed box that can only be opened by the matching unsealer. E has built-in support for sealer/unsealer pairs:
? def makeBrandPair := <elib:sealing.makeBrand>
# value: <makeBrand>
? def [sealer, unsealer] := makeBrandPair("BrandNickName")
# value: [<BrandNickName sealer>, <BrandNickName unsealer>]
? def sealedBox := sealer.seal("secret data")
# value: <sealed by BrandNickName>
? unsealer.unseal(sealedBox)
# value: "secret data"
If you hold the unsealer private, and give the sealer away publicly, everyone can send messages that only you can read. If you hold the sealer private, but give the unsealer away publicly, then you can send messages that recipients know you created, i.e., you can use it as a signature. If you are thinking that this is much like a public/private key pair from public key cryptography, you are correct, though in E no actual encryption is required if the sender, recipient, brand maker, and sent object all reside in the same vat.
While the Brand is built-in, it is possible to construct a sealer/unsealer pair maker in E without special privileges. The "shared variable" technique for making the maker is interesting, and because the same pattern appears in other places (such as the Notary/Inspector, coming up next), we demonstrate it here:
# E sample
def makeBrandPair(nickname) {
def noObject{}
var shared := noObject
def makeSealedBox(obj) {
def box {
to shareContent() {shared := obj}
}
return box
}
def sealer {
to seal(obj) {return makeSealedBox(obj)}
}
def unsealer {
to unseal(box) {
shared := noObject
box.shareContent()
if (shared == noObject) {throw("invalid box")}
def contents := shared
shared := noObject
return contents
}
}
return [sealer, unsealer]
}
The variable "shared" normally contains the value "noObject", which is private to a particular sealer/unsealer pair and so could never the the actual value that someone wanted to pass in a sealed box. The unsealer tells the box to shareContent, which puts the content into the shared variable, from which the unsealer then extracts the value for the invoker of the unseal method. In a conventional language that used threads for concurrency control, this pattern would be a disaster: different unseal requests could rumble through, overwriting each others' shared content. But by exploiting the atomicity of operational sequences enabled by promise pipelining, this peculiar pattern becomes a clean solution for this, and several other security-related operations.
Suppose Bob is the salesman for Widget Inc. He persuades Alice to buy a widget. Bob hands Alice a Widget Order Form with a money-receiving capability. It is important to Bob that Alice use the form he gives her, because this particular form (which Bob got from Widget Inc.) remembers that Bob is the salesman who should get the commission. It is important to Alice that she know for sure that, even though she got the order-form from Bob, this is really a Widget Inc. order form, and not something Bob whipped up that will transfer her money directly to his own account. In this case, Alice wants to have Widget Inc. vouch for the order-form she received from Bob. She does this using an Inspector that she gets directly from Widget Inc. The Inspector is the public part of a notary/inspector pair of objects that provide verification of the originator of an object. To be vouchable, the orderForm must implement the startVouch method as shown here:
# E sample
####### Widget Inc. software #####
#vouching system
#returns a private notary that offers a public inspector
#throws problem if the object being vouched is not vouchable
def makeNotary() {
def nonObject {}
def unvouchedException(obj) {throw(`Object not vouchable: $obj`)}
var vouchableObject := nonObject
def inspector {
to vouch(obj) {
vouchableObject := nonObject
try {
obj.startVouch()
if (vouchableObject == nonObject) {
return unvouchedException(obj)
} else {
def vouchedObject := vouchableObject
vouchableObject := nonObject
return vouchedObject
}
} catch err {unvouchedException(obj)}
}
}
def notary {
to startVouch(obj) { vouchableObject := obj}
to getInspector() {return inspector}
}
return notary
}
#create Widget Inc's notary
def widgetNotary := makeNotary()
#Order form maker
def makeOrderForm(salesPerson) {
def orderForm {
# .... methods for implementing orderForm
to startVouch() {widgetNotary.startVouch(orderForm)}
}
return orderForm
}
#publicly available inspector object
#(accessible through a uri posted on Widget Inc's web site)
def WidgetInspectionService {
to getInspector() {return widgetNotary.getInspector()}
}
##### bob software #####
#scaffold for sample
def getOrderFormFromBob() {return makeOrderForm("scaffold")}
########## Alice's software to vouch for the order form she received from Bob #####
def untrustedOrderForm := getOrderFormFromBob()
def inspector := WidgetInspectionService.getInspector()
def trustedOrderForm := inspector.vouch(untrustedOrderForm)
"Proof of Purchase" is the simplest of a series of capability patterns in which the goal is not to transfer an authority, but rather to show someone that you have the authority so that they can comfortably proceed to use the authority on your behalf. In Proof of Purchase, the requester of a service is demonstrating to the server that the client already has capability that the server is being asked to use. Unlike the more sophisticated upcoming Claim Check patterns, the proof of purchase client is not concerned about giving the server an excess authority.
An interesting example of Proof of Purchase is Component.transferFocus(fromComponentsList, toComponent) found in the tamed Swing package. In standard Swing, the holder of any panel reference can steal the focus (with component.requestFocus()). Hence any keystrokes, including passwords, can be swiped from wherever the focus happens to be by sneaking in a focus change. This is a clear violation of POLA. You should be able to transfer focus from one panel to another panel only if you have references to both panels. If you have a reference to the panel that currently has the focus, then you can never steal the focus from anyone but yourself.
A natural-seeming replacement for component.requestFocus() would be requestFocus(currentFocusHolder). It was decided during the taming of Swing, however, that this would be breach-prone. If Alice transferred to Bob, not a panel itself, but rather a simple transparent forwarder on that panel, Alice could collect authorities to other panels whenever Bob changed focus.
Since we had a convenient trusted third party available that already had authority over all the panels anyway (the javax.swing.Component class object), we used the "proof of purchase" pattern instead. The globally available Component.transferFocus method accepts two arguments:
In the common case where a whole window lies inside a single trust realm, the client can simply present a reference to the whole window and be confident that the focus will be transferred.
The interesting thing about Component.transferFocus is that the recipient does not actually need the client's authority to reach the component that already has the focus. The Component class already has that authority. The client must send the authority merely to prove he has it, too.
A common activity that brings security concerns into sharp focus is the delicate dance we perform when we use a car valet service. In this situation, we are handing the authority over our most precious and costly possession to a teenager with no more sense of responsibility than a stray cat. The whole valet system is a toothsome exercise in POLA.
In this example, we focus on the sequence of events and trust relationships involved in reclaiming our car at the end of the evening. The participants include the car owner, the valet service itself, and the random new attendant who is now on duty.
We have already chosen, for better or for worse, to trust the valet service with a key to the car. As the random new attendant comes running up to us, we are reluctant to hand over yet another key to the vehicle. After all, this new attendant might actually be an imposter, eager to grab that new Ferrari and cruise out over the desert. And besides, we already handed the valet service a key to the car. They should be able to use the key they already have, right?
Meanwhile, the attendant also has a trust problem. It would be a career catastrophe to serve up the Ferrari from his parking lot if I actually own the dented 23-year-old Chevy Nova sitting next to it.
In the physical world, we use a claim check to solve the problem. We present to the attendant, not a second set of car keys, but rather, a proof that we have the authority that we are asking the attendant to use on our behalf.
# E sample
def claimMgr := {
def carKeys := [].asMap().diverge()
def claimMgr {
to makeClaim(car) :any {
# the claim object has no interesting
# properties except it has
# a unique unforgeable identity
def claim {}
carKeys[claim] := car
return claim
}
to reclaim(claim) :any {
def car := carKeys[claim]
carKeys.removeKey(claim)
return car
}
}
}
def attendant {
#return a claim check when car is parked
to parkCar(car) :any {
# ...code to park the car...
return claimMgr.makeClaim(car)
}
to retrieveCar(claim) :void {
def car := claimMgr.reclaim(claim)
# ...code to retrieve car...
# no need to return the car reference, the owner of the claim
# presumably already has such a reference
}
}
def car {}
def carOwner := {
var claim := null
def carOwner {
to letAttendantParkCar() :void {
claim := attendant.parkCar(car)
println(claim)
}
to getCarFromAttendant() :void {
println(attendant.retrieveCar(claim))
}
}
}
carOwner.letAttendantParkCar()
carOwner.getCarFromAttendant()
Suppose the owner of the car loses the claim check. He can still prove he owns the car by presenting another key. In software, this situation would be comparable to the situation in which the owner hands the attendant a direct reference rather than handing the attendant merely the claim check. The car owner has violated his own security, but it is hard to visualize situations in which this is not a reasonable argument to pass to express his desire. We can cover this case with a small modification to the claimMgr's reclaim method:
to reclaim(claim) :any {
try {
def car := carKeys[claim]
carKeys.removeKey(claim)
return car
} catch prob {
if (carKeys.contains(claim)) {
return claim
} else {throw(prob)}
}
}
What should the claimMgr return if the carOwner hands in a car reference for reclaim, if the car is not actually in the claimMgr's parking lot? The answer depends on the application, but such a situation violates the expectations of all the participants sufficiently that throwing an exception seems the safest choice.
We are far from done with claim checks. A careful car owner will not actually hand his claim check to the parking lot attendant. Rather, he will merely show the claim check to the attendant. After all, if you hand the claim check over to an imposter, the imposter can turn around and hand the claim check to a real attendant, pretending that he is the owner of the car.
The problem is somewhat different in cyberspace. The good news is, in cyberspace the attendant doesn't return a reference to the car just because you hand him the claim check. Instead, he merely performs the action you ask of him with the car. This set of actions is limited by the attendant's willing behavior. So the claim check is not as powerful a capability as the car itself. But it can still be a powerful capability. And the bad news is, in cyberspace, you can't just "show" it to the attendant. You have to give it to him.
Here we demonstrate a "nontransferable" claim check. This claim check can be handed out at random to a thousand car thieves, and it does them no good. The trick is, before handing over the claim check, the owner inserts into the claim check a reference to the individual he is treating as an attendant. The ClaimMgr compares the person to whom the owner handed the claim, with the attendant who eventually hands the claim to the claimMgr . If these two people are the same, then the owner handed the claim directly to the attendant. Otherwise, there was an intermediary party, and the request should not be honored.
# E sample
def makeBrandPair := <elib:sealing.makeBrand>
def valetService := {
def [claimSealer, claimUnsealer] := makeBrandPair("claim")
def claimMgr {
to makeClaim(car) :any {
def transferableClaim {
to makeNontransferableClaim(intendedRecipient) :any {
return claimSealer.seal([car, intendedRecipient])
}
}
return transferableClaim
}
to reclaim(claim, actualRecipient) :any {
def [car, intendedRecipient] := claimUnsealer.unseal(claim)
if (actualRecipient == intendedRecipient) {
return car
} else {throw("claim not transferable, invalid attendant")}
}
}
def valetService {
to authorizeAttendant(attendant) :void {
attendant.setClaimMgr(claimMgr)
}
}
}
def makeAttendant() :any {
def claimMgr
def attendant {
to setClaimMgr(mgr) :void {bind claimMgr := mgr}
to parkCar(car) :any {
# ...code to park the car...
return claimMgr.makeClaim(car)
}
to retrieveCar(claim) :void {
def car := claimMgr.reclaim(claim, attendant)
# ...code to retrieve car...
# no need to return the car reference, the owner of the claim
# presumably already has such a reference
}
}
return attendant
}
def legitAttendant := makeAttendant()
valetService.authorizeAttendant(legitAttendant)
def carThief {
to parkCar(car) :any {
return println ("Ha! stole the car")
}
to retrieveCar(claim) :void {
try {
legitAttendant.retrieveCar(claim)
println("Ha! didn't get car, but got control")
} catch prob {println(`rats! foiled again: $prob`)}
}
}
def carOwner := {
def car{}
var claim := null
def carOwner {
to letValetPark(attendant) :void {
claim := attendant.parkCar(car)
println(claim)
}
to letValetRetrieve(attendant) :void {
def noTransferClaim := claim.makeNontransferableClaim(attendant)
println(attendant.retrieveCar(noTransferClaim))
}
}
}
carOwner.letValetPark(legitAttendant)
carOwner.letValetRetrieve(carThief)
carOwner.letValetRetrieve(legitAttendant)
Note an important implication in this pattern: the ClaimMgr winds up in a position to accumulate references to all the entities a claim check holder treats as an attendant. In this example, the ClaimMgr gets a reference to the carThief as a result of the carOwner's casual handing out of claim checks. It seems unlikely that the ClaimMgr can harm the carOwner's interests with this reference, but in other circumstances the reference may not be so harmless. To reduce our risk exposure to the alleged attendants to whom we hand the claim check, we have increased our risk exposure to the ClaimMgr.
Note also the limitations on the nontransferability of this pattern. The carOwner can still transfer authority, either by handing someone a reference to the car, or by handing out the transferableClaim from which nontransferableClaims can be made. The nontransferability is voluntarily chosen by the carOwner. You can't prevent people from delegating their authority, and even this pattern doesn't change that fact.
Let us consider a somewhat different problem. Suppose that different attendants are trusted to handle different cars by the valet service. One valet has a motorcycle license and parks all the Harleys. Another has a multi-engine pilot's license and parks all the Boeing 747s. Of course, the one with the motorcycle license is a teenager who has always wanted to try his hand at parking a 747, and knows his lack of experience is not a problem. In this situation, each attendant has a different set of authorities at his command; just because you hand your claim check to a legit attendant doesn't mean the valet service thinks it would be a good idea to let that attendant drive your vehicle. A more generalized way of stating the problem is, in this case the authorities of the individual receivers of the claim checks vary, and management of the individual receiver's authorities is beyond the scope of what the ClaimManager should be trying to figure out. After all, the individuals know all their own authorities; it would be poor design (and unmaintainable at scale) for the ClaimManager to try to duplicate this information.
This situation, while not a part of the real-world valet service problem, has a significant area of application in cyberspace. This area is Voluntary Oblivious Compliance, or VOC. Let us consider a more sensible example. Alice works for HPM, and Bob works for Intil. HPM and Intil are often competitors, but Alice and Bob are working on a joint project from which both companies expect to profit handsomely. The Official Policy Makers of HPM have identified a number of documents which can be shared with Intil, and indeed have forwarded to Intil references to the allowed docs. Alice wants to refer Bob to a particular HPM document, but only if sharing the document is allowed under the HPM policy. In this case, the VOCclaimCheck that Alice sends to Bob demands that Bob demonstrate to HPM's ClaimMgr that he already has authority on the document before fulfilling the claim. To prove his authority, Bob sends to the ClaimMgr all the HPM doc authorities he has (i.e., the list of docs HPM handed to Intil). Only if both Alice and Bob already have authority on the object does Bob get the document. This is sometimes called the Loan Officer Protocol, in reference to the old adage that a loan officer will not give you the loan unless you first prove that you don't need it.
Since bob can't get the reference unless he proves he doesn't need it, we can now see why this is a pattern of Voluntary Oblivious Compliance. It is voluntary, because Alice could just send Bob the document and circumvent the system. And it is oblivious, because Alice doesn't need to know whether a document can be shared with Bob before sending it.
Going back to the parking lot attendant who wants to park the 747, he has to demonstrate to the ClaimManager that he has the keys to the 747 before the Claim Manager will let him go for it. The owner of the 747 is much relieved.
Humorous as the 747 example might be, we will now switch to the scenario with Alice, Bob, HPM, and Intil for our sample code. The basic strategy is that the ClaimMgr not only examines the claim check from Alice, it also examines the list of authorities that Bob hands over to see if any of Bob's authorities match the one inside Alice's claim.
# E sample
def makeBrandPair := <elib:sealing.makeBrand>
def [claimSealer, claimUnsealer] := makeBrandPair("claim")
def HPMDocClaimMgr {
to makeClaim(doc) :any {
return claimSealer.seal(doc)
}
to matchClaim(claim, listOfCandidates) :any {
def doc := claimUnsealer.unseal(claim)
for each in listOfCandidates {
if (doc == each) {return each}
}
throw("No match from claimCheck to candidate auths")
}
}
def intilSharedDocs := [def doc1{}, def doc2{}, def doc3{}]
def bob {
to lookAtHPMDoc(claim) :void {
try {
def doc := HPMDocClaimMgr.matchClaim(claim, intilSharedDocs)
println(`reading doc: $doc`)
} catch prob {
println(`can't read: $claim`)
}
}
}
def alice {
to sendDoc(doc, recipient) :void {
def claim := HPMDocClaimMgr.makeClaim(doc)
recipient.lookAtHPMDoc(claim)
}
}
alice.sendDoc(doc2, bob)
alice.sendDoc(def doc4{}, bob)
The oblivious claim check pattern can require scattering even more authority to the winds than the nontransferable claim check. The recipient of the claim check (bob) needs to have all the authorities that alice might give to him, rather than merely the ones she actually does give to him. And the trusted third party who matches the claim check against the list of possibilities (the HPMDocClaimMgr) gets to accumulate authority to everything that bob thinks alice might be trying to send him. In the example scenario, this is just fine. But some care is required.
An elegant style of using VOC claim checks is to set up the pattern as a pair of guards. Alice would use the ClaimCheck guard to send a reference to Bob, and Bob would use the CheckedClaim guard, with the list of candidate references, to receive the authority. We will show the implementation of the ClaimCheck and CheckedClaim guards in Advanced Topics when we talk about writing your own guards, but their usage would appear as follows:
# E syntax
def bob {
to lookAtHPMDoc(doc :CheckedClaim[intilSharedDocs]) :void {
if (!E.isBroken(doc)) {
println(`reading doc: $doc`)
} else {println(`can't read: $claim`)}
}
}
def alice {
to sendDoc(doc, recipient) :void {
recipient.lookAtHPMDoc(doc :ClaimCheck)
}
}
The ClaimCheck guard coerces the doc into a sealed representation of itself. The CheckedClaim guard unseals the claim and matches it against the authorities in the intilSharedDocs list. As a guard, it is better style to return a broken reference than to throw an exception if something goes wrong. In the parameter guard, if an exception were thrown, the recipient would have no chance to do anything with the problem, the exception would be thrown directly back to the sender before the recipient was even aware there was an issue.
It is possible to make nontransferable oblivious claim checks as well. We leave the code as an exercise for the reader.
The powerbox pattern collects diverse elements of authority management into a single object. That object, the powerbox, then becomes the arbiter of authority transfers across a complex trust boundary. One of the powerbox's most distinctive features is that it may be used for dynamic negotiations for authority during operation. The less trusted subsystem may, during execution, request new authorities. The powerbox owner may, in response to the request, depending on other context that it alone may have, decide to confer that authority, deny the authority, or even grant the request authority after revoking other authorities previously granted.
The powerbox is particularly useful in situations where the object in the less trusted realm does not always get the same authorities, and when those authorities may change during operation. If the authority grant is static, a simple emaker-style authorization step suffices, and a powerbox is not necessary. If the situation is more complex, however, collecting all the authority management into a single place can make it much easier to review and maintain the extremely security-sensitive authority management code.
Key aspects of the powerbox pattern include:
Not all kinds of objects in the Java API can be made directly revokable at this time, because an E revokable forwarder cannot be used in all the places where an actual authority-carrying Java object is required. For example, the untrusted object may want to create a new image from an url. The natural way of doing this would be to call the Java Icon class constructor with (url) as the argument. But if an E forwarder were handed to this class constructor, the constructor would throw a type exception.
Because of this anomally, different solutions are required in different situation. For the icon maker, it might be acceptable to require that the untrusted object read the bits of the image from the url (through a revokable forwarder) and then convert the bits into an icon.
In the following example, the less trusted object may be granted a Timer, a FileMaker that makes new files, and a url. The object may request a different url in the course of operations, in which case the powerbox will ask the user for authorization on the objects's behalf; the old url is revoked, and the new one substituted, so that the object never has authority for more than one url at a time. The object that operates the powerbox may revoke all authorities at any time, or it may choose to revoke the Timer alone. Finally, the operator of this powerbox may, for reasons external to the powerbox's knowledge, decide to grant an additional authority during operations, an authority whose nature is not known to the powerbox.
# E sample
# The authorized url may change during operations, so it is a var
def makePowerboxController(optTimer,
optMakeFile,
var optUrl,
optMakeUrl,
makeDialogVow) {
# In the revokers map, the object being revoked is the key, the revoker
# is the value. Note it is the actual object, not the forwarder to the
# object, that is the key.
var revokers := [].asMap()
# when a revokable forwarder is made, the revoker is automatically
# placed in the map of revokers
def makeRevokableForwarder(object) :near {
var innerObject := object
def revoker {
to revoke() {innerObject := null}
}
revokers with= (object, revoker)
def forwarder {match [verb, args] {return E.call(innerObject, verb, args)}}
return forwarder
}
# makeFileFacet is exposed to less trusted realm; guard the path it receives
# This simple file facet only supplies 2 methods that return immutables
# If the file handed out mutable objects, such as streams, these would have
# to be wrapped with revokableForwarders as well.
def makeFileFacet(path :String) :near {
def theFile := optMakeFile(path)
def fileFacet {
to getText() :String {theFile.getText()}
to setText(text :String) {theFile.setText(text)}
}
return makeRevokableForwarder(fileFacet)
}
# makeFile is actually handed to the less trusted object
# It is either null or a revokable forwarder for makeFileFacet
# In other words, makeFile is a revokable maker of revokable file facets
def makeFile
if (optMakeFile == null) {
bind makeFile := null
} else {
bind makeFile := makeRevokableForwarder(makeFileFacet)
}
# Like the file facet, this url facet only supports methods that return
# immutables.
def makeRevokableUrlFacet(optUrl) :near {
if (optUrl == null) {
return null
} else {
def urlFacet {
to getBytes() :pbc {optUrl.getBytes()}
}
return makeRevokableForwarder(urlFacet)
}
}
# Return a vow for a new url
# Use dialog with user to determine if new url should be granted
# Vow resolves to null if anything goes wrong
def makeRevokableUrlVow := {
def makeUrlVow(requestedUrl :String, why :String) :vow {
def urlDialogVow :=
makeDialogVow("Url Request",
`<html>
Confined Object requesting url for this reason:<p>$why
</html>`,
requestedUrl,
["Grant", "Refuse"])
return when (urlDialogVow) -> {
if (urlDialogVow.getButton() == "Grant") {
optUrl := optMakeUrl(urlDialogVow.getText())
makeRevokableUrlFacet(optUrl)
} else {null}
} catch prob {null}
}
return makeRevokableForwarder(makeUrlVow)
}
var caps := [].asMap()
caps with= ("TIMER", makeRevokableForwarder(timer))
caps with= ("FILEMAKER", makeFile)
caps with= ("URL", makeRevokableUrlFacet(optUrl))
def powerbox {
# any of these capabilities may be null, i.e., all are optional
# in a powerbox, strictly at the whim of the powerbox operator
# who created the powerboxcontroller and the powerbox
to optCap(key :String) :any {return caps.get(key, null)}
# When the less trusted object requests a new url, the
# old url is immediately revoked, and the promise for the
# new url is substituted
# If the powerbox has revokedAll, any attempt to requestUrl
# will throw an exception back to the untrusted object
# The "why" parameter is the less trusted object's justification
# for needing the url
to requestUrl(requestedUrl :String, why :String):vow {
if (optUrl != null) {revokers[optUrl].revoke()}
def revokableUrlVow := makeRevokableUrlVow(requestedUrl, why)
caps with= ("URL", revokableUrlVow)
return revokableUrlVow
}
}
def powerboxController {
to revokeAll() {
for each in revokers {each.revoke()}
}
to revokeTimer() {revokers[timer].revoke()}
# Confer an additional capability during execution
to conferCap(key, cap) {
caps with= (key, makeRevokableForwarder(cap))
}
to getPowerbox() {return powerbox}
}
}
# now, show how to create a powerbox and hand it to an untrusted object
def makeUntrustedObject(powerbox) {
def timer := powerbox.optCap("TIMER")
def urlVow := powerbox.requestUrl("http://www.skyhunter.com")
def untrustedObject {
#...
}
return untrustedObject
}
def powerboxOperator() {
def makeDialogVow {#...construct dialog vow
}
# use real objects, not nulls, in typical operation, though nulls are valid arguments
def controller := makePowerboxController(null,
null,
null,
null,
makeDialogVow)
def powerbox := controller.getPowerbox()
def untrusted := makeUntrustedObject(powerbox)
# later, confer an additional authority
def special
controller.conferCap("SPECIAL", special)
# when done with the untrusted object, revoke its powers
controller.revokeAll()
}
Both the file facet and the url facet restrict the user to methods that return immutables. If these facets returned mutables, like streams, then those mutables would also have to be wrapped in revokable forwarders. In the general case, this requires the use of a membrane. write section on membranes, and link
E's encrypted authenticated references ensure that random attackers get no traction trying to break into your distributed system. As mentioned in the earlier minChat audit, this means that all interesting break-ins will be made by "insiders", i.e., people who have been granted some capabilities inside your system. We have talked so far about low-level attacks that attempt to acquire or use inappropriately conveyed authority. At a higher conceptual level, people can try to present themselves as someone else and get your user to intentionally, but erroneously, grant other information or capabilities.
minChat dodged this problem because it is a 2-person chat system: the person at the other end of the line will have difficulty convincing your user he is someone else, because your user knows the one and only person given the URI (because he used PGP encryption as described earlier to ensure that only one person got it). But in a multi-person chat system, Bill could try to pretend to be Bob and lure your user into telling him Bob's secrets.
There is no single all-purpose solution to this risk. However, one critical part of any specific solution must be this: Do not allow the remote parties to impose their own names for themselves upon your user.
link for pet names
The general solution is to use pet names, i.e., names your user chooses for people, regardless of how those people might name themselves. Of course, it is reasonable to allow the other people to propose names to your user (called nicknames), but the user must make the final decision. The user must also be able to change her mind at a later date, when her list of participants grows to include not only Bob Smith but also Bob Jones, and she must disambiguate the pet name "Bob" she had had for Smith.
The general layout of such a pet name setup might be as follows:
pet name sample
Earlier we presented a single-computer racetrack game as an example. Here we present a distributed version of Racetrack, allowing people to compete over the network. And, just for a bit of spice, we have thrown in the assumption that one of the drivers will be Satan. Satan will, of course try to win...or try to wreck other people's cars (keeping his own safe, of course)...or try to ensure that someone Satan favors will win, presumably someone who has offered up his soul in exchange for victory.
Fortunately, as we all know from the story of the time the Devil came down to Georgia, as long as we can force Satan onto a fair playing field, he can indeed be beaten. We will need scrupulous capability security, however, to keep that field fair.
As noted earlier, security within the E framework is largely a matter of architecture. Therefore we will look at the issues for a secure distributed architecture before we look at the code.
First of all, where do we want to put the divide between the server functionality and the client functionality? If the author of this book owned stock in a diskless workstation company, we would undoubtedly present a "thin client" architecture, wherein everything except user interface code lived on the server. We will look at other possibilities here, however, since other possibilities abound.
E allows us to distribute the computation to suit our needs. We could build a thin client, a thin server, or anything that lies between these extremes. The particular architecture strategy we will use for this security example, with our souls on the line because Satan is coming to the party, is a "thin network" layout. We will minimize the amount of information being sent back and forth between the server (owned by the trusted third party who authoritatively asserts who won the race) and the clients (owned by the race car drivers, including Satan). We will especially minimize the number and power of the references (capabilities) the clients have on objects on the server, since this is the major path Satan will have for attacking the system.
With this as the architectural goal, we can reduce the information flow from client to server to merely proposals for the acceleration changes for the cars. Each driver (i.e., each client) should have access to the server only to the extent of being able to tell their respective cars on the server what acceleration the driver proposes. The clients must absolutely not have access to any cars on the server but their own. Consequently, a method like, "setAccelerationForCar(name)" cannot be the way car directions are specified: Satan, upon learning the names of the cars, could easily start specifying accelerations for everyone's cars, not just his own.
The server will initially send out maps to the clients: rather than giving the clients read authority on the raceTrack's map, the server will send each client an immutable pass-by-copy description of the map and the car starting locations. Sending out immutable copies that are locally accessible will simplify the code, reduce the communication needed during the actual game, and most significantly, will eliminate a set of interfaces that would need security audits.
As in the single-computer version of the game, the raceTrack will collect accelerations input by the drivers until it has accelerations from all the drivers, then it will authoritatively describe those accelerations to all the drivers. Note the careful language we use to describe the accelerations being sent around the system: driver clients send proposed accelerations for their own accelerations to the server; the server sends authoritative accelerations to the clients. Satan's easiest technique for assaulting the track would be to simply modify his client so he can specify accelerations beyond -1,0,+1; it would be a substantial advantage for Satan to be able to accelerate +15 in a single turn. So the server must validate the accelerations, and not rely on validation in the user interface as the single-computer version did.
As noted earlier when discussing pet names, another clever attack is to forge someone else's signature. If Satan could pretend to be whichever driver actually wins the race, and persuade people to believe him, it would be as good as actually having won (and quite a bit sweeter as well). This racetrack does not use the full-strength pet name strategy to prevent this. Rather, the server assigns each car a name. No negotiation is allowed.
So far, we have only considered attacks that can be made from the client on the server. Are there any security concerns going the other way, i.e., secret information that, if acquired by Satan, would improve his situation? If this were a game of poker, not a game of racetrack, the answer would be yes, and we would have to audit the data being sent to the client as well as the capabilities. For racetrack, however, this is not an issue.
What else can Satan do to sabotage the race? He can, of course, flood the server with spurious messages in a denial of service attack. As stated earlier, E by itself cannot prevent this. But for this critical game, we can assume the trusted third party operating the server will work with the ISPs to prevent it from inhibiting game play. There is one other denial-of-service attack Satan can undertake, the simplest attack of all: he can simply walk away from his computer, leaving the game with one car that never gets new acceleration instructions. In the single-computer version of racetrack, the game never stepped forward to the next turn until all the cars had new instructions. Left unchanged, this would allow Satan to starve all the drivers to death and then claim victory. Consequently, we must put a time limit into the distributed version, so that game play continues even if one of the players loses his soul in the middle of the competition.
That completes the security-oriented architectural considerations for the racetrack. One other architectural note: The cars will be implemented using the unum design: each client system will have a client presence of the car locally instantiated, synchronized in each game turn with the host presence of the car that resides on the server. As noted briefly earlier, the unum is a supercharged version of the Summoner pattern; in the Summoner pattern synchronization is not performed. Before this becomes cast in stone, I don't love "host presence". "Authoritative presence" is too long. How 'bout "essence"? Or "true presence"?
racetrack code
There is a problem with the programs we explored in the last chapter. The MarketPlace server, the RaceTrack server, and eChat are not persistent: once you shut them down they are gone forever. You can start up a new server with the same code, but it will not return to life with the same uri it had before. As a consequence, if you restart a server, you must redistribute the capability to access that server to all the users. This is a logistical nightmare. When we restart one of these servers, it must be able to restart with the same vatID and object IDs it had before.
If E simply allowed objects to remember their own uris, there would be extensive opportunities for object forgery in the mobile code systems described in the next chapter. To prevent such forgeries, E uses swiss bases for persistence.
The key to running code from people you do not trust totally is to run those programs with the least possible authority, and to refuse to run the program if it requires different authority than what you are willing to grant. Straight E programs run with all your authority on your system, just like Java applications and C applications. However, emakers come to life in a confined world totally controlled by the E virtual machine. They have no capabilities except those explicitly passed in by the outside world. This forms the beginning of a trustworthy way of running untrusted software.
Java introduced the concept of a sandbox for its applets. The sandbox is a space in which no access to dangerous powers is available. To give greater flexibility they introduced the security manager and the ID badge architecture--if the user concludes the applet came from a "trusted source", the applet can get your ID badge and roam wantonly through your system.
A java application that runs in a sandbox is called an applet. An E emaker that embodies a full program is called a caplet. Caplets are not trapped in sandboxes. Instead, you may think of caplets as being trapped in safety deposit vaults. The caplet is started inside the vault, surrounded by hundreds of safety deposit boxes--but not one of the boxes is accessible unless and until you give the caplet a key to that particular box.
Caplets can be thought of as highly stylized emakers. They are granted one authority when they are brought to life: the authority to talk with a powerbox, a trusted part of the caplet launching system that holds, negotiates, and mediates authorities for the caplet. Caplets can, through the powerbox, request needed authorities from the user.
TBD when the caplet framework is good enough to document.
Some of the items in this potpourri of topics are not difficult to understand or use, they are simply used less often than the main features of the language.
MenuMaker
E installations include the props.txt properties file, chock full of information that is occasionally useful. You can access it through the interp object. In this example, we get the home directory for the E installation, which can be useful if you wish to store data in a known location to be shared by multiple E programs:
def ehomePath := (interp.getProps()) ["e.home"]
The properties are stored in a map which can be accessed in the usual manner. For a complete listing of the information in the props.txt file, open yours up and take a look.
So you think it would be cool to access your own data using the syntax of the style <getter:description>. No sweat: just implement an object with the single method get(description), and assign it to a variable with the name xxx__uriGetter:
# E sample
def myDir := <file:/home/usr/me/myDir>
def myDir__uriGetter {
to get(objectPath) {
return myDir[objectPath]
}
}
def myFile := <myDir:theFileOfMyDreams.txt>EAction and the trouble with listener callbacks
the entropy random number generator
Instead of using
def x := blah; if (blah != null) {...},
instead use
if (blah =~ x:notNull) {...}
for java calls to objects using deflectors on eventlistener interfaces, the calls are converted into sends
having multiple constructors and inheritance at the same time
Rights amplification
how to wait on a device like a socket
building your own quasiparsers
weak pointers
operators for twine like split and rjoin
twine versus string, run versus substring, etc.
Note that the revoker system presented earlier only works if the facetized object and the revokable user are not collaborating; if they are collaborating, use membranes.
S-expression/XML quasi-parser
Put in explanation of the following example. First put in a simple example. This is way too fancy as the basic intro to making a guard.
First, here is the emaker that makes a pair of VOC claim check guards, a proveAuth (i.e., a claimCheck), and a checkAuth (i.e., a checkedClaim):
# E sample
def makeVOCPair(brandName :String) :near {
var myTempContents := def none {}
def brand {
to __printOn(out :TextWriter) :void {
out.print(brandName)
}
}
def ProveAuth {
to __printOn(out :TextWriter) :void {
out.print(`<$brandName prover>`)
}
to getBrand() :near { return brand }
to coerce(specimen, optEjector) :near {
def sealedBox {
to getBrand() :near { return brand }
to offerContent() :void {
myTempContents := specimen
}
}
return sealedBox
}
}
def CheckAuth {
to __printOn(out :TextWriter) :void {
out.print(`<$brandName checker template>`)
}
to getBrand() :near { return brand }
match [`get`, authList :any[]] {
def checker {
to __printOn(out :TextWriter) :void {
out.print(`<$brandName checker>`)
}
to getBrand() :near { return brand }
to coerce(specimenBox, optEjector) :any {
myTempContents := null
if (specimenBox.__respondsTo("offerContent", 0)) {
# XXX Using __respondsTo/2 here is a kludge
specimenBox.offerContent()
} else {
myTempContents := specimenBox
}
for auth in authList {
if (auth == myTempContents) {
return auth
}
}
myTempContents := none
throw.eject(optEjector,
`Unmatched $brandName authorization`)
}
}
}
match [`__respondsTo`, [`get`, _]] {
true
}
match [`__respondsTo`, [_, _]] {
false
}
match [`__getAllegedType`, []] {
null.__getAllegedType()
}
}
return [ProveAuth, CheckAuth]
}
Now here is an example of the pair of VOC guards in use:
? def [ProveAuth, CheckAuth] := <elib:sealing.makeVOCPair>("voc")
# value: [<voc prover>, <voc checker template>]
? def f1 := <file:~/.bashrc>
# value: <file:c:/Documents and Settings/millerm1/.bashrc>
? def f2 := <file:~/Desktop>
# value: <file:c:/Documents and Settings/millerm1/Desktop/>
? def foo(f :CheckAuth[f1,f2]) :void {
> println(f.getPath())
> }
# value: <foo>
? foo(f1)
# stdout: c:/Documents and Settings/millerm1/.bashrc
#
? def f3 := <file:~>
# value: <file:c:/Documents and Settings/millerm1/>
? foo(f3)
# problem: Unmatched voc authorization
? foo(f1 :ProveAuth)
# stdout: c:/Documents and Settings/millerm1/.bashrc
#
? foo(f3 :ProveAuth)
# problem: Unmatched voc authorization
? def bar(f) :void {
> println(f.getPath())
> }
# value: <bar>
? bar(f1)
# stdout: c:/Documents and Settings/millerm1/.bashrc
#
? bar(f1 :ProveAuth)
# problem: <NoSuchMethodException: <a sealedBox>.getPath/0>
The Quick Reference Card can be seen here.
E does not attempt to control computing resources such as memory and disk space, so inside the context of E, it is not considered a conveying of a controlled authority if such compute resources are allocated. Objects which are transparent and transitively immutable (i.e., deep frozen) are considered to convey no authority. Strings, integers, ConstLists, ConstMaps, and eMakers all meet these criteria (though the elements of a ConstList, and the objects made by an eMaker, may very well convey authority).
respondsTo etc.
yourself used to get reliable "broken" behavior when sending to local object
the "opt" prefix for optional, substitute for get if null returnable
Are return objects that don't meet the guard simply coerced to null? warn developer that it won't raise an exception, this could be a source of a null value
must rethrow the catch clause if using the promise
coming out of when done
"bind" is now a standalone verb, no "def"
needed.
in walnut,
talk about printOn(stream), and use it in examples. In
security section, note that it must use guard printOn(out :TextWriter).
remember that printon reveals whatever you put on out. the other way to be
safe is to print the objects on the way to constructing what gets printed, as
in
"" + x
or
`$x`
also,
`$\n` is a newline now
AHK: There's also the scalability issue with ACLS. My door, CD cabinet,
and gun vault all need to know who to let in. Any change has to be
communicated to all of them in a timely manner. This becomes hard as the
number of users and control points goes up. Here's the way I describe it.
One essential difference between capbilities and ACLs is that the former
relates to a role and the latter to an identity. Here's an example from real
life.
Zebra Copy, a small business in Palo Alto and Cupertino, does business with
HP. Some 2,000 HP employees are permitted to order work from them. The system
in place uses ACLs, so Zebra Copy has a database of HP employees and what
each is allowed to do. Every time an employee changes roles, HP must notify
Zebra Copy, and they must update their database. HP has some 20,000 such
business partners, and Zebra Copy has several hundred companies it does
business with. What a nightmare. I thought the person describing this to me
was joking.
If capabilities were used, life would be much simpler. Zebra Copy would give
HP a capability for each access right. It would be up to HP to manage those
capabilities. When someone at HP changed jobs, it would be HP's
responsibility to make sure that the capability was transferred properly.
Should a capability be stolen or misused, HP would be responsible until it
notified Zebra Copy to revoke it. Zebra Copy would need only keep one set of
capabilities for each contract; HP would not need to keep suppliers informed
of personnel changes.