USAGE:

SYNOPSIS:

CONSTRUCTORS:

mlp(int nodetype,int *topology)

create network with general topology, with inner and/or output nodes of type nodetype (one of TANHxd,FERMIxd,LINEARxd,RATIOxd,LOGxd, x=1,2 or 3). Input nodes will be chosen as linear. Each connection is specified by a pair of successive integers in array topology, i.e., topology[]=i1,j1,i2,j2,...ik,jk specifies connections i1->j1,i2->j2,..ik->jk. Use neuron indices i,j from the range [-1,0,1,2,..n], where -1 represents an invisible bias neuron that has the constant value of 1. (for a network with general topology, bias connections are not automatically generated but must be explicitly included in the list).

mlp(int n1, n2, ... nk)

create network of k layers. The i-th layer has ni neurons, successive layers are fully connected. For each neuron, an additional connection to a fixed [invisible] bias neuron with fixed activity=1 is automatically generated.

mlp(int *layers, int *connections=NULL)

create layered network with layer sizes and layer node types specified in layers[]. If array connections is NULL, the default topology of a full set of connections between successive layers is chosen.

   layers[] = {4, 10, 5}
   connections[] = {0,1, 1,2, 0,2}

SPECIFYING ACTIVATION FUNCTIONS AND LAYER TOPOLOGY:

   layers[] = {LINEAR2d,10,20, FERMI3d,5,5,5, TANH1d,100}

Predefined activation function types are:

FERMI1d,FERMI2d,FERMI3d:

1d,2d,3d layers of fermi function neurons

TANH1d,TANH2d,TANH3d:

1d,2d,3d layers of tanh function neurons

LINEAR1d,LINEAR2d,LINEAR3d:

1d,2d,3d layers of linear function neurons (the output layer is always linear)

RATIO1d,RATIO2d,RATIO3d:

1d,2d,3d layers of neurons with the weakly decaying rational activity function x/(1+abs(x)).

LOG1d,LOG2d,LOG3d:

1d,2d,3d layers of neurons with the weakly unbounded activity function sgn(x)*log(1+abs(x)).

SPECIFYING TIME DELAY LAYERS FOR TDNN NETWORKS:

    layers[] = {LINEAR1d+TDL,4,2, FERMI1d+TDL,5,3, FERMI1d,1}

SPECIFYING CONNECTION TYPES:

   layers[] = {5, 10, 10, 2}
   connections[] = {0,HEBB_W,2, 2,ELMAN_W,1, 2,BPROP_W,3}

SPATIALLY RESTRICTED CONNECTIONS AND WEIGHT SHARING:

    layers[] = {10,8,5}
    connections[] = {BPROP_W+LOCAL_W, BPROP+SHARED_W}

USING AND TRAINING OF FEEDFORWARD NETWORKS:

void exec(float *inp=NULL)

evaluate network for input data inp (if inp is NULL, the current input activities will be used without change. The array order[] can be filled with the desired sequence in which neuron activities shall be updated.

void adapt(float *target)

perform one adaptation step (after a previous exec() call), using target values target. During online learning, each adapt() call will cause an immediate weight change. During epoch learning, error gradient results will be accumulated until a full epoch is completed. Only then will the accumulated values lead to weight changes.

void mode(int mode, float f1=0, f2=0, f3=0)

Choose learning mode (currently, one of BPROP_M or RPROP_M). For BPROP_M, any nonzero f1 will redefine the global learning rate, f2 the momentum alpha, f3 the flat spot elimination constant (the latter not yet implemented). For RPROP_M, any nonzero f1 and/or f2 will set the learning step decay and growth factors, and f3 the maximal step size.

void init(void)

initialize learning constants to values set with mode() function, clear all memorized gradients and start a new epoch, but don't change the current weights or activities (they can be changed explicitly through the weight[] and act[] arrays).

PARTIALLY RECURRENT NETWORKS:

RECURRENT BACKPROPAGATION:

    net.bgn_bptt(len);
    net.train(...);
    net.end_bptt(len);

bgn_bptt(int iSteps=0)

begin backpropagation-in-time sequence and prepare buffers for a time series of maximally iSteps. Assumes that up to iSteps exec()/adapt() calls follow (it is permitted to have longer time sequences. In this case, the necessary buffer enlargement will occur automatically, but it is more efficient to tell the appropriate buffer size in advance). For iSteps=0, a default of 32 steps will be assumed.

end_bptt(void)

end current backpropagation-in-time sequence and performs one learning step for the time series seen through the exec/adapt sequence since the most recent bgn_bptt() call. For BPTT training, this call is essential: without it, the preceding adapt() calls will be without any effect!!

no_bptt(void)

end BPTT mode altogether (frees all BPTT specific buffers).

PUBLIC CLASS VARIABLES:

float act[]

activities of all network neurones (when the bias neuron has been automatically created, it will not be included in act[]).

float weight[]

network weights, order in the same manner as the pairs in array topology or layerwise, if the constructors for layered networks are used (in this case, for each neuron its bias weight comes first).

float eta[]

for each weight a learning parameter. For learning mode BPROP_M, this is the learning rate, for RPROP_M, it is the learning step size.

float alpha

momentum term for BPROP_M learning mode: alpha=0 (the default) will yield standard backprop learning. A value 0<alpha<1 will modify the weight update to

   Delta(t) = eta*Dw(t) + alpha*Delta(t-1)

float lambda

a global weight decay parameter. After each update, all weights are multiplied by a factor 1-lambda (again we have two different timescales for online and epoch learning here!).

float mse[]

mean square error for each output value. Average will be over current epoch. During online learing, average will be since last init() call.

float margin[]

has one element for each output and allows to specify an error margin below which that output will be considered as error free (i.e., it then will not contribute to the computation of the average error). However, for positive margin value the associated error (even if below the margin) will still be backpropagated to drive the adaptation process. To omit below-margin errors in all or some output dimensions for the adaptation process, specify the corresponding margin values with a negative sign (which will only be recognized as a flag).

int order[]

order in which neuron activities will be recomputed by the exec() method (the adapt() method will use the reverse order). For pure feedforward networks, the constructor function will fill order[] with a suitable sequence automatically. For networks with feedback connections, order[] must be specified by the user.

int epochlen

if 0, adapt() will use online learning, otherwise epoch learning with epoch length epochlen.

int wtype[]

for each weight a type specifier. Usually set by the constructor function, but can be overridden with user-supplied values (however, changing the weight type, e.g., will not change the connection topology in any way and thus may lead to inconsistencies in some cases (e.g., when specifying type ELMAN, which assumes 1:1 connections)).

FURTHER METHODS FOR TRAINING OR TESTING:

float train(int epochs, int dim, float *data, int num, int first=0)

perform epochs training epochs for dim dimensional rows in array data. The initial numinputs elements in each row will be used as input data, the next numoutput elements will be the output data, any further row elements are ignored (the condition numinputs+numoutputs<=dim is checked). For num>=0, training will use num successive rows, starting at row first. For num<0, training will use the entire data set, except for abs(num) rows starting at row first (this is convenient for crossvalidation). Return value is the root mean square error over the epochs (the elementwise errors can be found in array mse[], but note that the latter are still squared!). For epochs<0, training will use maximally abs(epochs), but will terminate as soon as the error in one epoch has reached 0.

float train(int dim, float *data, int num, int first=0)

Shorthand for the above with epochs=1.

float test(int dim, float *data, int num, int first=0)

Similar to train, but without adaptation, i.e., only error statistics is computed and the root mean square error is returned.

int schedule(void)

find update order. When the network is purely feedforward, this function has already been called by the constructor. For recurrent networks, it should be called when weight types for a partially recurrent topology have been assigned to the wtype[] array. A nonzero return then indicates that the network can be trained with the usual exec/adapt methods (i.e., it is partially recurrent). (the schedule() function ignores any ELMAN connections for its check, ie., checks whether the network can be trained with the standard backprop method).

INSPECTION METHODS:

int neurons()

return nr of neurons (bias neuron not included)

int layers()

return nr of layers (includes input layer 0 in its count).

int layer_of(int id):

get layer of neuron id.

int for_inp_of(int n,char *fmt,void(*f)(float*),int *layers=NULL)

int for_out_of(int n,char *fmt,void (*f)(float*),int *layers=NULL)

Iterate function f() over inputs (outputs) of neuron n. When layers!=NULL, the iteration will be restricted to those inputs (outputs) that originate (end) in layers specified by the entries in layers. There is a second pair of routines where the last elements are up to three ints to allow a simpler specification when the number of included layers is small. Return value is the number of iteration steps done.

The following format characters are defined:

#

index of weight between distal and proximal neuron

w

value of connection weight between distal and proximal neuron

r

learning rate of weight between distal and proximal neuron

?

type of weight between distal and proximal neuron

a,A

activity of distal (proximal) neuron

e,E

error value at distal (proximal) neuron

x,y,z,X,Y,Z

coordinates of distal (upper case: proximal) neuron

l,L

layer to which distal (proximal) neuron belongs

n,N

index of distal (proximal) neuron (the latter coincides with arg1)

f,F

activity function type of distal (proximal) neuron

i,j,k,I,J,K

index positions of distal (upper case: proximal) neuron in its layer

<

nr of inputs of distal neuron

>

nr of outputs of distal neuron

float *for_inp_of(int n, char *fmt, int *layers=NULL)

float *for_out_of(int n, char *fmt, int *layers=NULL)

Similar as before, but instead of iterating, returns a newly allocated vector consisting of the concatenation of the argument vectors that would have been passed to the iterated function. E.g., to get the input weights of neuron n in layer l. concatenated into a single vector v, call

    v := for_inp_of(n,"w");

int for_neurons_of(int lay, char *fmt, void (*f)(float*))

float *for_neurons_of(int lay, char *fmt)

Similar iterators for iterating over all neurons of a layer. Format characters to be used for fmt are a subset of the above, ie.

a

activity of neuron

b

bias weight of neuron

e

error value at neuron

n

index of neuron

f

activity function type of neuron

i,j,k

index positions of neuron in its layer

x,y,z

coordinates of neuron

l

layer of neuron (always equal to arg1)

<

nr of inputs

>

nr of outputs

void query_neuron(int neuron, char *fmt, float *vec)

Similar as for_neurons_of, but queries data only for a single neuron and writes result into vector vec (clipping, if vec is too short).

AUXILIARY METHODS FOR VISUALIZATION:

void bbox(int l, float x,y,z=0,dx=0,dy=0,dz=0)

specify a bounding box geometry (position and edge lengths) for layer l.

void bbox(int l, float* x,y,z,dx,dy,dz)

similar, but to specify (retrieve) a bounding box for layer l. Zero delta values (last args) will cause no change to the current delta settings.

void bbox(int l, void (*fun)(float x1,y1,x2,y2))

void bbox(int l, void (*fun)(float x1,y1,z1,x2,y2,z2))

Iterate drawing function fun over edges of bounding box of layer l.

int pick(int l, int i,j=0,k=0):

get neuron id from layer number l and within-layer index coord i,j,k.

int pick(float x, float y):

get neuron id from location

void layout2d(float fGap=1,xll=0,yll=0,xur=1,

yur=1,int *layers=NULL)

void layout3d(float fGap=1,xll=0,yll=0,zll=0,

xur=1,yur=1,zur=0,int *layers=NULL)

Specify arrangement of layers in rectanglular 2d (3d) box. The box is specified by its lower left and upper right corner. fGap specifies the ratio of the gap between adjacent neurons of the different and in the same layer (i.e., fGap=2 chooses the gap between layers twice as large as within a layer). If layers=NULL, all layers will be chosen, otherwise the positioning will be restricted to the layers whose numbers are listed in layers[].

REMARKS:

ALGORITHMS:

NOT YET IMPLEMENTED:

   join(i,j,rule,type1,type2,type3)

int pick(float x,y,z):

get neuron whose position is closest to x,y,z.

FILE