architecture.texi from marsyas at Krugle
Show architecture.texi syntax highlighted
@node Architecture concepts
@chapter Architecture concepts
In order to fully take advantage of the capabilities of Marsyas it is
important to understand how it works internally. The architecture of
Marsyas reflects an underlying dataflow model that we have found
useful in implementing real and non-real time audio analysis and
synthesis systems. In marsyas 0.2 a lot of things can be accomplished
by assembling complex networks of basic building blocks called
MarSystems that process data. This is the so
called @qq{Black-Box} functionality of the framework. In addition the
programmer can also write directly her/his own building blocks directly
in C++ following a certain API and coding conventions offering the so
called @qq{White-Box} functionality. Building networks is described in
@ref{Writing applications}, and writing new MarSystems is described
in @ref{Programming MarSystems}.
@menu
* Architecture overview::
* Implicit patching::
* MarSystem composites::
@end menu
@node Architecture overview
@section Architecture overview
@menu
* Building MarSystems::
* Dataflow model::
@end menu
@node Building MarSystems
@subsection Building MarSystems
The basic idea behind the design of Marsyas is that any audio
analysis/synthesis computation can be expressed as some type of
processing object, which we call MarSystem, that reads data from an
input slice of floating point numbers, performs some
computation/transformation based on data, and writes the results to
another slice of floating point numbers. Networks of MarSystems can
be combined and encapsulated as one MarSystem.
For example consider an
audio processing series of computations consisting of reading samples
from a soundfile, performing an short-time fourier transform (STFT) to
calculate the complex spectrum, performing an inverse STFT to convert
back from the frequency domain to time domain, then applying a gain to
the amplitude of the samples and writing the result to a soundfile.
As is very frequently the case with audio processing networks objects
the input of each stage is the output of the previous stage. This way
of assembling MarSystems is called a Series composite. Once a Series
Composite is formed it can basically be used as one MarSystem that
does the whole thing. A figure showing a block diagram-like
presentation of this network is shown in the next section.
@node Dataflow model
@subsection Dataflow model
Marsyas follows a dataflow model of audio computation.
@image{images/dataflow,,5cm}
Marsyas uses general matrices instead of 1-D arrays. This allows
slices to be semantically correct.
@image{images/slices,,5cm}
@node Implicit patching
@section Implicit patching
@menu
* Implicit patching vs. explicit patching::
* Implicit patching advantages::
* Patching example of Feature extraction::
@end menu
@node Implicit patching vs. explicit patching
@subsection Implicit patching vs. explicit patching
Many audio analysis programs require the user to explicitly (manually)
connect every processing block,
@example
# EXPLICIT PATCHING: block definitions
source, F1, F2, F3, destination;
# connect the in/out ports of the blocks
connect(source, F1);
connect(source, F2);
connect(source, F3);
connect(F1, destination);
connect(F2, destination);
connect(F3, destination);
@end example
@image{images/explicit-patching,,4cm}
Marsyas uses @emph{implicit patching}: connections are made
automagically when blocks are created,
@example
# IMPLICIT PATCHING
source, F1, F2, F3, destination;
Fanout(F1, F2, F3);
Series(source, Fanout, destination);
@end example
@image{images/implicit-patching,,5cm}
@node Implicit patching advantages
@subsection Implicit patching advantages
Creating a neural network with explicit patching soon becomes
a mess,
@image{images/neural-explicit,,5cm}
With implicit patching, this is much more manageable.
@example
# IMPLICIT PATCHING
fanoutLayer1(ANN_Node11, ..., ANN_Node1N);
...
fanoutLayerM(ANN_NodeM1, ..., ANN_NodeMN);
ANN_Series(fanoutLayer1, ..., fanoutLayerM);
@end example
@image{images/neural-implicit,,5cm}
Implicit patching can automagically adjust the connections without
requiring any code recompiliation. For example, we can change
the number of bands in a filter bank without changing any code.
@image{images/implicit-filter-bank,,4cm}
@node Patching example of Feature extraction
@subsection Patching example of Feature extraction
Suppose we wish to create a typical feature extraction program:
@image{images/feature-extraction,,5cm}
@example
MarSystemManager mng;
MarSystem* Series1 = mng.create("Series", "Series1");
MarSystem* Fanout1 = mng.create("Fanout", "Fanout1");
MarSystem* Series2 = mng.create("Series", "Series2");
MarSystem* Fanout2 = mng.create("Fanout", "Fanout2");
MarSystem* Fanout3 = mng.create("Fanout", "Fanout3");
Fanout3->addMarSystem(mng.create("Mean", "Mean"));
Fanout3->addMarSystem(mng.create("Variance", "Variance"));
Fanout2->addMarSystem(mng.create("Centroid", "Centroid"));
Fanout2->addMarSystem(mng.create("RollOff", "Rolloff"));
Fanout2->addMarSystem(mng.create("Flux", "Flux");
Series2->addMarSystem(mng.create("Spectrum, "Spectrum");
Series2->addMarSystem(Fanout2);
Fanout1->addMarSystem(mng.create("ZeroCrossings", "ZeroCrossings");
Fanout1->addMarSystem(Series2);
Series1->addMarSystem(mng.create("SoundFileSource", "Source"));
Series1->addMarSystem(Fanout1);
Series1->addMarSystem(mng.create("Memory", "TextureMemory"));
Series1->addMarSystem(Fanout3);
Series1->addMarSystem(mng.create("classifier", "Classifier"));
@end example
@node MarSystem composites
@section MarSystem composites
@WANTED{ descriptions of composites, add the other composites }
@menu
* Series::
* Parallel::
* Fanout::
* Accumulator::
@end menu
@node Series
@subsection Series
@image{images/composite-series}
@node Parallel
@subsection Parallel
@image{images/composite-parallel}
@node Fanout
@subsection Fanout
@image{images/composite-fanout}
@node Accumulator
@subsection Accumulator
@image{images/composite-accumulator}
See more files for this project here