Some of my thoughts on graphical programming, and how they are linked up with work I've done proeviously on networking, programming, multithreading, and graphical interfaces.

Programming Considerations for 2D/3D graphical programming

14 October 1997 M.T. Verelst

Introduction to the problem

Graphical representations have a tendency to explain more than a thousand words could. Therefore it is not an unreasonable idea to use graphical representations as a programming aid, or even directly as a programming language or 'medium'. The main problem is then how to create, represent and handle graphical scenes with an algorithmic or 'program' meaning.

Well known examples are 2D and 3D graphs, comprised of blocks and wires connecting them up, representing finite state machine building blocks interfacing using a limited number of messages. More complex examples could include nodes with complex structures, non-trivial connections (e.g. rich message protocol), dynamic connections and unit constellation, timed interactions.

Some examples and explanatation of though frame

The latter arise in for instance in multi media applications, such as audio and video on a world wide web browser like Netscape. These are full blown, functional applications, and they have relatively simple and static (global) structure: one server connects to a number of clients using fixed type streams. Timing is relevant and in some cases dealt with in a non-trivial way, such as in the RealAudio connections (and comparable types), but usually implicit.
The applications are fixed, and the functionality to achive the right type of connection behaviour is programmed in an existing common language.

A block or 'node' with a certain behaviour largely coincides with a program running under some operating system. Apart from file access and standard interfaces (including window interfaces), programs do not have input output links associated with them which easily connect them up with other programs. Internally, object oriented languages offer interconnecting functionality by sending messages between objects, but this extends to out of process boundaries only in a limited number of cases (for instance NextStep has a standard way of passing messages between processes). Apart from efficiency, name-space, and connectivity questions, object oriented languages implement only a portion of what inter-proces communication could offer in terms of formal behaviour, strength of implementation, and standarization.

Efforts in the networking world (such as for ATM) lead to fairly formally specified channels, which would lend themselves to be used as standard building blocks. Apart from timing, Unix-like sockets can also be seen as more or less standard building blocks, mentally visualized as 'pipes' with standard ways of creating them and linking them with an application. Although there is a lot of variety there, and I would not dare to call it trivial to port between various machine and language types, and although the causal behaviour is also not well defined, the socket mechanism lends itself as a standard programming approach.

Synthesis in thoughts

Evidently, these developments have not occurred independently, and are supposed to work together, as witnessed in for instance the aforementioned multimedia applications. A challenging though however would be to take the concepts a few steps furter, and to actually program in a IPC (Inter Process Communication) oriented way. It can be argued that that is what an object oriented language offers, but that is not altogether true, as I've outlined above, and even then it is an interesting though to link this type of programming directly with a powerful graphical interface, not just for class browsing and graphical feedback, but more in the way some environments exist where for instance (audio) synthesizers are constructed simply by graphically connecting up generators, filters, envelope shapers, etc.

Further reaching chalenges

Node-link abstractions may hide far greater complexity, startig with more complex datatypes (such as multidimensional arrays) or even graphical data, such as seen in application level 'flow' like environments such as AVS and khoros, which are very suitable to perform for instance image processing operations, and programming them graphically.

These environments have their limitations: they are usually not a real programming environment, and the products don't run easily as standalone programs. The graphical program is of a straightforward data-flow type, with some exception, e.g. blocks showing bidirectional interactions connot be programmed, and messages cannot usually be split up or partially transmitted (i.e. realy pipelined). Further, the standarization in the blocks and channels makes it easy to build certain types of applications, but not all. Finally, the granularity of the communication between the blocks is fairly course (unix process switching times, i.e. in the order of milli seconds).

Having a

Links with brain like behaviour

Interaction at multi-point or semi field level, integrate approximation/interpolation strategies

Practical possibilities
my conneciton server, my multi threading with inter thread sockets package, tcl/tk interfaces (incl. my proto's)