5 Word-net and perceptual memory

In our network, we have up to this point only dealt with procedural memories, i.e., memories representing the connections between specific sensorimotor elements that are able to control specific behaviors (e.g., Swingnet, landmark). As a final extension, we will now show how the network might also be equipped with some aspect of semantic memory, such that meaning can be attributed to verbal expressions. To this end, the network can be expanded through the introduction of another layer (not shown in figure 6). In this fourth layer, verbal expressions are stored as procedures or “Word-nets”. These procedures can either be used to pronounce a stored word or to comprehend it, i.e., they can be used for motor control and for auditory perception. As is the case for other procedures, each Word-net is equipped with a motivation unit. As the motivation units of Word-nets have a specific function, for an easier distinction we will call them word units (WU). Following Steels (2007; Steels & Belpaeme 2005) each Word-net is related to a corresponding unit of the motivation network that carries meaning (e.g., the motivation unit for walking is connected to a Word-net “walk”). The meaning of the Word-nets is in this way grounded in the behaviors of the corresponding motivation units. As an example, figure 7 shows a possible detail of such a network, including some elements of Walknet and Navinet. The motivation units of a procedure (e.g., Swing net) and its corresponding Word-net (e.g., “Swing”) are coupled via bidirectional connections (dashed double-headed arrows). The connections cannot be active at the same time, but depend on an overall state of the network, termed “Report” and “Perceive”. In the Perceive state, only connections from the word unit to the motivation unit of its non-word procedure can be activated (from top to bottom in figure 7), whereas in the Report state only the opposite connections can be activated. As can be seen in figure 7, Word-nets can not only be connected with motivation units of the sensorimotor nets, but also with motivation units that do not directly control a sensorimotor element (e.g., Walk, Outbound).

What might be the function of this extension by Word-nets? In the Perceive state (or react state), a perceived word, uttered by another agent, will activate, via its word unit, its partner’s motivation unit, and thereby possibly influence behavior (depending on the actual internal state of the system and on the strength of the word input). When in the Report state, the actually active motivation units will in turn activate their corresponding word units, which may lead to an uttering of a word. As, of course, only one word can be activated at a given time, some kind of decision network (e.g., a WTA net) is required, though, for reasons of simplicity, not shown in figure 7. In any case, introduction of Word-nets allows for a very basic form of communication between the agent and any other partner, communication being limited to “one-word sentences”.

As indicated on the left side of figure 7 (units “front”, “left”), further motivation units might be introduced into the network that do not have a direct function within, in this case, the Walknet controller. Of course, these units may be connected to word units. (Note that we do not deal with the question how these units may be connected within the network through training.)

This architecture combines sensorimotor procedures with Word-nets (which by themselves represent specific sensorimotor procedures). Together, they form a simple case of semantic memory, because procedural memory representing an action (e.g., Swing-net) is connected with a memory element representing verbal symbols.

Figure 5: The body-model and its relation to the body of robot Hector (a). (b) shows the vectors forming the central body (left) and the vectors forming one leg model (right). The central model and the leg-models are connected via the shared “leg vector” (white arrows) that point from the hip to the tip of the leg (shown here for the left front leg only). Walking direction and velocity are controlled by the input vectors pull_fw (forward) or pull_bw (backward) provided by sensory input.

To illustrate the versatility of this architecture, we will briefly address how it can also be applied in order to embrace perceptual memory. Following ideas of O’Connor et al. (2009), Cruse & Schilling (2010a) have shown how an RNN, using the same elements as applied here for the motivation unit network, could be used to construct a perceptual memory. This network does not only allow the representation of directly perceived perceptual elements (e.g., the colour or shape of an object), but also of superordinate concepts (e.g., Cow, Animal, four-legged). Note that “four-legged” might also be a feature of non-animals, e.g., a table. Therefore, the ability of our network to deal with heterarchical structures is advantageous for perceptual memory, too. Elements of such a distributed memory can also be connected to specific Word-nets (e.g., “red”, “Cow”, “animal”), as has been explained above for the sensorimotor motivation units. Correspondingly, activation of one memory element of this perceptual memory may elicit the uttering of the corresponding word, and, in turn, when in Perceive mode, the hearing of a word may activate various elements of the procedural memory that are associated with this word.