This experimental tool allows the visually impaired to perceive the details in a painting.
Visual impairment may be broken down into two basic areas: problems in the perception of shape and in the perception of colour. In the case of shape perception, visually impaired people may use – within reasonable size limits – a tactile translation, by which the world of shapes can be sensorially perceived through touch.
With paintings, the model must also take into account the fact that paintings are typically a two dimensional projection of a three-dimensional scene: when we look at a painting, our brain reconstructs the complete shapes on the basis of the partial view provided by the two dimensional projection.
For the painting we chose to render perceivable to the visually impaired – the fresco of the infant Hercules strangling serpents at the House of Vettii – shape has been made perceivable by touch through the use of a relief transposition on a resin base to represent only a portion of the objects in the painting, so that the rest of the scene still has to be reconstructed mentally.
Of course, this tactile solution cannot be used for colour, which is an exclusively optical quality. We therefore employed “synaesthetics” to represent colour. This is the well-known phenomenon by which, in all individuals, one type of perceptual stimulus spontaneously elicits another. For instance, by saying that a colour is "warm" or "cold", we are associating a visual perception with a tactile one. A typical synaesthetic association is the one between colours and sounds, which is the one we made use of in this case.
We systematically matched every possible colour with a musical sound: therefore, every variation in colour corresponds to an identical (in terms of direction and intensity) variation in the musical sound. This was achieved by using an algorithm that defines a relationship between the levels of the three variables in any colour – tone, luminosity and saturation – and the three variables in any musical sound – timbre, pitch and intensity.
To enable a visually impaired person to perceive shape and colour together, a correspondence between the tactile sensation and the musical sound must be established, so that the sound emitted corresponds to the colour at the point touched by the person in that instant. This has been achieved by using a miniaturised three-dimensional plotter worn on the finger used to "explore" the painting by touch. The plotter instantly determines the position of the finger in space and, therefore, on the three-dimensionally translated shape of the painting. Obviously, this position corresponds to a particular colour on the painting. The algorithm now instantaneously processes this data and generates the musical sound corresponding to the colour touched.

This tool here is the second to be developed using this system. The first one was an experimental prototype created as part of the European HELP research project. The prototype, which uses exactly the same methods to define the three-dimensional model and the same software to translate colour into sound, was based on Raphael’s “The Woman with the Unicorn” displayed at the Borghese Gallery in Rome.

Development team
HELP project manager: Benedetto Benedetti
System design and development coordinator: Francesco Antinucci
Painting modeller: Paolo Carosone
Research and experimentation on synaesthetics: Sabina Rosafio
Musical translation of colours: Pierluigi Castellano
Music software: Michelangelo Nottoli and Luca Spagnoletti.
Plotter software and system integration: Infobyte (Antonio Palmacci)