Next generation, interactive media, enhancing dementia care and support

Dementia TV is an Australian world-first, arts/science research project, designed to transform the experience of television viewing for people living with dementia.

Populated with artificially intelligent [AI] characters, which personally interact with viewers, Dementia TV pioneers interactive screen-based media, oriented to the psychosocial needs of people living with dementia.

The challenge

For many people with dementia, TV is the main medium of entertainment, stimulation and companionship.1 It provides company for people who spend many hours alone and can potentially support the preservation of cognitive functions. But as dementia progresses, it becomes harder to engage with previously enjoyed programs. When people lose their connection to TV, this can have a profound impact on quality of life.

Research shows that people with dementia can maintain engagement with television for longer when programming is “cognitively appropriate”.2 But there is little such programming available. The features of today’s mainstream TV programs (such as complex plotting with multiple characters, fast paced action and dialogue, scene changes with fast cuts, interspersed with advertising) are likely to induce confusion rather than extended engagement. Cartoons and programming oriented toward younger children may be deemed “cognitively compatible” for people living with dementia but these do not reflect the interests, experience and identities of people who have led long lives.

Dementia TV aims to fill this gap, creating tailored TV content for people living with dementia. But it also goes much further in extending the interactive potential of TV to support the lost cognitive function in residents with cognitive impairment3 Interactive technology enables a form of personalised TV in which characters are ‘conversational agents’ actively engaging with viewers and attuning to their interests and psychosocial needs. More than just passively consumed entertainment, this form of TV provides meaningful companionship and cognitive support.

Dementia TV builds on fEEL’s award-winning screen experience, The Visit (see below). Underpinned by extensive research with people with dementia, fEEL delivers media that people like and want, carefully designed to fit into everyday settings.

In aged care TV has been central in providing companionship and entertainment. However, in its present form it has been described as an under-utilised and poorly adapted resource.

Dementia TV will work with residents and staff in the development of this project to reconceive television for this sector. 

What will we do and how

Develop conversational agents

A series of conversational agents will be developed with the ability to recognise the emotional state of the viewer and their level of attention. They will ‘see’ and ‘hear’ the viewer using computer vision and voice recognition and using machine learning, respond appropriately by modelling effective, positive, calming, and reassuring behaviours. The characters are experienced by the viewer as “real” people (as per the genre of drama used in TV and film).4 Engaging with characters in this way promotes psychological effects such as identity recognition, self-other turn taking, and empathic responses leading to stress and anxiety reduction. The characters address not only the needs arising from stress but also the need for enjoyment and stimulation. Thus, the characters can support wellbeing, enhance agency and personhood, and encourage relationships and social connection. 

The project will build on existing affective computing research and the development of machine learning systems that ‘read’ facial expressions to determine human emotions, gesture and tone and communication research relating to how facial expressions change with dementia. 6 To develop systems capable of interpreting the intricacies of human communication to provide appropriate companionship and entertainment solutions for people with dementia.

Build evidence base

We will collect in-depth qualitative data relating to verbal and non-verbal communications and the daily interactions of people living with dementia from the complex, ‘messy’, ‘real world’ aged care spaces.7 The data includes neuropsychology of facial responses and gesture, observation behaviours, studies of emotions, cognitive change, and memory loss, and psychosocial engagement with social networks to understand group dynamics.

Creating relatable characters

Character development for the conversational agents will be iterative and participatory using innovative and creative approaches, co-design, ethnography, and micro-phenomenology to create responsive characterisations informed by the needs arising from the wide range of participants in the projects. 8 The diverse characters can, for example, address the re- traumatisation of people by eliminating triggers of trauma and by modelling unthreatening, empathetic engagement.

Who is doing this?

The fEEL research team consists of artists, media designers, psychologists and ageing and dementia researchers. We will combine clinical and psychotherapeutic know-how, experience of working closely with people living with dementia, capabilities in developing the technology to create impactful, immersive multimedia artworks, and design effective tools and experiences that engage and appeal.

Previous work

The Visit is an immersive interactive engagement produced by fEEL won the Award for Originality and Impact at the 25th ACM Symposium on Virtual Reality Software and Technology (VRST), 2019.9 It characterises Viv, a woman who shares her experiences of dementia using a near-verbatim script created from the interviews of four women sharing their insights in to their auditory and visual hallucinations, and developed using Virtual Production (VP) techniques.

Funding and support

The research base for Dementia TV is supported by an Australian Research Council Laureate Fellowship of $3.2M awarded to Scientia Professor Jill Bennett. The ARC funds fEEL lab to conduct arts-science research using immersive technologies in relation to ageing and mental health.