Autism ASSISTant

Virtual assistant for the enhancement of cognitive skills of children with an autism spectrum disorder.

Autism ASSISTant is a research & development project awarded within the CENTRIC project, which stands for Center for knowledge transfer to enterprises in the ICT field. 

The project envisaged research activities to create an intelligent software platform dedicated to developing the autonomy of children affected by autism spectrum disorders (ASD). This was accomplished by creating modules designed to contribute to their cognitive development, enhance their memory and reasoning skills, and develop other social and practical skills needed for daily routines. 

To ensure that the needs of children with autism and their caretakers are met, the project was implemented by a multidisciplinary team of IT specialists, researchers, and ABA therapists (Applied Behavior Analysis). The platform’s design is rooted in the extensive expertise of psychologists and ABA therapists, laying a solid foundation for efficiently implementing Applied Behavior Analysis (ABA) therapeutic protocols.

Autism ASSISTant is a comprehensive platform consisting of two main components: a web application exclusively dedicated to therapists, offering therapeutic information and guidelines, and a mobile application designed for parents, caregivers, teachers, and children diagnosed with Autism Spectrum Disorder (ASD) aged between 0 and 8 years old.

Autism ASSISTant platform’s web application

The web application was developed using the React framework and Ruby on Rails. Its primary role is to manage and transmit therapeutic information to the mobile application. It specifically caters to psychotherapists specialized in treating individuals with ASD who possess the necessary expertise to conduct treatment sessions. The web application features a goal configuration interface, enabling therapists to create objectives in four categories: social/emotional, language/communication, cognitive, and movement/physical development.

Therapists can define specific tasks that patients or groups of patients need to regularly complete to enhance their skills. Additionally, specialists can create parent guides that provide instructions for parents on conducting therapeutic interactions with their child, particularly focusing on those under 2 years of age. The web application serves as a powerful tool for delivering therapeutic information and methodology in accordance with Applied Behavior Analysis (ABA) protocols.

Autism ASSISTant platform’s mobile application

The mobile application was developed using Unity game engine and the C# programming language. Unity is a cross-platform game engine developed by Unity Technologies, capable of generating realistic scenarios based on the laws of physics. Unity game engine was the ideal choice due to its development capabilities for iOS and Android platforms. Additionally, Unity offers extensive integrations with services and tools, such as data analysis and automated testing. Unity Analytics played a role in enhancing the platform’s functionality by capturing specific events related to user profiles, group organization, daily tasks, and learning progress. The insights obtained through Unity Analytics were used to improve the educational content, ensuring alignment with the user’s personalized progress.

Designed to maximize learning potential and enhance autonomy, Autism ASSISTant’s mobile application incorporates five modules that cater to the specific needs of individuals with autism and their caregivers. 

• Base Module;  
• Module I - Parent Guide;  
• Module II - Individual Games;  
• Module III - Group Games;  
• Module IV - Video Modeling. 

Autism ASSISTant modules

▶ Base Module - Building the Foundation for Success: 

The base module serves as the backbone of the Autism ASSISTant platform, facilitating seamless communication between the interactive modules and essential components such as the server, user interface manager, and audio manager. This module also encompasses critical functionalities like account and profile management, allowing users to create personalized data sets for each individual receiving therapy. Furthermore, it streamlines the organization of therapy data by enabling the creation of specific groups, ensuring a comprehensive and efficient therapy experience.

▶ Module I: Parent Guide - Empowering Parents and Therapists:

Early intervention is essential to maximizing a child's learning potential, but diagnosis is often delayed for various reasons, including parents' reluctance to accept that their child may have a disorder and a lack of access to specialized services. This prevents autistic children from receiving therapy at a younger age, potentially impeding their development. Therefore, it is crucial to inform parents about early signs of autism and protocols for working with children from different age groups, starting as early as two months old.  

Recognizing the significance of early intervention, the Autism ASSISTant platform introduces Module I, the Parent Guide. This module aims to support parents and therapists in maintaining therapy sessions by providing them with therapeutic information about procedures and protocols. Children with autism or other developmental disorders need around 4 to 6 hours of individual therapy daily and continuous intervention to generalize and consolidate the notions and behaviors learned during therapy sessions with specialists. This is why parents or other family members must be trained to become co-therapists for their child.

Integrated with the OneSignal package, the application sends alerts at predetermined times, keeping parents and therapists informed about the status of active tasks. This feature ensures consistent therapy over extended periods and generates an exhaustive journal with detailed records of ongoing and completed treatments. By incorporating guidance from specialists, this module empowers parents to actively participate in their child’s therapy journey.

Parent Guide

Daily Treatment

▶ Module II: Individual Games - Fostering Autonomy through Engaging Scenarios:

Module II of the Autism ASSISTant platform is dedicated to enhancing children’s autonomy through interactive game scenarios. These scenarios can be generated using Machine Learning (ML) algorithms or created by adult users, such as parents, caregivers, or therapists, using the scenario configurator.

Scenario manager

Scenario configurator

The platform leverages well-defined techniques, primarily based on Applied Behavior Analysis (ABA) therapy procedures, to replicate therapy sessions effectively. The Receptive Mode prompts children to perform a series of actions, helping them associate different concepts and develop essential skills. In contrast, the Expressive Mode automates action steps, requiring children to identify various elements involved. This module not only fills the resource gap for parents seeking therapeutic materials but also allows customization of scenarios based on each child’s unique abilities.

Interactive learning of the “Drying Hair” action through therapeutic scenarios in Receptive Mode

“Drying Hair” action’s animation

▶ Module III: Group Games - Fostering Collaboration:

Module III of the Autism ASSISTant platform introduces play sessions designed for teams of two players. Users have the flexibility to create and configure these sessions or to use suggested scenarios generated by ML based on the evolution of previous sessions. When selecting a session, the platform prompts the user to specify the second player, who can be an external person, such as a therapist or parent, or another individual within the same account. This module enables comprehensive progress analysis and fosters collaboration between therapy participants.

▶ Module IV: Video Modeling - Enhancing Learning through Engaging Videos: 

Module IV of the Autism ASSISTant platform combines video modeling techniques with interactive approaches. It offers three distinct modes: Library Mode, where parents can access and visualize a library of educational videos; Decisional Mode, which requires users to make specific decisions to progress through video modeling sessions; and Mini-session Mode, where users complete mini-scenarios from Module II after viewing each video. This module provides a dynamic and engaging learning experience that reinforces therapy concepts through video modeling and interactive gameplay. 

Creating therapeutic scenarios in Decisional Mode

Interactive learning of the “Washing Hands” action through therapeutic scenarios in Decisional Mode

The Autism ASSISTant platform enhances therapeutic interventions by incorporating various modules tailored to the unique needs of individuals with Autism Spectrum Disorder. From the foundational Base Module to the interactive Individual and Group Games modules, and the immersive Video Modeling module, this comprehensive platform empowers parents, therapists, and individuals with autism to actively participate in and benefit from therapy sessions. With its innovative features and user-friendly design, the Autism ASSISTant platform offers a promising future for optimizing therapy outcomes and transforming the lives of individuals with autism.

☆ Objectives:

• Designing and implementing Autism ASSISTant modules meant to enable caretakers to continue the learning programs from therapy sessions at home, integrating them into the child’s routine  
• Developing the autonomy of children affected by autism spectrum disorders  
• Enabling parents to become co-therapists, thereby supporting the child’s developmental progress  
• Early detection of autism symptoms
   

ASSIST Software conducted research and development activities and was responsible for UI/UX prototyping, software architecture elaboration, and software development of multiple components, including frontend, backend, and ML.

Project ID: Nr. 22080/05.10.2022 

Funding: European Regional Development Fund, Competitiveness Operational Programme 2014-2020, CENTRIC