Creating a Digital AI Twin to Connect with Loved Ones’ Memories and Work Through the Grief Stages

Posthumously partnered with Neurons Lab to create a cutting-edge platform which allows users to store and engage with memories of deceased loved ones in a secure manner whilst working through the stages of grief with a digital AI Twin.

Partner Overview

Posthumously is a forward-thinking company that aims to revolutionize the way people preserve and access the memories and legacies of their loved ones. Baraka Ekpo, the visionary CEO of Posthumously, founded the company with a mission to create a digital platform that combines cutting-edge technologies with a deep understanding of human emotions and connections.

Baraka’s vision was inspired by the loss of a family member, which made him realize the importance of preserving the essence of a person’s life story even after they are gone. His experience led him to the idea of creating a unique platform that would serve as a digital cryonics system, giving individuals a chance to immortalize their memories and stories for future generations through 3D avatars.

The Posthumously mission is to ensure that the memories, thoughts, and experiences of individuals are stored, protected, and accessible in a meaningful and empathetic manner. Using cutting-edge technology and the latest psychological and scientific research on the stages of grief, the platform enables users to engage with the memories of the deceased and cope with grief stages to overcome depression.

By combining advanced Generative AI and Conversational AI technologies to create an AI digital twin, the platform creates a digital space where people can engage with the memory of their loved ones in an immersive and personal way. 

With a strong commitment to privacy, security, and user trust, Posthumously has partnered with Neurons Lab to leverage their expertise in AI/ML consulting and their status as an AWS Advanced Partner and AWS ML Competency Partner. Together, they aim to develop a groundbreaking solution that will redefine how we remember, celebrate, and connect with the people who have shaped our lives.

Business Challenges

In order to create an immersive experience to preserve memories and connection, Posthumously sought to develop a 3D avatar that would realistically replicate a deceased individual, giving relatives a chance to connect with the individual’s essence after they had passed. 

Their main challenge was that the solution space for options and technologies available to construct digital replicas of human beings are relatively restricted. This would make the development process more complex and require a highly specific set of expertise. As well as this, Posthumously also faced a range of other complex challenges:

• A lack of automated tools. Currently, there are no readily available tools or frameworks that enable the creation of digital replicas of real people solely based on multiple photos, without requiring manual adjustments.
• The use of multiple tools. Creating digital replicas often involves utilizing a combination of tools and technologies. There is no single solution that can accomplish the entire process at once, necessitating the use of a set of tools to achieve the desired outcome.
• Ensuring privacy and security. With sensitive personal data being collected, stored, and processed, maintaining the privacy and security of users’ information is of utmost importance.
• Accurate and empathetic memory extraction. Developing a memory extraction process that is efficient, accurate, and capable of capturing the nuances of an individual’s experiences and emotions.
• Creating photorealistic digital clones. Generating realistic and convincing digital avatars that can effectively convey the personality, mannerisms, and communication styles of the individual.
• Developing user-friendly and engaging interfaces. Designing intuitive and engaging interfaces for both Stoic Users and Mourning Users, which allow them to interact with the system effectively and meaningfully.
• Providing long-term data storage and accessibility. Ensuring the reliability and longevity of the memory storage infrastructure, allowing users to access their loved ones’ memories and stories for generations to come.
• Adapting to advancements in technology. Continuously updating and refining the system to incorporate new developments in AI, data storage, and digital communication, ensuring that Posthumously remains at the forefront of digital cryonics.

These challenges highlight the current state of the field and emphasize the need for innovation and advancements in developing efficient and streamlined processes for visual 3D avatar creation.

By tackling these challenges with support from Neurons Lab, Posthumously aimed to create a unique and impactful solution that allows people to connect with their loved ones’ memories and stories, preserving their legacies for future generations to cherish and learn from.

Solution

To build this groundbreaking solution, Posthumously has developed a comprehensive and innovative digital legacy system that effectively captures, preserves, and presents the memories and stories of individuals in a sensitive and authentic manner. 

By integrating cutting-edge AI technologies and taking a user-centric approach, the system creates deeply personal digital representations that preserve the essence of individuals for future generations. The solution consists of the following components:

1. Mobile Application

A user-friendly interface that connects to the Posthumously System’s backend services, enabling users to interact with their loved ones’ stories and memories. This component is built using AWS API Gateway and AWS Lambda for serverless computing, ensuring a responsive and scalable solution.

2. Memory Extraction System

An interactive introspection tool that recursively collects user memories to be stored. This component leverages AWS Bedrock and Anthropic’s Claude models for advanced conversation and memory extraction capabilities, ensuring natural and engaging interactions.

3. Memory Base Storage

A reliable, long-lasting, and secure storage infrastructure using Amazon Neptune graph database. This powerful graph database enables complex relationship mapping between memories, stories, and contextual information. The system utilizes Neptune’s native graph capabilities for:

– Relationship-based memory retrieval
– Context-aware story connections
– Dynamic knowledge graph expansion
– Semantic search and discovery
– Multi-dimensional data relationships
– Pattern recognition in memory associations

4. Digital Avatar

A comprehensive subsystem that captures unique features of Posthumously users and reproduces them as digital avatars. This component incorporates:
– Advanced deep learning models for photorealistic avatar generation
– Whisper model for accurate speech-to-text conversion
– AWS Transcribe for high-quality voice-to-text processing
– Neural voice synthesis models for voice cloning
– Advanced lip synchronization models (Wav2Lip)

For the realistic AI video avatars, the system utilizes:

1. Deep generative models for high-quality avatar creation
2. Neural rendering for realistic motion synthesis
3. Advanced video synthesis frameworks
4. State-of-the-art lip synchronization neural networks

5. Avatar Creation Pipeline

The system implements a sophisticated pipeline for generating personalized avatars through a multi-step process:

The pipeline consists of several stages:

1. Users upload images through API Gateway which are stored in Amazon S3
2. Amazon SageMaker Async Interface processes these images through fine-tuning of stable diffusion models
3. The fine-tuned models are hosted on SageMaker GPU Multi-model endpoints
4. The system utilizes Amazon SQS and SNS for asynchronous processing notifications
5. Final personalized models are served through API Gateway for inference

6. Video Chatbot System

The video chatbot component enables natural and engaging conversations with digital avatars:

The system processes interactions through:

1. Video recordings are uploaded to Amazon S3
2. Amazon Transcribe converts speech to text
3. AWS Lambda invokes NeuraChat, Neurons Lab’s specialized chatbot solution, for generating contextually relevant and engaging responses. The system leverages NeuraChat’s advanced capabilities for managing chat sessions and knowledge graph relationships through Amazon DynamoDB and Neptune
4. Amazon SageMaker handles voice synthesis using Whisper
5. The system generates synchronized video with matching lip movements
6. Results are stored in S3 for delivery

The system leverages AWS Bedrock’s foundation models for natural language understanding and generation, ensuring high-quality, contextually appropriate responses. Voice cloning is handled through specialized neural voice conversion models that create realistic voice reproductions while maintaining ethical considerations.

The integration of AWS services offers numerous advantages, such as increased scalability, improved reliability, and reduced operational costs. The system can efficiently process large volumes of data and adapt to changing user demands, ensuring a comprehensive, user-friendly, and emotionally engaging solution that effectively addresses the challenges associated with digital legacy preservation.

Through this technology, the system creates a powerful and lasting connection between generations, ensuring that their stories and memories live on in a meaningful and authentic way.

The system’s knowledge graph in Neptune enables sophisticated memory connections and contextual understanding, allowing for more natural and meaningful interactions with the digital legacy content. This graph-based approach provides enhanced capabilities for discovering relationships between memories, identifying patterns, and creating rich, interconnected narrative experiences.

More on the Digital Avatar approach 

For the Posthumously team, it was key to work with an existing avatar solution which would allow us to speed up delivery time. For this reason, although the Neurons Lab team has the expertise to produce these models from scratch, we decided to explore existing models which would allow us to work faster. 

We carried out various experiments using DECA, MICA, Thin-Plate-Spline-Motion and Photometric among others. Finally, we decided to work with the Photometric model and were able to make some improvements to the existing code and retrain the model. 

The above image shows our proof of concept to test the approach. It allowed us to have a clear understanding of work breakdown structure, timeline and budget needed. 

The Design Sprint format

The Posthumously project follows an innovative design sprint format (originally developed by Google Ventures), which creates the environment for a time-constrained, user-centered, and iterative two week sprint to solve design and product challenges efficiently. It is a framework that allows stakeholders both from Neurons Lab and Posthumously to ideate, prototype, and validate solutions to complex problems.

The objective of the 2-week design sprint was to create a comprehensive plan and framework for the development of a groundbreaking application that would revolutionize digital commemoration. The specific goals of the design sprint were as follows:

1.Define the Work Breakdown Structure (WBS) for the application development. The team aimed to create a detailed breakdown of tasks, deliverables, and timelines to ensure a well-organized development process.

2.Incorporate AI-driven technologies for avatar creation and content generation. The design sprint sought to explore the integration of advanced AI technologies, such as LLM and generative AI, to enable the creation of lifelike digital avatars and personalized content generation.

3.Address psychological aspects of grief. The team wanted to ensure that the application was sensitive and supportive in addressing the psychological aspects of grief, providing users with a meaningful and healing experience.

4.Validate and refine the user experience. Through wireframing, prototyping, and user testing, the team sought to create a user-centric design and enhance the overall user experience.

5.Identify risks and mitigation strategies. The design sprint aimed to identify potential risks associated with developing the Minimum Viable Product (MVP) and devise strategies to mitigate them effectively.

Through the design sprint model, the team achieved focus on the real market needs, development of features that help stand out from the competition, user-centric design, redefined Digital Commemoration, positive impact on users, and leveraged the power of AI. Deliverables included:

• Market Research Report. Insights into user needs, market trends, and competitive analysis.
• User Interviews Analysis and Customer Persona Report. Detailed analysis of user interviews and creation of customer personas.
• Problem Statement and Value Proposition Report. Clear articulation of the problem being addressed and the unique value the application offers.
• User Journey Map. Visual representation of the user’s emotional journey and interactions with the application.
• UX Wireframes and Prototypes. Interactive representations of the application’s interface and user flows, refined through user testing.
• LLM and 3D Photogrammetry Prototypes. Demonstrations of personalized content generation and lifelike digital avatars.
• MVP Risks Matrix. Identification and mitigation strategies for potential risks associated with MVP development.
• WBS for the MVP. Detailed breakdown of tasks, deliverables, and timelines for the development of the Minimum Viable Product.

Results

The Posthumously System is a groundbreaking solution to the challenges of digital cryonics, offering significant potential in terms of user engagement, data accuracy, and customer satisfaction.

By integrating advanced technologies such as Amazon Web Services (AWS), OpenAI’s GPT-4 and GPT-3.5, and custom ML models, the system captures, preserves, and presents the memories and stories of individuals in a comprehensive and emotionally engaging experience.

1. User Engagement. Based on initial feedback and user testing, the Posthumously System has the potential to significantly increase user engagement compared to traditional digital memory preservation methods, as users find the digital avatars and interactive storytelling more compelling and emotionally resonant.

2. Data Accuracy. Leveraging AWS services and custom ML models, the memory extraction and preservation process aims to achieve a high accuracy rate, ensuring that the preserved memories remain true to the individual’s experiences and stories.

3. System Evolution. By continuously integrating the latest advancements in AWS and OpenAI technologies, the Posthumously System is well-positioned to adapt to changing user demands and maintain its position at the forefront of the digital cryonics industry.

The Posthumously System is a promising, user-friendly, reliable, and scalable solution that effectively addresses the challenges associated with digital cryonics. With its potential to deliver quantifiable results in the form of increased user engagement, data accuracy, and customer satisfaction, the system ensures that the legacies of individuals are preserved, enabling future generations to connect with their ancestors, learn from their experiences, and keep their memories alive.

The outcome of the design sprint has greatly assisted Posthumously to establish a workable plan of action for building the first version of the application and roadmap for accurately predicting the number of sprints needed to get a working first version of the Posthumously application. I believe every tech startup should consider adding a design sprint, pre-MVP development stage in order to re-evaluate and vet product or application requirements before embarking on development.

About us: Neurons Lab

Neurons Lab delivers AI transformation services to guide enterprises into the new era of AI. Our approach covers the complete AI spectrum, combining leadership alignment with technology integration to deliver measurable outcomes.

As an AWS Advanced Partner and GenAI competency holder, we have successfully delivered tailored AI solutions to over 100 clients, including Fortune 500 companies and governmental organizations.