When it Rains: An AR storybook

Padmasree M M

Introduction

Augmented Reality (AR) is revolutionizing education, making learning fun and engaging. AR books transform storytelling with animations, sound, and vivid visuals, fostering imagination and critical skills. Unlike traditional books, AR brings stories to life, making reading exciting and nurturing a love for learning in the digital age.

The project

What am I doing?

This is an Interactive children’s storybook for ages 5-6 years. The stories come alive in front of eyes in Augmented Reality. The Illustrations in each page in the book act as a marker. It comes alive when scanned through the app. It can be used with ipad, tablets and smartphones. With the AR, the storybook gets a new dimension, where children can experience animation and sound along with the comfort of portability.

Narrative

About story

It is a simple story about two friends coming back from their school on a rainy day. It’s about things they see and the little moments. Story depicts the love, kindness and friendship for each other. Story is narrated through conversation between the kids.

Character Design

There are two main characters in the story. Sophie and Ammu are best friends. They care for each other. Both are mischievous and playful.

Storyboard

Story

Simple story from childhood About two friends coming back from the school on a rainy day Things they see and little moments About their love, kindness and friendship for each other Narrated through the conversation between each other

1.Ammu and Sophie are on their way back home from school enjoying the scenic scenery in the background of kerala. Passing by a flower bloomed meadow.

Girl1(Sophie) : Look at those flowers, I can make a beautiful garland with those.
Girl 2(Ammu): Oh no...Don’t pluck them they will love us if we let them live. Sophie: They are alive? Can they see us? Ammu: Yeah, they will die if we pluck them. They have emotions too..

2. Weather is changing to slightly dark cloudy- about to start rain and windy. A frog croaks next to them- it’s about to rain.

Sophie smiles at her best friend and leaves the flower unplucked.

3. Both children are sharing one umbrella, leaving them wet on both sides anyway.

Sophie: I really really love this fragrance of wet mud. And look at those greenery around us! Wow! Ammu: Yeah true! How good is that every year our school reopens on ‘Edavapathy rains’. And we get to enjoy this rain so much.

4. They decide to drop the umbrella and move forward without it and actually enjoying it!

5. Both about to start a race from the hilly top to bottom.
Sophie: Who wins will buy the other ‘Mittayi’ from Dasettan’s shop, cross your heart. Ammu: Yippie, let’s start the race!

6. Local fisherman catching fish. Children are both watching in wonder
Ammu: I will also catch ‘Piloppi’ Sophie: Uncle will you give us one ‘Piloppi’ so that will take it home and put it in the tank?

Uncle(Local fisherman) laughs Children Laughs

7. Both children are about to turn in two different directions, to their homes.
Sophie: If Mummy finds me wet, she will surely scold me. Ammu: Don’t tell her. As soon as you reach home, change your clothes and wipe yourself well. Sophie: Yes, you too and take this ‘Piloppi’ with you.

8. A distant discussion of friends
Ammu: Don’t forget to bring ‘Morukoottan’ tomorrow for lunch Sophie: Yes, will tell Mummy.

9. A traditional home interior, where the kid is half dried with the towel.
Sophie/Ammu: Achheeee!!(Sneeze)

Visual Development and final illustrations

Research and Inspirations

As part of research, I looked at current AR projects, Interactive storytelling including comics and storybooks. I also looked at books with physical interactivity.

Explorations

Slide AR

Slide AR is an augmented reality app where artists can upload layer by layer illustrations and view it in AR. But the disadvantage of the app is that you cannot export it.

Adobe Aero

Adobe aero is another augmented reality app where you can upload the psd and view it as AR. It has a lot of options to add interactivity also without any coding knowledge. But the problem with the app is that you can only send through links to view this. And the export formats won’t support unity or any other software. So you can’t make it into an apk file.

Unity

Augmented reality (AR) can be defined as the overlaying of digital content (images, video, text, sound) onto locations and it is typically experienced by looking through the camera of an electronic device such as a smartphone, tablets, etc.

Using sprites

The illustration is kept as a marker for the content. Each layer is imported to unity as sprites and sprite animations. It is kept above the marker in the order of appearance and height. Sprite masking is also done to view it inside panels.

In order to view it hold the illustration in front of the webcam. The layers and animation will appear above the illustration.

Using video playback

Instead of uploading layer by layer, a video playback is kept by adding a video component for the marker. On viewing the marker illustration, an animated video will be playing.

Animation playing with marker illustrations

Production Process

Final illustrations were done layer by layer from the storyboard in the procreate app in ipad. Small animations were also done along with it.

Layer by layer pngs are imported to unity. Illustrations of the book are kept as image targets. Portal is implemented on a plane above image targets. Through Portal users can view animated layers with 3D parallax effect. Illustration is kept layer by layer in different depths in another place called WorldB(B is 1 to 12 for each image target). This is captured by another camera(CameraB). As the image target rotates and moves (That is, image target rotates and moves) WorldB is also moved accordingly. This is viewed through the portal plane.

PortalSetup.cs is the script to change the rotation and position of WorldB. Portal plane rendertexture is initialised as the CameraB target texture. In update, WorldB position and rotation is changed according to image target position and rotation in local space.

Along with Augmented Reality content sound is also played while viewing through the AR app. Dialogues between the characters and background sounds are added for each image target. Dialogues are recorded and mixed with background sounds in Premiere pro software and exported in mp3 format. These sounds are then taken to unity and added for each image target by adding script in DefaultTargetEventHandler.

Challenges and Learning

Portal implementation was the most challenging thing I faced in this project. Several methods were tried and finally it was achieved with the help of Prof. Jayesh Pillar and Amarnath Murugan helped me achieve the same. To find the correct positioning of layers in the space took several iterations. Through this project i have learned a lot in Unity, C# coding. I have also improved my skills in illustration.