The basic concept of Variable Reality is essentially based on the idea of enabling users to read a digital book anywhere while maintaining the physical quality of a real book. The Variable Reality system uses a lightweight and compact head- mounted display (HMD) device that is see-through. Two types of books designed for the system are the Augmented Book and the Virtual Book.
Variable Reality Virtual Book
Variable Reality Virtual Bookshelf
Variable Reality Demonstration Video
Variable Reality Setup —a physical book (left) and the HMD (right)
(a) Index Fingertip —the user’s index
fingertip is used for interaction just
like a mouse cursor on screens
(b) Quick Swipe —a quick opening and
closing of the hand motion
(c) Stay Click —staying (or hovering) at a
place for 1.8 seconds using the index
(d) Open-Close Hand —a quick motion of
moving the hand from right to left while
keeping the pointing gesture
Based on the stable book- and hand-tracking algorithms along with the recognition of various gestures designed for user interaction, I built several Variable Reality applications for both the Augmented Book and the Virtual Book. These applications import convenient navigation techniques available on digital books and yet maintain the physicality of a paper book. The goal of each application is to create an intuitive and effective reading experience for the users. Incorporating natural hand gestures associated with reading activity adds to the positive user experience in reading.
In the Augmented Book scenario, the user can take any book from the real environment and turn it into a digital book, which can be seen through the HMD (as shown in the image above). The book becomes an augmentable surface for overlaying paper pages with digital information without the need of a visual marker for tracking. As a result, it enables the user to quickly switch between reading a paper book and reading a digital book with a single physical book. The title of a book is detected using a feature detector and the page number can be detected with the help of the OCR technology. This enables the user to store and retrieve digital data on a specific physical page. The Augmented Book also integrates natural hand gestures associated with reading activity in order to provide an intuitive reading experience. A fingertip is predominantly used as a mouse cursor but is used as a pen tip for annotations.
Ideally, the Virtual Book makes it possible for users to hold a 3-D virtual book in the hand and read it anywhere. It is designed to bring extended mobility and accessibility to the digital library designed for on-the-go reading experience. The user does not need anything other than the hand to bring up a virtual book on the palm. Similar to the Augmented Book, it integrates the hand gesture recognition system, which enables users to efficiently manipulate the book.
Variable Reality further incorporates a three-dimensional digital bookshelf that closely resembles the look of a real bookshelf. Using a simple gesture, the user can display it, zoom into a portion of it and pick out a book to read. This book can be in the form of either the Augmented Book or the Virtual Book.
(a) Open/Close Hand (Face Up) —a quick
opening and closing the hand with the
palm looking up
(b) Right Hand Swipe —a swift, right-to-left
hand movement using the right hand
(c) Right Hand Twist —a quick rotating
motion of the right hand around the
middle finger using the wrist
(d) Right Hand Pinch —a hand gesture similar
to how one would pinch someone
(e) Two-Hand Swipe Open/Close — a quick
two hand movement separating apart
from one another
(f) Finger Fold —folding one of the five
(g) One-Hand Swipe —left hand movement
from left to right with the palm facing up
(h) One-Hand Quick Bounce —a sudden
down and up hand movement
(i) Quick Open-Close (Face Down) —a quick
closing and reopening of the left hand
Variable Reality is a novel Augmented Reality (AR) based interactive system focused on bringing intuitive on-the-go reading experiences to the users by enabling always-available access to and presentation of any digital book contents in a complete spatial form of a book. Variable Reality introduces two types of books: the Augmented Book and the Virtual Book.
The Augmented Book incorporates computer vision based corner detection and tracking systems to detect the corners of a physical book, information of which is used to overlay paper book pages with digital pages using a Head-Mounted Display (HMD) device. As a result, any real book can instantly turn into a different book while still retaining the physical properties of the book, conforming to the legacy of the paper book metaphor. The Virtual Book is an entirely virtual 3-D book that extends the accessibility and mobility of reading materials. When in absence of a physical book, the user can retrieve this virtual book on the palm of a hand –a well-established pose of how one holds a book when reading. The weightless and portable nature of the Virtual Book makes it possible for users to comfortably hold larger physical books such as unabridged dictionaries or the World Book Encyclopedia.
In both augmented and virtual book scenarios, natural hand gestures inherently associated with the reading activity are defined and used in various Variable Reality applications.