\begin
Discover why 18 million people worldwide trust Overleaf with their work.
\documentclass[11pt, a4paper, twocolumn]{article}
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage{ulem}
\usepackage{lettrine}
\title{Enhancing education through natural interaction with physical paper}
\author{Victor François}
\date{\today}
\begin{document}
\maketitle
\section{Context}
\lettrine{N}{owadays access to technology} \\in schools is no longer a luxury. New technologies become necessary for the survival of the students. The integration of new technologies can help restructure the classroom with activities fostering collaboration. It can also provide effective tools to improve interpretation and data management. Technology can also provide an excellent avenue for student motivation, exploration and instruction in a multi sensory, diverse world. Technology, however, is only a tool. The challenge rests with educators to effectively integrate it in appropriate places throughout the curriculum \cite{AEBGWO}.
The problem is to know how to improve the education of students. One maner is to extend human capabilities in order through physical interactions. These are taken into account by different frameworks that can support natural interactions with physical objects like books or pens. Such system must support a wide range of types of education to fit as naturally as possible in each school. It must be intelligent to detect books or pens, adopting learning practices currently used such as read or write and to give additional information on them.
Now we will focus on existing systems and then we will study more precisely one of these systems based on an AR (Augmented Reality) Study Desk.
\section{State of the art}
In the 90s, the idea of having digitally augmenting physical papers began with two projects : DigitalDesk and a little later EnhancedDesk. These systems use computer vision technologies for physical objects recognition and localization, enabling intuitive interfaces that seamlessly link physical and digital documents. They are the precursors of the system that I am going to present.
\begin{figure}[!ht]
\caption{\label{Sys1} EnhancedDesk}
\hspace*{0mm}
\vfill
\begin{center}
\includegraphics[scale = 0.3]{enhanced_desk.png}
\end{center}
\vfill
\hspace*{0mm}
\end{figure}
The MagicBook[9] explores seamless transition between reality and virtual reality. When users look at the pages of a real book through a hand held display they can see virtual content superimposed over the real pages. When they see an Augmented Reality scene they like, the user can fly into the scene and experience it as an immersive virtual environment. In addition, the book serves as a focus for collaboration. When several users look at the same book page they can see the AR image from their own viewpoint. In this way, the MagicBook supports collaboration as a physical object, a shared AR experience and a multi-user immersive virtual environment. The MagicBook has many possible applications including education.
\begin{figure}[!ht]
\caption{\label{Sys2} MagicBook}
\hspace*{0mm}
\vfill
\begin{center}
\includegraphics[scale = 0.14]{MagicBook.png}
\end{center}
\vfill
\hspace*{0mm}
\end{figure}
The Book of Elie is an another interesting experience that can identify books and printed cards on the desktop. When a user's finger touches a book page the system responds by providing additional information or launching an exercise. Cards that are known models of the system allow the user to meet the different proposed exercises. The Book of Ellie addresses younger children and edutainment activities, where learning is achieved by play.
PapierCraft and SESIL are systems using pen gestures on paper to support active reading. They allow users to carry out a number of actions, such as to copy and paste information from one document to another. It is interesting to use a pen to add information to the hand attached to a document. They were designed having in mind a typical classroom activity of reading and exercise-solving.
\begin{figure}[!ht]
\caption{\label{Sys3} PapierCraft}
\hspace*{0mm}
\vfill
\begin{center}
\includegraphics[scale = 0.3]{PapierCraft.png}
\end{center}
\vfill
\hspace*{0mm}
\end{figure}
\section{AR Study Desk}
\subsection{Motivations}
In order to integrate these technologies, the system must be invisible \cite{GDA}, using techniques of passive vision in real time and non-instrumented input devices i.e. that the system must be seamlessly without the need to be constantly reprogrammed by users. It must be a smart environment that evolves from the definition of ubiquitous computing and calm technology [3]. There should be an extension of our unconscious to feel no gene for the best use \cite{MW}. Furthermore, it should be easily adopted in the programs of education and support. For this it is necessary to integrate "general public" equipments for schools and students wishing to easily get them at a reasonable cost.
We can classify previous models in three different groups : the first recognizes the shape of the hands for them to interact with a book. The second uses its real patterns recognizable by the system, each of which corresponds to a response to this one. And the third uses the interactions of the user with a pen. AR Study belongs to the first group that we will compare to others in the results section to test if it is a simple and intuitive model.
\subsection{Realization}
The AR study desk targets more exploratory activities where the learner receives information about a topic alone or in collaboration.
It is a smart environment that allows to provide additional information to a virtual real support with a computer, a motion sensor and pattern recognition, and a display device.
Choosing a computerized environment was undeniable/essential to combine the benefits of a conventional education process on paper with those of e-learning and using augmented reality.
According to Cook et al. \cite{DJCSKD}, smart environments can be broken down into four basic layers playing a different role in the environment, facilitating various operations and addressing specific requirements :
\begin{itemize}
\item Physical : It’s the vision processing components.
\item Communication : ICS FORTH Middle-ware — a distributed service oriented middle-ware, provides a common dialect for applications to coexist and communicate in the context of the developed application scenarios while using sensing for decision making \cite{NPCSMA}.
\item Decision : This component has been developed for educational book modeling and to facilitate the necessary context sensitive information provision to educational applications that integrate the proposed framework. The recognition library has been enriched with educational meta-data to provide additional content to the book and facilitate learning.
\item Information : It's the educational application AR Study desk.
\end{itemize}
\subsection{Results}
The main evaluation is to measure how each system is intrusive compared to other existing systems mentioned previously. ref 51 58 59
For this evaluation, 60 participants tested the different types of system. All participants know how to use a computer for various multimedia applications. 75\% are students in computer science and 25\% are parents who use educational software for their children.
After testing each system a questionnaire (see figure) is filled by the participants to evaluate them.
\begin{figure}[!ht]
\caption{\label{quest1} Questionnaire used to analyze the different systems.}
\includegraphics[scale = 0.3]{questionnary.png}
\end{figure}
According to the evaluation, Sesil seems to be the most difficult system to use. This is due to the interaction employed techniques. For example, for the system can interpret the user's text, it is imperative that he writes in large letters. Moreover, it is difficult to know what is the gesture associated with a task. In the other hand, AR Study desk is considered more intuitive in the gestures to use. Finally, the Book of Ellie was also considered straightforward, enjoyable, and easy to predict.
\begin{figure}[!ht]
\caption{\label{Res1} Analysis of the evaluation questionnaire results}
\includegraphics[scale = 0.25]{res1.png}
\end{figure}
\begin{figure}[!ht]
\caption{\label{Res2} Analysis of the evaluation questionnaire results regarding interaction techniques}
\includegraphics[scale = 0.3]{res2.png}
\end{figure}
\section{Implementation}
AR Study desk System uses the ASUS Xtion Pro composed by a camera color for printed matter recognition and a depth sensor which is able to localize and track the hands and fingertips of the user to provide interaction data \cite{A}.
The page recognition module uses electronic representations of books saved in the data base of the framework and compares them with the live image seen by the camera. Each page of each book is a representation obtained by detecting the SIFT keypoint feature of the image. To generate these features the system uses invariant features to image scaling and rotation, and partially invariant ones for change in illumination and 3D camera viewpoint to be analyzed in many situations.
The major stages of computation used to generate the set of image features are in first a calculation of the scale and positions of the images, in second the keypoint localization and the orientation assignment which one or more orientations are assigned at each keypoint location based on local image gradient directions. (PARAPHRASE) {Keypoint descriptor: The local image gradients are measured at the selected scale in the region around each keypoint. These are transformed into a representation that allows for significant levels of local shape distortion and change in illumination \cite{DGL}. Many candidates can be chosen to be compared between them. That is a "hit ratio" is computed as the fraction of the number of the matched features over the number of the total page features. We deduce the current image as the one with the best ratio. Furthermore to improve performance, we will first considering the previous and next page. If the ratio of one of them is high it is chosen immediately.
During installation, the camera is calibrated with the plan of the desk. This allows to estimate the location and orientation of the book by retrieving the representative image synthesis. With this estimation the keypoints associated to the page permit to localize it.
Using correspondences between real image and virtual representative image and approximating the 3D shape of a page with a planar patch, the homography that maps an imaged page to the electronic one is estimated through a RANSAC (RANdom SAmple Consensus) procedure, i.e. a procedure for estimating the parameters of a model in such a way to explain the best possible data. Here that permits to predict the corner of each scanned page in the 3D world and the acquired image and thereby the image surface. The process is performed for each page on the desktop.
To be able to interact with the system the user must use his fingers for pointing data. With the calibration of the camera relative to the desk we know the equation of the plane associated with it. The depth camera can calculate the depth map and transform it into 3D world coordinates. Then it’s possible to calculate the distance for each pixel to the desktop. Two thresholds are required to provide fingers with a contact on the desk. First to indicate whether a user has a finger to a sufficient distance from the desktop for this to be considered as an interaction and a second to remove all the parts that appear too far from the desktop.
The interaction with the book is necessarily different as the book is not a plan, it has curves. With the estimation of the position and orientation of the book it is simple to extract contours and thus to extract this depth map that will be transformed into the desk’s coordinate frame. Using this information, a height map of the book’s surface is generated. So if a finger touches a book, this height map will be incremented at this location.
\section{Conclusion}
\sout{results indicated that touch-based interaction was considered intuitive and easy to use, card-based interaction was characterized as appropriate in the context in which it was proposed, while pen-based interaction was more cumbersome for the users, due to technical difficulties with the handwriting process. Furthermore, an important parameter affecting the unobtrusiveness of a given system in the educational process was the usability of the system and the straightforward nature of the interaction technique. Another evaluation conclusion was that most of the users were positive toward using technology in the context of
educational activities and that the usability problems that they encountered with a system did not affect their view regarding the adoption of the system in the educational process.}
The results obtained by the survey of sixty people showing positive points in AR Study Desk system. In fact, respondents presents a simple and intuitive system in interactions through touch. It's easy to predict. Furthermore, adding virtual information to real objects is appreciated by the users.
As we have seen The Book of Ellie is presented as the simplest and fastest system to handle with this cards. However, it is for a very young audience, it is difficult to imagine to use it for the education of more experienced people despite the limitations of its use.
Unlike AR Study desk system is viewed as able to be used in a large number of educational activities. These are significant points for the development of new technologies in teaching.
+Parler des faiblesses
lack of assistive applications
We can imagine other solutions to complete such systems as recognition and interpretation of user speech.
\onecolumn
\bibliographystyle{plain}
\bibliography{biblio.bib}
\end{document}