Goal
Our research is currently being focused on "enhancing handwriting interfaces" which is aimed to provide creative human computer interfaces by handwriting.
Motivation
Although character recognition is language dependent, handwriting itself is borderless. English, Chinese, Arabic, Japanese or even pictorial languages can be expressed with a single pen. With handwriting, one can express one's thinking most easily using pen-trace patterns. Moreover, thinking is not interrupted by the actions for handwriting. Thinking and handwriting forms a positive feedback to grow and clarify one's idea. These natures of handwriting are suited for creative work rather than labor intensive tasks.
To realize creative human computer interfaces by handwriting while inheriting the above natures, handwriting recognition technology, handwriting human interfaces as well as applications must be studied and enhanced.
Even without machine recognition, however, there are many applications where the pen is mightier than the mouse. With recognition, the power of handwriting is greatly extended.
Structure
Roughly speaking, our research is composed of four layers as shown in the above figure.
The Infrastructures to promote handwriting interfaces and recognition include efforts to define electronic ink format, development of library routines to process electronic ink and provision for the large database of on-line handwritten character patterns which includes more than 1 million patterns at the moment and will grow up to 3.5 millions in two years.
We have been also working on on-line handwriting recognition technology for more than 15 years and possess one of the highest performance systems in Japan. Our latest system is composed of coarse classification, linear-time elastic matching, structured character pattern representation and context post-processing. It has marked 90 to 95 % correct recognition rates without learning to the above large database of on-line handwritten Japanese text. The system is not only robust to pattern distortions but also highly customizable for personal use. Upon the request of learning an input pattern, it investigates which subpattern (radical) or the pattern as a whole is non-standard, registers the (sub)pattern and extends the effect of the registration to all the character categories whose shapes include it.
Human interfaces are equally important as handwriting recognition technology so that handwriting interfaces can be accepted by users. We have been proposing lazy recognition scheme since 1990 which delays the display of recognition until needed and provides easier structure to employ context post-processing. Until then, experimental systems and products had been trying handwriting recognition immediately after each pattern was written with the result of interrupting user's thinking due to correction of misrecognition and confirmation of correct recognition. One's thought is better developed by working with one's handwriting. Pattern recognition cannot be dispensed with only when handwriting is viewed as a group of codes.
So far, human interface studies have been conducted which clarifies merits and demerits of pen against mouse, and several specific interfaces have been developed for text preparation, figure drawing, mathematical formula input by handwriting, creating animation and so on.
Last, but not least are applications. Without meaningful applications, nobody would use handwriting interfaces. We think that a group of such applications is that involves creative thinking. Moreover, pen is a scalable interface, i.e., it can be used common for PDAs, desktop display integrated tablets and large interactive boards. The total system may provide such usage as the following: You write down memos and ideas on a PDA while you are away from your office and download them to the desktop environment and make them recognized when necessary. You communicate others by handwritten e-mail and search handwritten mails using a searcher based on pattern matching technology. You may invoke recognition engines to process handwriting and utilize computer processing. You may also prepare materials for the lecture on the desktop and make a lecture on the electronic board while utilizing the materials prepared on the desktop. The live lectures are recorded and reviewed on the desktop or PDA. The lazy recognition scheme is employed throughout the system where appropriate.
Combinations of small, medium-size, and large pen input devices may seems similar to the ubiquitous computing at PARC. But, our aim is to provide the power of computing to our behaviors of handwriting which are made on media including various size of papers, notebooks and a white/black board. In contrast to this, the PARC's research is oriented more for collaboration where the small device (that is called ParcTab) is mainly used to locate where the user is rather than for writing down memos or ideas.
Research strategy
Our basic stance is to study handwriting recognition, handwriting interfaces and applications all spirally and pursuit improvement and enhancement in each of them. Of course, each cannot be easily perfected. Even if each is neither perfect nor generic enough, however, a well balanced combination of a well-tuned engine, well-designed interface and useful, effective or enjoyable applications may provide the basis to work on and enhance each of them. Even if each of them has some weak points, such a combination to compensate them could be found if they are studied at the same place and grasped by all the members.
After the boom of pen computing ended in a short period, it has been suffering from the disappointment that the premature products for inappropriate applications have brought to users. Nevertheless, we think that the pen has enormous potentials to future applications.
Financial support
A large financial support is being given by the Advanced Software Enrichment Project of IPA under MITI of Japan. Several companies are also collaborating on this subject with us.