LAVER LABS

Intro: How to teach robots to perform complex tasks

Until now, many robots have performed tasks through motion based on pre-input location information under structured environments such as factories. Unlike factories, however, many daily tasks are performed in unstructured environments and require physical interactions with diverse external objects.

Let’s imagine, for example, the task of spreading jam on a piece of bread. Depending on the stiffness of the jam and bread, you need to adjust the pressing force properly while moving. How about erasing a whiteboard with an eraser? We understand that, in order to erase a whiteboard cleanly, we need to move the eraser by pushing it down with adequate force. We also know that a greater force is needed for parts that were written a longer time ago and thus are more difficult to erase. As you can see, it is extremely important to control the force (of contact) when coming into contact with an object while performing a daily task.

In other words, a robot needs to know not only how it should “move” to perform a task, but also what kind of “force” it has to exert when interacting with the environment, because physical interactions that enable many of the daily tasks are achieved by force as the medium. In contrast, the environment in a factory is nearly perfectly known, and almost no interactions exist other than grabbing of objects. Since the characteristics of the environment and tasks required are completely different, teaching of daily tasks must be different from the method applied to factory robots.

Physical intelligence learned from humans

To get an idea of how to teach a robot a task that requires physical interactions, let’s think about how a person teaches and learns a task. For a person, there are many things that one does unconsciously in addition to activities done consciously when interacting with the external environment by handling an object or executing a certain action. When spreading jam, one does not think about every movement of the fingers or the force exerted on the wrist and fingers. The same goes for the task of removing coins from a pocket; the task itself is executed easily without being conscious of it and thinking of every small motion of the arm and fingers. This is because such tasks take place in the domain of unconscious motion, which is why it is difficult to explain or program such processes and principles implemented in this domain using a language. We call this knowledge and intelligence based on such physical experiences “physical intelligence”.

Figure 1. (Source: IKEA)
If you look at a furniture assembly manual, it explains the assembly method in an extremely abstract manner. It has almost no detailed explanation of how firmly a part needs to be inserted, how it has to be secured, or in which direction the force needs to be exerted. Despite missing all this information, how can a person assemble furniture just by seeing a manual like this?

The reason that we can perform a complex task like assembling furniture from a simple assembly manual alone is the physical experience that we commonly possess. What if we are able to teach such intelligence to robots? The problem is that, because physical intelligence takes place in the domain of unconscious motion as mentioned earlier, it is impossible for us to express it in words, letters, or codes to deliver it to robots. If we can effectively deliver such physical intelligence to robots, we will be able to teach them a greater volume of tasks in a remarkably more effective manner.

Development of the haptic device

What if a coach has to teach a new skill to an athlete using only words? It sounds difficult enough just thinking about it. It would probably be faster and more accurate for the coach to show the motion in person, then have the athlete try it on their own. Would this be possible for a robot as well? The haptic device began from this question. Through a haptic device manufactured at a scale identical to humans and by seven degrees of freedom just as that of a person’s arm, the human physical intelligence and force control ability that cannot be explained by programming or words are extracted to be used as learning data for robots.

Unlike common control devices, the haptic device delivers location and force data bilaterally. It not only allows a user to deliver commands to a robot on where to move, but also enables the user to feel the force experienced by the robot. A person performs a task by controlling a robot through a haptic device, and in the process the robot indirectly obtains human physical intelligence.

Figure 2. Haptics: Augmented experience of human physical intelligence
Experiencing something and “gaining” new knowledge from an experience or “learning” intelligence are different concepts. Despite having the same experience, individual people may see results of dissimilar degrees of learning. However, even if two people possess perfectly identical learning abilities, the effect of the learning may vary significantly depending on what kind of “information” has been gained through their own respective “experiences”.

I will now explain the method through which a haptic device enables a robot to have physical “experiences” and what kind of “information” is accumulated. To be able to deliver human physical intelligence to a robot, a person needs to perform the particular task similarly to how they would normally execute it and as naturally as possible. To accomplish this, the process is being enhanced in the following direction.

Design structure capable of delivering human motions to robots well

As mentioned earlier, a haptic device, which has to teach human tasks, must of course cover the space throughout which a person’s arm can move. This is how the haptic device has been developed, with seven joints, possessing similar degrees of freedom to those of a human arm. Six joints are sufficient to express all locations and directions of a space, but the posture of the haptic device is made more natural by assigning an additional redundant degree of freedom.

Parts with relatively fewer movements have been designed for lightweight motors to exert a sufficient amount of force, whereas parts with greater movement have been manufactured in such a way that they easily sense and accept a human force, and movements related to wrist motion have been designed so as to be executed independently. Such a structure is convenient to control and capable of effectively delivering a human-like force, which has led to the following form for the haptic device.

Control algorithm for safe interactions

To say that a human motion can be naturally conveyed to a robot means that a robot’s movements must also be naturally conveyed to a human. For such mutual exchange of motions, a robot and a human need to be physically connected to each other. However, since these two entities are constituted of physically independent and separate systems, we have created what we call “bilateral teleoperation”, by which the information of force and location can be shared between a human and a robot.

Bilateral force control achieved by linking with a simulator (view all demo videos)

Despite such efforts, however, differences between a robot and a human are inevitable in expressing a robot's motions using those of a person. In terms of the scope of limitation in physical movements, limits of exertable force, types and composition of joints, etc., humans and robots possess similar but distinct characteristics. Thus, if a person cannot concentrate on the target task and has to consider various limitations of a robot when using a haptic device to operate it, the robot cannot help but produce a movement that is different from the otherwise natural human motions in executing a given task. To exclude the considerations based on such dissimilarities between a human and a robot, we have utilized the optimization controller of a hierarchical structure so that the robot considers its own limiting conditions with the utmost priority while mimicking human movements as closely as possible.

The haptic device and control method developed through this process can be summarized as shown in the image below. Through this structure and algorithm, the NAVER LABS haptic device enables a person to convey a task to a robot in an easier, more convenient manner.

Figure 3. Control structure of AMBIDEX and haptic device

Variety of demos using the haptic device

This is a demo video of executing various tasks using the haptic device explained above. Though they appear simple to us, they are tasks that can only be performed through elaborate force control of the robot and bilateral force transfer of the haptic device. If any of these do not work properly, excessive friction force would be exerted between the knife and the potato to potentially destroy the ingredient, the tool, or the robot itself.

Demo of peeling a sweet potato

The video below is a demo of catching a flying ball in a net. If you look closely at the video, you can see that the robot is grabbing a pole with both of its arms. Here, the operator controlling the haptic device also feels the same restraints of motion as if grabbing the pole with both arms. If bilateral force control is not implemented properly and the operator moves the haptic device carelessly, the pole or the robot may break as excessive force is applied when trying to follow the motion of the haptic device. In the demo below, you can see that it operates rapidly and flexibly.

Demo of catching a flying ball

I have thus far explained why NAVER LABS is creating the haptic device and how we have developed it. The purpose of the haptic device is to acquire data for conveying physical intelligence of a human to a robot, so ultimately, the data gained from human movements needs to be used and learned for robots to possess physical intelligence. NAVER LABS will continue to research and develop not only haptic devices to obtain better, more enhanced learning data, but also ways to utilize the data acquired.

AMBIDEX task learning project interview video