JP7388352B2 - Control device, control method, and program - Google Patents

Description

The present technology relates to a control device, a control method, and a program, and particularly relates to a control device, a control method, and a program that can notify a user of an abnormality that has occurred in a robot in a form that can be easily confirmed.

Robots are beginning to be used for a variety of purposes, such as domestic service robots and industrial robots.

If a robot part breaks, it must be repaired or replaced. It is difficult for the average user to confirm abnormalities such as broken parts by analyzing information such as error logs output by the robot system. Such problems are particularly noticeable in home-use service robots.

Japanese Patent Application Publication No. 2002-154085 Japanese Patent Application Publication No. 9-212219

It is desirable that even a general user can easily check the abnormality that occurs in the robot.

The present technology has been developed in view of this situation, and is intended to make it possible to notify a user of an abnormality that has occurred in a robot in a form that can be easily confirmed.

A control device according to an aspect of the present technology includes an abnormality detection unit that detects an abnormality that occurs in a predetermined part of the robot, and a control device that detects an abnormality that occurs in a predetermined part of the robot, and that controls the robot so that the abnormal part detected by the abnormality detection part is included in the angle of view of the camera. The apparatus includes an attitude control section that controls the attitude, and a recording control section that controls the camera and captures a moving image of the abnormal region while it is being driven.

In one aspect of the present technology, an abnormality occurring in a predetermined part of the robot is detected, and the posture of the robot is controlled so that the detected abnormal part is included in the angle of view of the camera. Further, the camera is controlled to take a moving image of the abnormal region while it is being driven.

According to the present technology, it is possible to notify a user of an abnormality that has occurred in a robot in a form that can be easily confirmed.

Note that the effects described here are not necessarily limited, and may be any of the effects described in this disclosure.

1 is a diagram illustrating a configuration example of an information processing system according to an embodiment of the present technology. FIG. 3 is a diagram showing an example of an abnormality notification. FIG. 2 is a block diagram showing an example of a hardware configuration of a robot. FIG. 3 is a block diagram showing an example of a functional configuration of a control unit. FIG. 3 is a diagram showing an example of a world coordinate system. FIG. 3 is a diagram showing an example of a coordinate point sequence. FIG. 3 is a diagram showing an example of an abnormality notification image. 3 is a flowchart illustrating abnormality notification processing of a robot. 9 is a flowchart illustrating posture control processing performed in step S4 of FIG. 8. FIG. 7 is a diagram showing another example of an abnormality notification image. FIG. 7 is a diagram illustrating an example of an alternative process of having another robot take a picture. FIG. 7 is a diagram illustrating an example of alternative processing that is directly shown to the user. FIG. 7 is a diagram illustrating an example of alternative processing using a detachable camera. FIG. 7 is a diagram illustrating an example of an alternative process of photographing an image reflected in a mirror. 1 is a diagram showing an example of the configuration of a control system. FIG. 2 is a block diagram showing an example of the hardware configuration of a computer.

Hereinafter, a mode for implementing the present technology will be described. The explanation will be given in the following order.
1. Configuration of abnormality notification system 2. Example of robot configuration 3. Robot movement 4. Example of abnormality notification image 5. Example of alternative processing 6. Variant

<Configuration of abnormality notification system>
FIG. 1 is a diagram illustrating a configuration example of an information processing system according to an embodiment of the present technology.

The information processing system shown in FIG. 1 is configured by a robot 1 and a mobile terminal 2 connected via a network 11 such as a wireless LAN or the Internet. The robot 1 and the mobile terminal 2 are allowed to communicate with each other.

In the example of FIG. 1, the robot 1 is a humanoid robot that can walk on two legs. The robot 1 executes a predetermined program using a built-in computer and takes autonomous actions by driving each part such as the head, arms, and legs.

A camera 41 is provided in front of the head of the robot 1. For example, the robot 1 recognizes the surrounding situation based on the image taken by the camera 41, and takes actions according to the surrounding situation.

In this example, a robot that can walk on two legs is used, but robots with other shapes such as robots that can walk on four legs or arm-type robots used for industrial purposes may also be used. good.

As a result of moving your arms, legs, etc., abnormalities may occur in certain parts of your body, such as your joints. Joints are equipped with devices such as motors that physically drive the joints, and deterioration of the devices can cause abnormalities such as the joints not moving as expected. In the robot 1, a process of checking whether each device is operating normally is repeatedly performed at a predetermined cycle.

FIG. 2 is a diagram showing an example of an abnormality notification.

As shown in FIG. 2, for example, when it is detected that an abnormality has occurred in the device provided at the joint of the left arm, the robot 1 controls its posture so that the joint of the left arm falls within the angle of view of the camera 41, and detects the abnormality. The device at the location is photographed by the camera 41. The robot 1 performs image processing to emphasize abnormal areas on the image obtained by photographing, and transmits the image obtained by performing the image processing to the mobile terminal 2.

In the mobile terminal 2, the image transmitted from the robot 1 is displayed on the display, thereby notifying the user that an abnormality has occurred in the device provided at the joint of the left arm of the robot 1. The image displayed on the display of the mobile terminal 2 in FIG. 2 is an image transmitted from the robot 1.

In this way, in the information processing system of FIG. 1, when an abnormality occurs in a device installed at a predetermined part of the robot 1, the robot 1 itself photographs the abnormal part, and an image showing the abnormal part is sent to the user. will be presented. The information processing system shown in FIG. 1 can be said to be an abnormality notification system that notifies a user of an abnormality in the robot 1.

By looking at the display on the mobile terminal 2, the user can easily confirm that an abnormality has occurred in the robot 1.

Furthermore, since the abnormal location is shown in the image displayed on the mobile terminal 2, the user can easily identify the abnormal location compared to when performing tasks such as analyzing the action log of the robot 1. can. The user can quickly repair the abnormal part by himself or inform a contractor about the abnormal part and request repair.

Note that in the example of FIG. 1, a smartphone is used as the device to receive notification of the abnormal location, but other devices with a display, such as a tablet terminal, PC, or TV, may be used instead of the mobile terminal 2. You may also do so.

A series of operations of the robot 1 that detects an abnormal location and notifies the user as described above will be described later with reference to a flowchart.

<Example of robot configuration>
FIG. 3 is a block diagram showing an example of the hardware configuration of the robot 1. As shown in FIG.

As shown in FIG. 3, the robot 1 is configured by connecting an input/output section 32, a drive section 33, a wireless communication section 34, and a power supply section 35 to a control section 31.

The control unit 31 is constituted by a computer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The control unit 31 executes a predetermined program using the CPU and controls the overall operation of the robot 1. A computer forming the control unit 31 functions as a control device that controls the operation of the robot 1.

For example, the control unit 31 checks whether the devices provided at each location are operating normally, based on information supplied from each drive unit of the drive unit 33.

Whether or not the robot 1 is operating normally may be confirmed based on information supplied from sensors provided at each position of the robot 1, such as an acceleration sensor or a gyro sensor. Each device constituting the robot 1 is provided with a function to output information used to confirm whether or not it is operating normally. The device whose operation is to be checked may be a part that makes up the robot 1 and may be a moving part or a non-moving part.

When the control unit 31 detects that an abnormality has occurred in a device provided at a predetermined location, the control unit 31 controls the posture of the robot 1 by controlling each drive unit as described above, and detects the abnormal location using the camera 41. Photographed by The control unit 31 performs image processing on the image taken by the camera 41, and then causes the wireless communication unit 34 to transmit the image to the mobile terminal 2.

The input/output unit 32 includes a camera 41, a microphone 42, a speaker 43, a touch sensor 44, and an LED (Light Emitting Diode) 45.

The camera 41 corresponds to the eyes of the robot 1 and sequentially photographs the surrounding environment. The camera 41 outputs data of a captured image, which is a still image or a moving image, to the control unit 31 .

The microphone 42 corresponds to the ear of the robot 1 and detects environmental sounds. The microphone 42 outputs environmental sound data to the control unit 31 .

The speaker 43 corresponds to the mouth of the robot 1, and outputs predetermined sounds such as spoken voice and BGM.

The touch sensor 44 is provided at a predetermined location such as the head or back. The touch sensor 44 detects a touch by the user and outputs information representing the content of the touch by the user to the control unit 31 .

The LED 45 is provided at each part of the robot 1, such as the position of the eyes. The LED 45 emits light under the control of the control unit 31 and presents information to the user. Instead of the LED 45, a small display such as an LCD or an organic EL display may be provided. Images of various eyes may be displayed on a display provided at the position of the eyes, thereby expressing various facial expressions.

The input/output unit 32 is provided with various modules such as a distance sensor that measures distances to surrounding objects and a positioning sensor such as a GPS (Global Positioning System).

The drive unit 33 is driven under the control of the control unit 31 to realize the actions of the robot 1. The drive unit 33 includes a plurality of drive units provided for each joint axis such as roll, pitch, and yaw.

Each drive unit is provided at each joint of the robot 1, for example. Each drive unit consists of a combination of a motor that rotates around an axis, an encoder that detects the motor's rotational position, and a driver that adaptively controls the motor's rotational position and rotational speed based on the encoder output. Ru. The hardware configuration of the robot 1 is determined by the number of drive units, the positions of the drive units, and the like.

In the example of FIG. 3, drive units 51-1 to 51-n are provided as drive units. For example, the drive unit 51-1 includes a motor 61-1, an encoder 62-1, and a driver 63-1. Drive units 51-2 to 51-n also have the same configuration as drive unit 51-1.

The wireless communication unit 34 is a wireless communication module such as a wireless LAN module or a mobile communication module compatible with LTE (Long Term Evolution). The wireless communication unit 34 communicates with various devices including the mobile terminal 2 connected to the indoor network and external devices such as servers on the Internet. The wireless communication unit 34 transmits data supplied from the control unit 31 to an external device, and receives data transmitted from the external device.

The power supply section 35 supplies power to each section within the robot 1. The power supply unit 35 includes a charging battery 71 and a charging/discharging control unit 72 that manages the charging/discharging state of the charging battery 71 .

FIG. 4 is a block diagram showing an example of the functional configuration of the control section 31. As shown in FIG.

As shown in FIG. 4, the control unit 31 includes an abnormality detection unit 101, an attitude control unit 102, an imaging/recording control unit 103, a notification information generation unit 104, and a notification control unit 105. At least some of the functional units shown in FIG. 4 are realized by executing a predetermined program by the CPU that constitutes the control unit 31.

- Abnormality Detection The abnormality detection unit 101 determines whether the devices provided at each part are operating normally, based on information supplied from each device including the drive units 51-1 to 51-n of the drive unit 33. Check whether

There are various methods for detecting abnormalities in motors and the like provided in joints. For example, Japanese Unexamined Patent Application Publication No. 2007-007762 discloses a technique for attaching a distance meter to a joint and detecting the occurrence of an abnormality based on distance information.

Furthermore, Japanese Patent Laid-Open No. 2000-344592 discloses that the functions and movements of a robot can be controlled by combining the outputs of various sensors such as a vision sensor, microphone, ranging sensor, and posture sensor with the outputs of joint actuators. A method for autonomous diagnosis is disclosed.

Japanese Unexamined Patent Publication No. 2007-306976 discloses a technique for detecting the occurrence of an abnormality based on a motor current value and position information.

Another possible method is to use the error between a measured value and a predicted value representing the driving status of a motor or the like.

When a predetermined action is output (when a control command value is output to the actuator (drive unit)), how the joint angle changes at the next observation time can be determined using a physical model of the robot. It is possible to predict by solving forward kinematics.

If the error between the actual measured value and the predicted value observed at the observation time continues for a certain period of time, for example, it is determined that there is an abnormality in a device such as an actuator or sensor related to the action. Normally, one action is performed by a complex movement of multiple joints, so by moving the devices related to each action one by one and calculating the error from the predicted value, it is possible to identify the abnormal location. becomes.

When the abnormality detection unit 101 detects that there is a device that is not operating normally, that is, there is an abnormality, it outputs information indicating the abnormal location to the attitude control unit 102.

- Attitude control for photographing an abnormal location The attitude control unit 102 has information on the installation position of each device. The installation position of each device is represented by three-dimensional coordinates in a world coordinate system whose origin is an arbitrary point when the robot 1 is in its initial posture.

FIG. 5 is a diagram showing an example of a world coordinate system.

In the example of FIG. 5, a world coordinate system whose origin is a point on the floor directly below the center of gravity of the robot 1 is shown. The posture of the robot 1 shown in FIG. 5 is the initial posture. A world coordinate system may be set with the origin at another point such as the vertex of the head.

The installation position of each device provided at a predetermined position such as a joint is represented by the values of three-dimensional coordinates (x, y, z) in such a world coordinate system.

Additionally, the attitude control unit 102 has information on the three-dimensional coordinates of each point on the device in a local coordinate system with an arbitrary point on each device as the origin. For example, a local coordinate system whose origin is a movable joint point of a rigid body that constitutes the device is set.

Posture control unit 102 calculates the coordinates of the abnormal location detected by abnormality detection unit 101 based on information on these three-dimensional coordinates.

For example, the posture control unit 102 obtains a matrix product by serially integrating the posture matrices in the local coordinate system of the device provided at each joint in the order of joint connections from the origin of the world coordinate system to the abnormal location. . The attitude control unit 102 calculates the coordinates of the device detected as the abnormal location in the world coordinate system by performing coordinate transformation based on the matrix product obtained by the integration. A method for calculating the coordinates of such a specific position is described, for example, in "Ohmsha, 'Humanoid Robot'," edited by Shuji Kajita.

The attitude control unit 102 also manages, for each device, information on a coordinate point sequence (point sequence A) surrounding the area where the device exists.

FIG. 6 is a diagram showing an example of a coordinate point sequence.

In the example of FIG. 6, a coordinate point sequence surrounding a device provided at the elbow of the arm is shown. Each small circle surrounding the cylindrical device represents a coordinate point. Further, the coordinates of the coordinate point at the lower left end are shown as coordinate 1, and the coordinates of the coordinate point next to the right are shown as coordinate 2. Coordinates 1 and 2 are, for example, coordinates in the local coordinate system.

The attitude control unit 102 manages coordinate information of each of a plurality of coordinate points that constitute a coordinate point sequence in association with a device.

The attitude control unit 102 photographs the abnormal area based on the position of the abnormal area calculated as described above, the position of the camera 41, the attitude of the camera 41, and camera parameters such as the angle of view. Identify the coordinates of the area where the abnormality appears on the image. The attitude control unit 102 also has information such as camera parameters.

The coordinates of an arbitrary point in space on an image taken by a camera can be determined by, for example, projective transformation using a pinhole camera model commonly used in the field of computer vision. It is possible to specify by

The posture control unit 102 controls the posture of each part of the robot 1 based on information such as the position of the abnormality and the coordinates of the area in which the abnormality appears, so as to satisfy the condition that the abnormality is visible in the image. . A control command value is supplied from the attitude control section 102 to each drive unit of the drive section 33, and the drive of each drive unit is controlled based on the control command value.

Note that there are usually a plurality of postures that satisfy the above conditions. One posture is determined from among the plurality of postures, and each part is controlled so as to take the determined posture.

For example, the following criteria are used as criteria for determining one posture.

Standard example 1
a: Determine the posture under the constraint that the abnormal part does not move.
b: Determine the posture under the constraint that the amount of change in the joint angle at the abnormal location is minimized.

Standard example 2
Determine the posture that minimizes the amount of change and amount of current consumption when changing the joint angle.

Standard example 3
A posture that satisfies both the criteria of Reference Example 1 and Reference Example 2 is determined.

Standard example 4
There are cases where it is okay to move the abnormal area. For example, in a case where the user is notified of the period up to the occurrence of an abnormality, it is possible to drive the abnormal location at the current time. In this case, reference example 1 is excluded and only reference example 2 is applied.

For example, it is possible to estimate the period until an abnormality occurs by comparing the drive time of the device represented by the behavior log with the service life according to the device specifications. The attitude control unit 102 has a function of estimating the period until an abnormality occurs based on an action log and specifications.

It is possible to control the posture according to these various standards.

- Shooting of the abnormal location and recording of driving sound When the attitude control unit 102 performs attitude control and the abnormal location is included in the angle of view of the camera 41, the imaging/recording control unit 103 controls the camera 41 to record the abnormal location. Take pictures of. The image obtained by photography is recorded, for example, in a memory within the control unit 31.

In addition to still images, moving images may also be captured. The moving image is captured by capturing the state of the abnormal location during operation.

The sound generated from the abnormal location may be recorded along with the moving image. The photographing/recording control unit 103 controls the microphone 42, collects the sound generated when the posture control unit 102 drives the abnormal location, and records the sound as a drive sound. This makes it possible to present the sound generated from the abnormal location to the user together with the moving image.

The photographing/recording control unit 103 outputs the image obtained by photographing to the notification information generating unit 104 together with information on a coordinate point sequence (point sequence A) surrounding the device in which the abnormality has occurred.

- Highlight display of abnormal location The notification information generation unit 104 performs image processing on the image photographed by the camera 41 to highlight (emphasize) the abnormal location.

For example, the notification information generation unit 104 sets a point sequence B by converting a point sequence A whose coordinates are represented by the information supplied from the photographing/recording control unit 103 into coordinates on a photographed image. The point sequence B is the coordinate points on the image surrounding the abnormal location.

The notification information generation unit 104 performs image processing to highlight the area surrounded by the dot sequence B in the photographed image. For example, highlight display is performed by superimposing an image of a conspicuous color, such as red, with a predetermined transparency set on the area surrounded by the dot sequence B.

Processing different from the processing of superimposing images of a predetermined color, such as effect processing and icon composition, may be performed. A specific example of highlight display will be described later.

The notification information generation unit 104 outputs an image obtained by performing image processing for highlighting to the notification control unit 105 as an abnormality notification image that is an image for notifying an abnormal location.

- Notification to User The notification control unit 105 controls the wireless communication unit 34 and transmits the abnormality notification image supplied from the notification information generation unit 104 to the mobile terminal 2. The mobile terminal 2 receives the abnormality notification image transmitted from the robot 1 and displays it on the display.

If the abnormality notification image is a moving image and the driving sound is being recorded, the data of the driving sound is also transmitted from the notification control unit 105 to the mobile terminal 2 as appropriate. In the mobile terminal 2, driving sound is output from the speaker in accordance with the display of the moving image.

FIG. 7 is a diagram showing an example of an abnormality notification image.

The abnormality notification image P in FIG. 7 shows the joint of the left arm of the robot 1. The devices forming the joints of the left arm are highlighted by superimposing an image 151 of a predetermined color. The image 151 is superimposed on an area surrounded by a dot sequence (dot sequence B) indicated by a small circle.

In FIG. 7, a diagonal line inside a narrow rectangular area indicates that an image 151 with a predetermined transparency is superimposed on that area. From such a display, the user can easily confirm that an abnormality has occurred in the joint of the left arm of the robot 1.

<Robot movement>
Here, with reference to the flowchart of FIG. 8, a series of processes performed by the robot 1 to notify the user that an abnormality has occurred will be described.

In step S1, the abnormality detection unit 101 detects that there is an abnormal device based on information supplied from each device.

In step S2, the abnormality detection unit 101 identifies an abnormal location based on a predetermined detection method. Information indicating the abnormal location is output to the attitude control unit 102.

In step S3, the attitude control unit 102 calculates a point sequence A surrounding the area where the abnormality detected by the abnormality detection unit 101 exists.

In step S4, the attitude control unit 102 performs attitude control processing. By the attitude control process, the attitude of the robot 1 is controlled so that the abnormal location falls within the angle of view of the camera 41. Details of the attitude control process will be described later with reference to the flowchart in FIG.

In step S5, the attitude control unit 102 determines whether the abnormal location falls within the angle of view of the camera 41. If it is determined that the abnormal location falls within the angle of view of the camera 41, the process proceeds to step S6.

In step S6, the photographing/recording control unit 103 controls the camera 41 to photograph the abnormal location. The image obtained by photographing is output to the notification information generation unit 104 along with information on the point sequence A surrounding the area of the device where the abnormality has occurred.

In step S7, the notification information generation unit 104 converts the point sequence A of the abnormal location supplied from the photographing/recording control unit 103 into a point sequence B in the image coordinate system.

In step S8, the notification information generation unit 104 performs image processing on the photographed image so that the area surrounded by the dot sequence B is displayed as a highlight. The abnormality notification image generated by image processing is output to the notification control unit 105.

In step S9, the notification control unit 105 transmits the abnormality notification image to the mobile terminal 2, and ends the process.

On the other hand, if it is determined in step S5 that the abnormal location does not fall within the field of view of the camera 41 even if the posture is controlled, an alternative process is performed in step S10.

The alternative process is a process of notifying the user of the occurrence of an abnormality using a method different from the above-described method using the abnormality notification image when the abnormal location cannot be photographed. The alternative processing will be described later. After the alternative process notifies that an abnormality has occurred, the process ends.

Next, the attitude control process performed in step S4 of FIG. 8 will be described with reference to the flowchart of FIG.

In step S31, the attitude control unit 102 calculates the three-dimensional coordinates of the abnormal location in the world coordinate system in the initial attitude.

In step S32, the attitude control unit 102 calculates the three-dimensional coordinates of the abnormal location in the world coordinate system in the current attitude.

In step S33, the attitude control unit 102 calculates the coordinates of the abnormal location in the image coordinate system based on information on the three-dimensional coordinates of each point on the device that is the abnormal location. As a result, the area on the image in which the abnormal location appears is specified.

In step S34, the attitude control unit 102 determines whether the abnormal location appears near the center of the image. For example, a predetermined range is set in advance based on the center of the image. If the abnormal location is captured inside the predetermined range, it is determined that the abnormal location is captured near the center, and if the abnormal location is not captured inside the predetermined range, it is determined that the abnormal location is not captured near the center.

If it is determined in step S34 that the abnormal part is not captured near the center of the image, in step S35, the posture control unit 102 sets the amount of correction for each joint angle based on the difference between the position of the abnormal part and the center of the image. . Here, the amount of correction for each joint angle is set so that the abnormal location appears closer to the center of the image.

In step S36, the posture control unit 102 controls the drive unit 33 based on the amount of correction to drive each joint.

In step S37, the posture control unit 102 determines whether the joint angle correction has been repeated a predetermined number of times.

If it is determined in step S37 that the joint angle correction has not been repeated the predetermined number of times, the process returns to step S32, and the joint angle correction is repeated in the same manner.

On the other hand, if it is determined in step S37 that the joint angle correction has been repeated a predetermined number of times, the process returns to step S4 in FIG. 8 and the subsequent processes are performed.

Similarly, if it is determined in step S34 that the abnormal location appears near the center of the image, the process returns to step S4 in FIG. 8 and subsequent processing is performed.

Through the above processing, the user can easily confirm not only that an abnormality has occurred in the robot 1, but also the location of the abnormality.

The robot 1 can also notify the user that an abnormality has occurred in its own device.

Furthermore, by using a moving image as an abnormality notification image, the robot 1 can reproduce the problem situation and present it to the user. By presenting the driving sound together with the moving image, the robot 1 can not only visually present the abnormal location but also auditorily present the abnormal location. This allows the user to learn more about the abnormality situation.

When presenting a moving image as an abnormality notification image, a moving image representing movement during normal operation reproduced by CG (Computer Graphics) may be superimposed on a portion of an abnormality. This makes it possible to tell the user in more detail what kind of condition the symptom is that is determined to be an abnormal condition.

<Example of abnormality notification image>
FIG. 10 is a diagram showing another example of the abnormality notification image.

As shown in A to C of FIG. 10, an icon may be displayed on the abnormality notification image. The abnormality notification images shown in A to C of FIG. 10 show the joints of the left arm as in FIG. 7 . A colored oval image is superimposed on the joint of the left arm to highlight that part.

Icon I1 shown in A of FIG. 10 is a countdown timer icon representing a period until the occurrence of an abnormality. For example, if the period until the occurrence of an abnormality is shorter than a predetermined period, an abnormality notification image in which the icon I1 is combined is presented to the user.

Other images such as a calendar or a clock representing the period until the occurrence of the abnormality may be displayed as an icon.

An icon depending on the type of abnormality may be displayed on the abnormality notification image.

For example, if the type of abnormality is overcurrent, the icon I2 of B in FIG. 10 representing this is displayed. Further, if the type of abnormality is motor overheating, an icon I3 shown in C in FIG. 10 representing this is displayed.

When presenting the abnormality notification image including the icon I2, for example, an image taken using a thermography camera and representing the actual heat generation situation at the abnormal location may be superimposed. As a result, if the abnormal location is accompanied by heat generation, it is possible to notify the user of the details of the heat generation situation.

Such an icon may be displayed on a moving image. In this case, an icon will be synthesized for each frame of the moving image.

Note that when presenting a moving image as an abnormality notification image, the moving image is captured for a period of time including a predetermined time before and after the timing at which a symptom determined to be an abnormal state occurs. This means that the captured video images show the state of the abnormal location from just before the symptom that is determined to be an abnormal state occurs to after the symptom occurs.

In this case, the above-described highlight display and icon display are performed, for example, only while a symptom that is determined to be an abnormal state is occurring. This makes it possible to clearly convey to the user the state at the moment the symptom occurs.

<Example of alternative processing>
Since the joints of the robot 1 have a range of motion, depending on the abnormal location, the camera 41 may not be able to photograph the abnormal location even if the posture is controlled.

If the abnormal location cannot be photographed by the camera 41, the following alternative processing (step S10 in FIG. 8) is performed.

(i) Example of having another robot take a picture FIG. 11 is a diagram showing an example of an alternative process of having another robot take a picture.

In the example of FIG. 11, it is assumed that an abnormality has occurred in the device provided on the waist of the robot 1-1. The robot 1-1 cannot photograph the abnormal location with its own camera 41.

In this case, the robot 1-1 transmits information on the three-dimensional coordinates of the abnormal location to the robot 1-2 and requests photography of the abnormal location.

In the example of FIG. 11, the robot 1-2 is the same type of robot as the robot 1-1, and has the same configuration as the robot 1 described above. A camera 41 is provided on the head of the robot 1-2. Communication is possible between the robot 1-1 and the robot 1-2.

The robot 1-2 uses the three-dimensional coordinates of the abnormal location represented by the information sent from the robot 1-1, the relative positional relationship between itself and the robot 1-1, etc. to determine the location in its own coordinate system. , calculate the three-dimensional coordinates of the abnormal location.

Based on the calculated three-dimensional coordinates, the robot 1-2 controls its own posture so that the abnormal location falls within the field of view of its own camera 41, and photographs the abnormal location of the robot 1-1.

The image obtained by photographing by the robot 1-2 may be transmitted to the mobile terminal 2 via the robot 1-1, or directly from the robot 1-2 to the mobile terminal 2. Good too.

Thereby, even if an abnormality occurs in a device located at a position where the robot 1-1 cannot take a picture with its own camera 41, the robot 1-1 can notify the user that an abnormality has occurred.

(ii) Example of showing directly to the user FIG. 12 is a diagram illustrating an example of an alternative process of showing directly to the user.

In the example of FIG. 12, it is assumed that an abnormality has occurred in the device provided on the waist of the robot 1.

The robot 1 recognizes the user's position based on the image taken by the camera 41 and moves to a position near the user. The robot 1 is provided with a function of recognizing a user based on a face appearing in a photographed image.

After moving to a position near the user, the robot 1 controls its own posture so as to point the abnormal location toward the user, and presents it to the user.

A voice such as "Take a picture of this place with your smartphone" may be output from the speaker 43 to ask the user to take a picture of the abnormal area. The image taken by the user is transmitted from the mobile terminal 2 to the robot 1.

In this way, it is also possible to directly notify the user that an abnormality has occurred through the operation of the robot 1.

(iii) Example using a detachable camera FIG. 13 is a diagram showing an example of alternative processing using a detachable camera.

In the example of FIG. 13, it is assumed that an abnormality has occurred in a device provided on the back side (occipital area) of the head of the robot 1. The robot 1 cannot photograph the abnormal location with its own camera 41. The robot 1 has a detachable (removable) camera at a predetermined position of the housing.

The robot 1 removes and grasps the detachable camera 161, controls its own posture so that the abnormal location is within the field of view of the camera 161, and photographs the abnormal location. The image taken by the camera 161 is transferred to the robot 1 and transmitted to the mobile terminal 2.

(iv) Example of photographing a figure reflected in a mirror FIG. 14 is a diagram showing an example of an alternative process of photographing a figure reflected in a mirror.

In the example of FIG. 14, it is assumed that an abnormality has occurred in the device provided at the base (neck) of the head of the robot 1. The robot 1 cannot photograph the abnormal location with its own camera 41.

In this case, the robot 1 moves in front of the mirror M based on information stored in advance. Information indicating the position of the reflective surface of the mirror M is set in the robot 1. The position of the reflective surface of the mirror M may be specified by analyzing the image taken by the camera 41.

After moving in front of the reflective surface of the mirror M, the robot 1 controls its own posture so as to point the abnormal location toward the mirror M, and photographs the abnormal location.

As described above, when an abnormality occurs in a device located at a position that cannot be photographed by the camera 41, it is possible to notify the abnormal location by various methods as an alternative process.

<Modified example>
- Example of control system A part of the function for notifying the user that an abnormality has occurred may be provided in an external device such as the mobile terminal 2 or a server on the Internet.

FIG. 15 is a diagram showing a configuration example of a control system.

The control system in FIG. 15 is configured by connecting the robot 1 and the control server 201 via a network 202 such as the Internet. The robot 1 and the control server 201 communicate via a network 202.

In the control system of FIG. 15, an abnormality occurring in the robot 1 is detected by the control server 201 based on information transmitted from the robot 1. Information representing the status of each device of the robot 1 is sequentially transmitted from the robot 1 to the control server 201.

When detecting that an abnormality has occurred in the robot 1, the control server 201 controls the posture of the robot 1 and causes the abnormal location to be photographed. The control server 201 acquires images taken by the robot 1, performs image processing for displaying highlights, etc., and transmits the images to the mobile terminal 2.

In this way, the control server 201 functions as a control device that controls the robot 1 and notifies the user of abnormalities that occur in the robot 1. In the control server 201, each functional unit shown in FIG. 4 is realized by executing a predetermined program.

- Example of computer configuration The series of processes described above can be executed by hardware or software. When a series of processes is executed by software, a program constituting the software is installed from a program recording medium into a computer built into dedicated hardware or a general-purpose personal computer.

FIG. 16 is a block diagram showing an example of a hardware configuration of a computer that executes the above-described series of processes using a program. The control server 201 in FIG. 15 also has a configuration similar to that shown in FIG. 16.

A CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

An input/output interface 1005 is further connected to the bus 1004. Connected to the input/output interface 1005 are an input section 1006 consisting of a keyboard, a mouse, etc., and an output section 1007 consisting of a display, speakers, etc. Further, connected to the input/output interface 1005 are a storage unit 1008 consisting of a hard disk or non-volatile memory, a communication unit 1009 consisting of a network interface, etc., and a drive 1010 for driving a removable medium 1011.

In the computer configured as above, the CPU 1001 executes the series of processes described above by, for example, loading a program stored in the storage unit 1008 into the RAM 1003 via the input/output interface 1005 and the bus 1004 and executing it. will be held.

A program executed by the CPU 1001 is installed in the storage unit 1008 by being recorded on a removable medium 1011 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

Note that the program executed by the computer may be a program in which processing is performed chronologically in accordance with the order described in this specification, in parallel, or at necessary timing such as when a call is made. It may also be a program that performs processing.

In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether or not all the components are in the same casing. Therefore, multiple devices housed in separate casings and connected via a network, and a single device with multiple modules housed in one casing are both systems. .

The effects described in this specification are merely examples and are not limiting, and other effects may also exist.

The embodiments of the present technology are not limited to the embodiments described above, and various changes can be made without departing from the gist of the present technology.

For example, the present technology can take a cloud computing configuration in which one function is shared and jointly processed by a plurality of devices via a network.

Moreover, each step explained in the above-mentioned flowchart can be executed by one device or can be shared and executed by a plurality of devices.

Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

- Examples of combinations of configurations The present technology can also have the following configurations.

(1)
an abnormality detection unit that detects an abnormality occurring in a predetermined part of the robot;
A control device comprising: a posture control unit that controls the posture of the robot so that the predetermined region where the abnormality occurs is included in the angle of view of the camera.
(2)
The control device according to (1) above, wherein the camera is provided at a predetermined position of the robot.
(3)
a recording control unit that controls photography by the camera;
The control device according to (2), further comprising: a notification control unit that transmits an image taken by the camera to an external device and notifies the occurrence of an abnormality.
(4)
further comprising an information generation unit that performs image processing on the image to highlight an area in which the predetermined body part is captured;
The control device according to (3), wherein the notification control unit transmits the image subjected to the image processing.
(5)
The control device according to (4), wherein the information generation unit performs the image processing according to the type of abnormality occurring in the predetermined region.
(6)
The control device according to (4), wherein the information generation unit combines an icon corresponding to the type of abnormality that has occurred in the predetermined region with the image.
(7)
The control device according to (4), wherein the recording control unit causes a still image or a moving image showing the predetermined region to be photographed.
(8)
The recording control unit according to (7) above, when an abnormality occurs when performing a specific action in the predetermined region, causes the video image to be captured at a time including a predetermined time before and after the timing at which the abnormality occurs. Control device.
(9)
The control device according to (8), wherein the information generation unit combines an image representing the normal specific motion with the moving image.
(10)
The control device according to (8) or (9), wherein the recording control unit records a sound when performing the specific operation.
(11)
The control device according to any one of (2) to (10), wherein the attitude control unit controls the position of the camera when the predetermined portion is not included in the angle of view of the camera after controlling the attitude.
(12)
The control device according to (11), wherein the camera is a device that is removable from the predetermined position of the robot.
(13)
Any one of (3) to (10) above, wherein the recording control unit causes another robot to photograph the predetermined region if the predetermined region is not included in the angle of view of the camera after controlling the posture. The control device described in .
(14)
The control device according to any one of (3) to (10), wherein the notification control unit notifies, by an operation of the robot, that an abnormality has occurred in the predetermined region.
(15)
The control device is
Detects abnormalities that occur in predetermined parts of the robot,
A control method comprising: controlling the posture of the robot so that the predetermined part where the abnormality has occurred is included in the angle of view of the camera.
(16)
to the computer,
Detects abnormalities that occur in predetermined parts of the robot,
A program for controlling the posture of the robot so that the predetermined part where the abnormality has occurred is included in the angle of view of the camera.

1 robot, 2 mobile terminal, 11 network, 31 control unit, 33 drive unit, 41 camera, 101 abnormality detection unit, 102 attitude control unit, 103 photographing/recording control unit, 104 notification information generation unit, 105 notification control unit

Claims (15)

an abnormality detection unit that detects an abnormality occurring in a predetermined part of the robot;
a posture control section that controls the posture of the robot so that the abnormal region detected by the abnormality detection section is included in the angle of view of the camera;
A control device comprising: a recording control unit that controls the camera and captures a moving image of the abnormal region while it is being driven. The control device according to claim 1, wherein the camera is provided at a predetermined position of the robot. The control device according to claim 1 or 2, further comprising a notification control unit that transmits the moving image to an external device and notifies the occurrence of an abnormality. further comprising an information generation unit that performs image processing on the moving image to highlight a region in which the abnormal region is captured;
The control device according to claim 3, wherein the notification control unit transmits the moving image subjected to the image processing. The control device according to claim 4, wherein the information generation unit performs the image processing according to the type of abnormality occurring in the abnormal region. The control device according to claim 4, wherein the information generation unit combines an icon corresponding to the type of abnormality occurring in the abnormal region with the moving image. The control device according to claim 4, wherein, when an abnormality occurs when performing a specific operation, the recording control unit causes the moving image to be captured for a time including a predetermined time before and after the timing at which the abnormality occurs. The control device according to claim 7, wherein the information generation unit combines an image representing the normal specific motion with the moving image. The control device according to claim 7, wherein the recording control unit records a sound when performing the specific operation. The control device according to claim 2, wherein the attitude control unit controls the position of the camera if the abnormal region is not included in the angle of view of the camera after controlling the attitude. The control device according to claim 10, wherein the camera is a device that is removable from the predetermined position of the robot. The control device according to claim 3, wherein the recording control unit causes another robot to photograph the abnormal region if the predetermined region is not included in the angle of view of the camera after controlling the posture. The control device according to claim 3, wherein the notification control unit notifies the occurrence of an abnormality by an operation of the robot. The control device is
Detects abnormalities that occur in predetermined parts of the robot,
controlling the posture of the robot so that the detected abnormal region is included in the angle of view of the camera;
A control method comprising: controlling the camera to take a moving image of the abnormal region while it is being driven. to the computer,
Detects abnormalities that occur in predetermined parts of the robot,
controlling the posture of the robot so that the detected abnormal region is included in the angle of view of the camera;
A program for controlling the camera and executing a process of photographing a moving image of the abnormal region while it is being driven.

Images