JP4028192B2 - Operation input method, apparatus, program, and recording medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
  The present invention is suitable for use in inputting and reproducing data for controlling the operation of an object such as a robot.Operation input method, apparatus, program, and recording mediumAbout.
[0002]
[Prior art]
Conventionally, when constructing a teaching device that teaches robot operation, it is necessary to design and control a GUI (graphical user interface) and operation commands that match the interface, operation commands, and driver. there were. In this case, the teaching device is composed mainly of a computer and its peripheral devices and a program executed by the computer, and data input in real time according to the operation of the operator, A control signal for causing a desired operation to be performed on an object such as a robot is output based on data recorded in advance.
[0003]
Next, an example of the conventional technique as described above will be specifically described. In this example, when generating control data (driving data) for causing an object such as a robot to perform a desired motion in accordance with an operation by an operator, the object is first modeled as a CG (computer graphics). Furthermore, the motion of the object is obtained based on the driving range of the object and its conditions, and the motion of the object can be simulated and displayed as a CG model in the computer. Then, after modeling, move the object model while converting the coordinates according to the operation amount of the pointing device, etc., record the change of the coordinate data during operation, and perform predetermined arithmetic processing on it, The control data required for operation is obtained. That is, control data (driving data) of the object is generated according to the amount of operation of the pointing device performed on the CG modeled object. In such a case, the operation of the pointing device is often complicated, and the operator cannot easily operate.
[0004]
Further, when the robot operation is reproduced in synchronism with sound or image, complicated processing such as timer counting is required.
[0005]
[Problems to be solved by the invention]
  In view of the above problems, the present invention can simplify the operation when inputting or reproducing data related to the motion of an object such as a robot.Operation input method, apparatus, program, and recording mediumThe purpose is to provide. The present invention also makes it possible to easily input, record, and play back robot movements with intuitive operations.Operation input method, apparatus, program, and recording mediumThe purpose is to provide.
[0006]
  Furthermore, the present invention makes it possible to reproduce the operation and the sound and the image while synchronizing them with a simple operation.Operation input method, apparatus, program, and recording mediumThe purpose is to provide. In other words, it is suitable for inputting data for synchronous playback of content such as sound and images and robot motion.Operation input method, apparatus, program, and recording mediumThe purpose is to provide. Specifically, for example, in the movement of an object having a meaning in the whole movement such as a gesture, there may be a case where a movement with good followability to the input is required at the expense of strict speed / position control. . In other words, it is desirable to ensure CG reproduction and robot motion response at the expense of frequency and accuracy. Therefore, the present invention can perform input with good follow-up and synchronized playback with a simple operation.Operation input method, apparatus, program, and recording mediumThe purpose is to provide.
[0007]
[Means for Solving the Problems]
  In order to solve the above problems, the first aspect of the present invention is:A method for teaching a real object performing an operation corresponding to the control data by inputting control data using a predetermined operator, and sequentially transmitting the control data. The real objectShooting the movement ofWhen a change in motion of a real object exceeds a predetermined magnitude in a captured image, a simulated image for the image frame is created, a moving area of the created simulated images is defined, and a representative point of the moving area is moved The guidance information is created by converting the obtained locus into display information obtained by expanding the obtained locus, the coordinate range in the operation target space of the guidance information is associated with each simulated image, and each simulated image is associated with each simulated image. On the other hand, mapping control data for operating the real object,Simulated imageAny one ofTheSaidDisplay with guidance information,When the coordinates specified by the predetermined operator are within the coordinate range in the operation target space of the guidance information, a simulated image corresponding to the coordinate range is displayed and control data mapped to the simulated image is displayed. And the acquired control data is output to the moving object, or the acquired control data is recorded as time-series information.
[0008]
  In the invention described above, the present invention is characterized in that the simulated image is obtained by performing CG processing on a captured image.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
  DETAILED DESCRIPTION OF THE INVENTIONOperation input method, apparatus, program, and recording mediumThe embodiment will be described. FIG. 1 is a diagram showing the overall configuration of a robot system. A personal computer 1 and a robot (for example, a robot in the form of an animal such as a mouse) 2 are connected by an audio line 3. The personal computer 1 includes a main body device 10 and a display device 11. In addition to the robot 2, a pointing device (operator) 4 such as a mouse and an image input device 5 are connected.
[0011]
The robot 2 can swing the head 20 in the left-right direction by a built-in servo motor, and the right front leg 21 and the left front leg 22 can be moved up and down.
The robot 2 can emit sound from the built-in speaker into the air. In this robot system, a signal (frequency-multiplexed signal) obtained by synthesizing a synthesized voice signal and drive data is transmitted from the personal computer 1 to the robot 2 via the audio line 3. Among these, the synthesized voice signal is an audible frequency band signal that forms voice information indicating, for example, words, words, or singing voice, and is output from a speaker built in the robot 2. The other drive data is a signal having a frequency equal to or higher than the audible frequency band (for example, a signal exceeding 18 kHz), and is used as a control command signal for the servo motor. That is, generation of sound and control of the drive part can be performed by information simultaneously sent by the audio line 3.
[0012]
FIG. 2 is a system block diagram for explaining the connections and signal flows of the components shown in FIG. As shown in FIG. 2, the personal computer 1 shown in FIG. 1 includes an input / recording / playback / display control device 100, which is a main characteristic part of the present invention, in the main body device 10. The input / recording / playback / display control device 100 is configured as an application program that operates on a predetermined operating system using a storage device (memory or hard disk) (not shown) or an input / output interface circuit.
The input / recording / playback / display control apparatus 100 inputs a signal indicating the operation state sent from the pointing device 4 and captures a moving image (video) or a still image captured by the image input apparatus 5. Record (capture). Here, in the present embodiment, the image input device 5 is used for photographing the operation of the robot 2. The input / recording / playback / display control device 100 also outputs a display control signal to the display device 11 and outputs a synthesized voice signal and drive data to the robot 2 via the audio line 3.
[0013]
Next, functional blocks constituting the input / recording / playback / display control apparatus 100 shown in FIG. 2 will be described with reference to FIG. However, here, in order to facilitate understanding, first, the drive data reproduction unit 110 in each block will be described, and then the simulated image data / control data storage unit 104 will be described. Other functional blocks will be described.
[0014]
First, with reference to FIG. 3 and FIG. 5 to FIG. 7, functional details of the drive data reproducing unit 110 and the robot 2 will be described. Here, FIG. 5 is a block diagram showing an internal configuration of the robot 2. FIG. 6 is a diagram showing an example of the correspondence between the control data file for generating each drive data for the robot 2 and the control contents thereby. FIG. 7 is an explanatory diagram showing the relationship between the robot 2 and each drive data waveform.
[0015]
The drive data reproducing unit 110 in FIG. 3 is composed of a software player and a sound device for reproducing sound files in the existing wave format, midi format, au format, etc., and stored data The control data file called by the reproduction unit 107 is received, and the drive data is reproduced and output based on the control data file. The reproduced drive data passes through the audio line 3 and is sent to the audio input unit 201 (see FIG. 5) of the robot 2 connected via the main line 3. The input drive data is sent to the control signal generation unit 202 in FIG. Here, the control signal generation unit 202 detects the level of each predetermined frequency component included in the drive data. When a signal matching a predetermined frequency component is detected, a control signal for driving a motor provided in the robot drive unit 203 is generated based on the detected signal.
[0016]
6 shows the relationship between the name of the control data file that is called by the stored data reproducing unit 107 in FIG. 3 and sent to the drive data reproducing unit 110, and the contents of the control performed by the robot driving unit 203 in FIG. ing. For example, when the control data file name Sound1 is sent to the drive data reproduction unit 110, the robot drive unit 203 swings −30 degrees (in this example, the head swings 30 in the right direction (leftward in the figure)). Degree) is controlled. Also, depending on the control data file names Sound 2, 3, 4,..., The control of swinging in the same direction of 20 degrees, 10 degrees, 5 degrees,. In this case, the drive data reproduced by the control data files Sound1, 2, 3, 4,... Is a signal having a different frequency or a composite signal of a plurality of different frequency components.
[0017]
FIG. 7 shows the correspondence between the waveform of the drive data reproduced by the drive data reproducing unit 110 and the operation of the robot 2. In this example, when the drive data of the waveforms Wa, Wb, Wc is generated. The head 20 of the robot 2 is controlled to swing in the right direction (robot 2a), the vertical direction (robot 2b), and the left direction (robot 2c). In this case, the drive data of the waveforms Wa, Wb, and Wc are signals having different frequencies (or signals having different frequency components), and the waveform Wa is data of the control data file name Sound1 described with reference to FIG. It corresponds to the playback waveform.
[0018]
Next, the simulated image data / control data storage unit 104 shown in FIG. 3 will be described in detail with reference to FIGS. FIG. 8 is a block diagram showing the configuration of the simulated image data / control data storage unit 104, and FIG. 9 is a flowchart for explaining the operation. FIG. 10 is a diagram for explaining an automatic simulation image generation function by the image / control data mapping unit 1042 of FIG. FIGS. 11 to 16 are diagrams showing each data and each image used when a specific example of the operation of the simulated image data / control data storage unit 104 is described.
[0019]
As illustrated in FIG. 8, the pseudo image data / control data storage unit 104 includes an image information input unit 1040 that captures image data input from the image input device 5, and a plurality of images captured by the image information input unit 1040. The image information calculation unit 1041 inputs image data and performs image information processing calculation, and the image / control data mapping unit 1042 associates image data with a control data file based on the calculation result. .
[0020]
With the above configuration, the pseudo image data / control data storage unit 104 converts the substantial image including the plurality of frames representing the operation of the robot 2 input from the image input device 5 by the image information input unit 1040 into image data (moving image data). And is stored in a predetermined storage means (step SA1 in FIG. 9).
Next, the image information calculation unit 1041 creates data of the simulated robot image 124 (124a to 124c) as shown in FIG. 11 (step SA2). Simulated images 124a,..., 124b,..., 124c shown in FIG. 11 are taken when the robot 2 is driven and controlled based on the drive data of the waveforms Wa,..., Wb,. , 2b,..., 2c in FIG. 7 is obtained by simplifying the image by CG processing.
[0021]
Imaging of the operation of the robot 2 is performed by causing the image input device 5 to start imaging in accordance with an instruction from the simulated image data / control data storage unit 104 in FIG. This is started by sequentially calling the data files Sound 1, 2, 3, 4,... At a predetermined time interval. The control data file called by the stored data calling unit 103 is sent to the stored data reproducing unit 107 for reproduction processing, and an analog driving data signal such as waveforms Wa, Wb, Wc in FIG. Is converted to output. Here, from the robot 2a having the head 20 most inclined to the right in FIG. 7 to the robot 2c having the head 20 most inclined to the left via the neutral robot 2b, For example, the image input device 5 outputs image data of about several tens to hundreds of frames.
[0022]
Thus, the image of each frame input by the image information input unit 1040 is temporarily stored in a predetermined storage unit. The image information calculation unit 1041 creates robot simulation images 124a to 124c as shown in FIG. 11 based on the stored image data. At this time, the image information calculation unit 1041 does not create the number of simulated robot images 124 corresponding to each image frame but corresponds to all the frames, and in addition to the start point and the end point, the change in operation has a predetermined magnitude. A new robot simulation image 124 is created every time the number of times is exceeded. This is a process for preventing the creation and storage of a plurality of similar images with almost no change in motion, and preventing an increase in data capacity and an increase in processing time. With reference to FIG. 10, the basic concept of the generation process of the simulated robot image in this embodiment will be described.
[0023]
FIG. 10 shows an image at which time point is retained as a simulated robot image when the actual motion of the robot 2 occurs as shown in this figure from time t to time t + 3k (k is a constant (ms)). It is a figure for demonstrating what is done. Here, the actual actions at times t, t + k, t + 2k, and t + 3k are as follows: the head 20 is in the vertical direction and the right and left front legs 21 and 22 are facing downward; The right and left front legs 21 and 22 are directed downward, the head 20 is further inclined to the right and the right and left front legs 21 and 22 are directed downward, and the head 20 In the same manner, the right front leg portion 21 is directed downward and the left front leg portion 22 is directed upward.
[0024]
In this example, a moving region is extracted for each frame by the background difference method, a difference between frames is obtained for the extracted region, and a determination using a predetermined evaluation function is performed with respect to a temporal change in the size. The process of extracting the simulated robot image is performed. As the evaluation function, a variable ΔD = D (t + mk) −D (t + nk) representing an image feature amount (a prime number of the moving region) is used. When ΔD> L, D (t + mk) is held as a robot simulation image. Here, ΔD is the number of pixels in the moving region, D (t + mk) is the robot image data at time (t + mk), and D (t + nk) is the robot image data held as a robot simulated image in the vicinity. Further, m = 0, 1, 2, 3,..., N = 0, 1, 2, 3,... (K> n), k is a constant (msec), and L is a moving region. It is a constant showing the threshold value in determination about the number of pixels.
[0025]
In the example shown in FIG. 10, when the image feature amount ΔD is obtained based on the robot image data D (t) at time t after time t, the image feature amount ΔD exceeds the threshold value L at time t + 2k. Therefore, the robot image data D (t + 2k) at that time is held as a robot simulation image. Then, when the image feature amount ΔD is obtained with reference to the robot image data D (t + 2k) at time t + 2k, the image feature amount ΔD again exceeds the threshold value L at time t + 3k. (T + 3k) is held as the next robot simulation image.
As described above, the image information calculation unit 1041 in FIG. 8 creates data of a plurality of robot simulation images and stores the relationship with the drive data at that time.
[0026]
Next, when the robot simulation image 124 is created as described above in step SA2 of FIG. 9 (in this case, assuming that the robot simulation images 124a to 124c of FIG. 11 are created as the robot simulation image 124), FIG. The image information calculation unit 1041 in FIG. 8 performs processing for assigning common guide information 125 to the plurality of simulated robot images 124a to 124c for a series of operations as shown in FIG. 12 (step SA3 in FIG. 9).
The guide information 125 is display information (image information) corresponding to a change in an actual driving part, that is, a movement locus. The guide information 125 is used as guide information for easily indicating the operation position, operation direction, operation amount, and the like of the pointing device 4 when inputting the operation of the robot 2. In the present embodiment, a combination of the simulated robot image 124 and the guide information 125 is referred to as a simulated image 123.
[0027]
The guide information 125 is determined, for example, by setting a region of interest for a moving region extracted from the robot simulated image 124 by the background subtraction method and obtaining a locus of the center of gravity of the region of interest. In the example of FIG. 12, first, the head 20 is set as a region of interest, and the locus of change in the center of gravity is obtained. Then, when the guide information 125 is added to the simulated image 123, the trajectory is determined in consideration of the ease of viewing the guide information 125 and the moving area (head 20) and the operability with the pointing device 4 described later. The guide information 125 is created by moving the simulated image 123 downward in the center direction and converting it to display information expanded in the horizontal and vertical directions.
[0028]
When the process for giving the guide information 125 (step SA3 in FIG. 9) is finished, the image / control data mapping unit 1042 in FIG. 8 performs a mapping process of the simulated image to the guide information (in FIG. 9). Step SA4). The simulated image mapping process for the guide information is a process for associating each simulated image 123 with a coordinate point (coordinate range) in the operation target space corresponding to the guide information 125.
Here, the process of step SA4 will be described with reference to FIGS. 13 is a diagram illustrating simulated images 123a to 123c obtained by adding the guide information 125 to the robot simulated images 124a to 124c of FIG. 11, and FIG. 14 is a diagram illustrating a table obtained as a result of the mapping process. is there.
[0029]
For example, in the simulated image 123a shown in FIG. 13, the head 20 is inclined to the rightmost side (left side in the figure). Therefore, for the simulated image 123a, an indicator 127a is set at a position located on the left side in the guide information, and a rectangular coordinate range 128a (range indicated by a broken line) when the indicator 127a is selected by the pointing device 4 is set. ) Is set. The indicator 127 is display information used as an interface for an input operation that indicates an approximate position of an input point when data is input using the pointing device 4.
In this case, the guide information 125 and the indicator 127a respectively correspond to a slider control (or track control) bar (which defines a range) and an indicator (which indicates the current value) that are normally used in a computer application. Can be seen as a thing. The coordinate range 128a is a range to be specified by the pointing device 4 when setting the indicator 127a at the position (or selecting the simulated image 123a).
[0030]
In FIG. 13, similarly, an indicator 127b and a coordinate range 128b are set near the center position of the simulated image 123b in which the head 20 is facing directly upward, and the head 20 is located on the leftmost side (toward the drawing). In the simulated image 123c inclined rightward), an indicator 127c and a coordinate range 128c are set at the right position.
As shown in FIG. 14, the setting information includes an X coordinate value (value indicating a position in the vertical direction on the simulated image 123) and a Y coordinate value (simulated image) indicating the coordinate range 128a,..., 128b,. 123, a value indicating a position in the horizontal direction). That is, here, as shown in FIG. 14, each coordinate range is mapped to a value indicating the coordinate range in the space to be operated by the pointing device 4, and image data of each simulated image 123a, ..., 123b, ... 123c. A table for associating file names (FIG. 1, FIG. 2, FIG. 3, FIG. 4,.
[0031]
In the present embodiment, the coordinate ranges 128a to 128c are not set to a constant size, but the range of the length in the direction substantially perpendicular to the guide information 125, that is, the vertical direction of the simulated image 123 is gradually increased or decreased. It is set to do. That is, with respect to the arc-shaped guide information 125 extending in the horizontal direction, the coordinate ranges 128a to 128c are ranges that surround the upper and lower sides. Further, the vertical length of the coordinate range 128b located near the center of the guide information 125 is set to be larger than the vertical length of the coordinate range 128a or 128c located near the periphery. According to this, for example, when the indicator is moved in the horizontal direction, it is not always necessary to move the pointer of the pointing device 4 in an arc shape opened downward along the guide information 125; It may be moved so as to have an arcuate shape that is open. That is, the indicator can be moved by moving the pointer within the range indicated by the chain line in FIG.
[0032]
Assuming that the result of the simulated image mapping process for the guide information is obtained as shown in FIG. 14, the image / control data mapping unit 1042 in FIG. 8 next performs the control data mapping process for the simulated image (FIG. 9). Step SA5). This is a process for mapping the control data necessary for operating the entities like the robots 2a to 2c in FIG. 7 to the simulated images 123a to 123c. That is, here, as shown in FIG. 15, the control data file names (Sound1, Sound2, Sound3) described with reference to FIG. 6 and the like with respect to the image data file names (FIG1, FIG2, FIG3, FIG4,...). , Sound4,...) Is associated with the table data. In this case, the image data file name Fig1 corresponds to the simulated image 123a.
[0033]
As described above, in the simulated image data / control data storage unit 104 in FIG. 8, the coordinate data in the operation target space of the guide information and the image data file name indicating the simulated image data as shown in FIG. And a table showing the correspondence with the control data file name indicating the control data is created, and this is stored and held.
[0034]
The description of the functions of the drive data reproduction unit 110 and the simulated image data / control data storage unit 104 in the operation input / recording / reproduction / display control apparatus 100 in FIG. Next, the entire operation including other functional blocks in FIG. 3 will be described with reference to FIGS.
[0035]
First, referring to FIG. 17, at the same time as the data input operation for the simulated image 123 displayed on the display device 11 shown in FIG. 4, the operation for reproducing the drive data corresponding to the operation to the robot 2 will be described. explain.
[0036]
FIG. 4 is a diagram illustrating an example of the display screen 121 in the display device 11 illustrated in FIGS. 1 and 2. A window 122 created by an application corresponding to the input / recording / playback / display control apparatus 100 is displayed on the display screen 121, and a simulated image 123 is displayed in the window 122. In this case, the simulated image 123 displays a pointer 126 that indicates the position indicated by the operation of the pointing device 4. However, here, the case where the position of the pointer 126 is at the position indicated by the solid line in FIG. 4 is indicated by reference numeral 126a. In response to the operation of the pointing device 4, the pointer 126a moves to, for example, the pointer 126b or moves to the pointer 126b.
[0037]
In the example illustrated in FIG. 4, the simulated image 123 is a simulated image 123 a including the robot simulated image 124 a. Also, the coordinates corresponding to the pointer 126a are values within the coordinate range 128a of FIG. Furthermore, it is assumed that the pointers 126b and 126c indicate values in the coordinate ranges 128b and 128c in FIG.
[0038]
17, the pointing device event input unit 101 in FIG. 3 acquires event information of the pointing device 4 when the operator operates the simulated image 123 with the guide information 125 in FIG. Information is sent to the pointing device event determination unit 102 (step S1). Here, the event information includes coordinate information of the pointer 126 in the operation target space (range in the simulated image 123) and an event occurrence time.
[0039]
The pointing device event discriminating unit 102 discriminates whether or not the simulated image data that is the operation target that matches the condition exists in the event information that is sent while sequentially judging whether or not the pointing has been performed ( Steps S1-S3). In other words, the coordinates indicated by the pointer 126 are stored in association with any one of the image data file names included in the table shown in FIG. 16 (the X coordinate value and the Y coordinate value). Whether it corresponds to one of the specified range). For example, when the X coordinate is in the range of 10 to 30 and the Y coordinate is in the range of 32 to 48, the image data file name FIG1 corresponding to the simulated image 123a exists, so it is determined that it exists. A result is obtained (“Yes” in step S3).
[0040]
If it is determined that there is data, the stored data calling unit 103 displays the simulated image data 123 (123a) associated with the event information of the pointing device and the control data file name (FIG. 1) associated with the image data file name (FIG. 1). Sound1) is acquired from the simulated image data / control data storage unit 104, and the image data file is displayed (step S4) and the control data file is reproduced (step S5). When the image data file name Fig1 corresponding to the simulated image 123a and the control data file name Sound1 are called, they are reproduced by the stored data reproduction unit 107. Then, the simulated image display unit 11 serving as an interface with the display device 11 displays the simulated image 123a corresponding to the reproduced image data file name Fig1 on the display device 11 (FIG. 4), and the drive data reproducing unit 110. As a result, drive data (waveform Wa) corresponding to the control data file name Sound1 is output, and the robot 2 is driven and controlled as in the robot 2a of FIG.
[0041]
Similarly, in FIG. 4, when the pointer 126 moves like the pointer 126b and the pointer 126c, the display device 11 displays the simulated image 123b and the simulated image 123c shown in FIG. The operation of the robot 2 is driven and controlled like the robot 2b and the robot 2c in FIG.
[0042]
Next, referring to FIGS. 18 to 19, simultaneously with the data input operation for the simulated image 123 displayed on the display device 11 shown in FIG. 4, the drive data corresponding to the operation is reproduced for the robot 2. At the same time, the operation for recording the operation history will be described.
[0043]
FIG. 18 is a flowchart showing the flow of each process in this operation, and steps T1 to T5 in FIG. 18 correspond to the processes in steps S1 to S5 in FIG. In this case, in steps T1 to T5 in FIG. 18, processing corresponding to the processing in steps S1 to S5 in FIG. 17 is executed. In step T6, the event information acquired in step T1 is sent to the pointing device event recording unit 105 in FIG. The pointing device event recording unit 105 acquires the control data file name associated with the simulated image data file name that matches the input event information from the simulated image data / drive data recording unit 104, and the event number as shown in FIG. Are recorded as time series information together with the time of each event.
[0044]
Note that the process of step T6 in FIG. 18 may be provided between the processes of step T3 and step T4. Further, the process of step T5 can be omitted and the drive control of the robot 2 can be omitted.
[0045]
Next, referring to FIG. 20, based on the time series information recorded as shown in FIG. 19, the simulated image 123 corresponding to the information is displayed on the display device 11 in time series, and at the same time, The operation when reproducing the drive data corresponding to the information will be described. In FIG. 20, step R2 and step R3 correspond to the processing of step T4 and step T5 in FIG. 19, respectively.
[0046]
In this case, at step R1 in FIG. 20, the recording data calling unit 106 in FIG. 3 calls the time series information as shown in FIG. 19 recorded in the pointing device event recording unit 105, and this is the recording data reproduction. It is reproduced by the unit 108. The data reproduced by the recorded data reproducing unit 108 is reproduced by the simulated image display unit 109 and the drive data reproducing unit 110 in the same manner as the case of the stored data reproducing unit 109. Then, the corresponding simulated image 123 is displayed on the display device 11 and the operation of the robot 2 is controlled (steps R2 to R3).
[0047]
In FIG. 20, either step R2 or step R3 may be omitted.
[0048]
Next, a modified example of the pointing device event determination unit 102 in FIG. 3 will be described with reference to FIGS. 21 and 22. FIG. 21 is a block diagram illustrating a configuration of the pointing device event determination unit 102a in the present example. FIG. 22 is a diagram illustrating an example of guide information 125a and the like displayed on the simulated image 123 in the display device 11 in this example.
[0049]
As shown in FIG. 21, the pointing device event determination unit 102a includes a coordinate input unit 1021, a mapping unit 1022, a guide information selection unit 1023, a guide information display unit 1024, and an event determination unit 1025. . In this case, the coordinate input unit 1021 has a function of inputting the coordinate information input by the pointing device input unit 101 in FIG.
The event determination unit 1025 has the same function as the function of the entire pointing device event determination unit 102 of FIG. However, the coordinate information to be determined is the coordinate information itself sent from the pointing device event input unit 101 in the pointing device event determination unit 102, whereas the coordinate information to be determined is from the pointing device event input unit 101 in the event determination unit 1025. The difference is that the coordinate information sent is converted by the mapping unit 1022 into coordinate information on the guide information 1025a, for example.
[0050]
The guide information selection unit 1023 has a function of determining which guide information to select when a plurality of guide information is displayed on the same simulated image 123. FIG. 22 shows a case where two pieces of guide information 125a corresponding to the movement of the head 20 in FIG. 1 and guide information 125b corresponding to the movement of the left front leg portion 22 are displayed simultaneously. In this case, the guide information selection unit 1023 selects the guide information 125a closer to the pointer 126 and notifies the mapping unit 1022 and the event determination unit 1025.
[0051]
The mapping unit 1022 has a function of converting the coordinates indicated by the pointer 126 into coordinate information on the guide information 125a selected by the guide information selection unit 1023. For example, when the pointer 126 indicates the coordinates of the point E, the coordinates are converted into the coordinates of the point G on the guide information 125a. When the pointer 126 moves from the point E to the point F, the coordinates corresponding to the points G to B to H on the guide information 125a are obtained as a conversion result. That is, in this example, the mapping unit 1022 performs coordinate conversion (mapping) as follows.
First, a perpendicular line is drawn from the polygonal line ABC constituting the guide information 125a to the coordinate point of the pointer 126 (for example, point E), and the coordinates of the foot of the perpendicular line (point G) are converted into coordinate points on the guide information. Do. However, when the pointer 126 is located in a range where the perpendicular cannot be drawn (regions (a), (b), and (c)), the coordinates of the point A, the point B, and the point C are used as conversion destinations, respectively. select. As a result, the coordinates indicated by the pointer 126 are replaced with the coordinate information on the guide information 125a.
[0052]
The guide information display unit 1024 displays an indicator 129 on the guide information (guide information 125a in this example) selected by the guide information selection unit 1023 so as to correspond to the coordinate information after conversion obtained by the mapping unit 1022. To do. In addition, when the indicator 129 is moved, the guide information display unit 1024 performs a redrawing process or the like of the guide information 125a after the movement.
[0053]
As described above, the event determination unit 1025 is based on the converted coordinate information sent from the mapping unit 1022 via the guide information display unit 1024 and the guide information selection result by the guide information selection unit 1023. It is determined whether or not the corresponding image data file and control data file exist. The determination result is output to the stored data calling unit 103 and the pointing device event recording unit 105 in the same manner as the pointing device event determining unit 102 in FIG.
[0054]
According to the present modification, the coordinate information input by the pointing device 4 is once converted into the coordinate information on the guide information 125a and the guide information 125b, and then the event information is determined. In the case of a complicated case, an effect that a more intuitive operation can be expected.
[0055]
In this example, in order to simplify the explanation, the guide information is a polygonal line composed of two line segments connecting three points. However, even if it is a polygonal line, the guide information is composed of a plurality of line segments or has a curved shape. Etc. can be changed.
[0056]
As described above, according to the embodiment of the present invention, for example, in an apparatus that supports input of the motion of an object such as a robot that is interesting for the viewer, the motion of the object such as a robot can be simplified. , Can be entered and coded. At this time, in the present embodiment, information indicating the locus of movement of the object is displayed as guidance information on the display device, and when drive control is performed in accordance with the operation of the pointing device 4 that performs movement input. Since the same simulated image as the motion of the obtained object is displayed while being sequentially updated, the operation (direction and amount of movement) of the pointing device 4 and the corresponding movement change of the actual object can be easily and intuitively performed. Can grasp.
[0057]
In addition, since the phase (direction and size) of the operation instruction is obtained from the coordinates given by the pointing device in accordance with the guide information, the degree of freedom of operation of the pointing device is appropriately reduced, and , It is possible to improve the response by reducing the amount of calculation required for reproduction.
[0058]
The embodiment of the present invention includes a computer and a program executed using the hardware resources, and the program can be distributed via a computer-readable recording medium or a communication line. Is possible. In addition, the data area secured when the program is executed is secured using a recording medium such as a memory.
[0059]
In the above-described embodiment, the case where the head 20 is moved has been mainly described. However, it is possible to deal with the case where the movement of another part is added to the target. For example, regarding the left and right front legs of the robot 2, when considering the movement pattern of moving the right leg up / middle / bottom and the left leg up / middle / bottom, there are nine movements with the square of 3. In such a case, pseudo images are prepared for all of these combinations and are held in advance. As a method for generating a simulated image at that time, it is conceivable to use a background extraction method (more precisely, a background difference method and an interframe difference method).
Here, the inter-frame difference method is a method of subtracting the previously captured image from the periodically captured input image and extracting a region having a large difference value as a moving region. The background subtraction method is a method of subtracting a background image created in advance from an input image and extracting a region having a large absolute value as a region of the drive target object.
[0060]
Further, according to the present invention, if there is a frame advance image for each part of the real robot, it becomes possible to easily input and operate the operation of the robot.
[0061]
【The invention's effect】
By the above means, according to the present invention, when creating a robot motion teaching application capable of inputting / recording / reproducing a robot motion (when inputting robot motion teaching data), the application can be easily created ( (Data input) can be supported, and by using the created application (data), the operation of the robot can be easily recorded and reproduced by an intuitive operation.
Further, by controlling the robot using a synthesized signal including a drive instruction and audio information that should be generated at the same time, it is possible to reproduce the motion, sound, and image while synchronizing them. .
[0062]
Also, when inputting / recording / reproducing / displaying the robot motion, the operator operates the guide along the guide information displayed together with the simulated image showing the robot on the operation target screen, thereby providing an event-driven guide. Since the simulated image of the robot associated with the information can be changed and operation data can be input and recorded at the same time, for example, robot operation data that matches the operation data can be easily input by intuitive operation. It is possible to record / reproduce, and to easily reproduce the operation while synchronizing with the sound and the image.
[Brief description of the drawings]
FIG. 1 is an external view showing a configuration of an embodiment of the present invention.
FIG. 2 is a block diagram showing a configuration of the embodiment shown in FIG.
3 is a block diagram showing a functional configuration of the input / recording / playback / display control apparatus 100 shown in FIG. 2; FIG.
4 is a diagram showing a display example on the display device 11 shown in FIG. 1. FIG.
FIG. 5 is a block diagram showing a configuration of the robot 2 in FIG. 1;
6 is a diagram for explaining an example of a relationship between a control data file for generating drive data output from the input / recording / playback / display control apparatus 100 of FIG. 3 and control contents in the robot 2. FIG. .
7 is a diagram showing the relationship between the operation of the robot 2 in FIG. 1 and the waveform of the drive data reproduced by the drive data reproducing unit 110 in FIG. 3;
8 is a block diagram showing a configuration of a simulated image data / control data storage unit 104 in FIG. 3. FIG.
9 is a flowchart showing the operation of the simulated image data / control data storage unit 104 of FIG.
10 is an explanatory diagram for explaining a basic operation for creating a simulated image in the image information calculation unit 1041 of FIG. 8. FIG.
11 is a diagram showing an example of a robot simulation image created by the image information calculation unit 1041 in FIG.
12 is a diagram showing an example of guide information 125 given to a robot simulated image by the image information calculation unit 1041 in FIG. 8. FIG.
13 is a diagram showing an example of a simulated image to which guide information 125 is given by the image information calculation unit 1041 in FIG.
14 is a diagram showing a table showing correspondence between guide information and image data files in the image / control data mapping unit 1042 of FIG. 8. FIG.
15 is a diagram showing a table showing the correspondence between image data files and control data files in the image / control data mapping unit 1042 of FIG. 8. FIG.
16 is a diagram illustrating an example of a table stored in the simulated image data / control data storage unit 104 of FIG. 3;
17 is a flowchart showing an operation in the case where an operation input and reproduction by the input / recording / reproduction / display control apparatus 100 of FIG. 3 are performed simultaneously.
18 is a flowchart showing an operation in the case where an operation input and recording / reproduction by the input / recording / reproduction / display control apparatus 100 of FIG. 3 are simultaneously performed.
FIG. 19 is a diagram showing an example of data recorded during operation input and recording / reproduction shown in FIG. 18;
20 is a flowchart showing an operation at the time of reproduction of recorded data by the input / recording / reproduction / display control apparatus 100 of FIG. 3;
21 is a block diagram illustrating a modification of the pointing device event determination unit 102 of FIG. 3;
22 is an explanatory diagram for describing a function of a pointing device event determination unit 102a in FIG. 21;
[Explanation of symbols]
1 Personal computer
2 Robot
3 audio lines
4 pointing devices
5 Image input device
10 Main unit
11 Display device
100 Input / Recording / Playback / Display Control Device
101 Pointing device event input section
102, 102a Pointing device event discriminating unit
104 Simulated image data / control data storage unit
105 Pointing device event recording unit
109 Simulated image display
110 Drive data reproduction unit
123 Simulated image
124 Robot simulation image
125 Guide information
1040 Image information input unit
1041 Image information calculation unit
1042 Image / control data mapping unit

Claims (7)

A method for teaching an operation of a real object that performs an operation corresponding to the control data by inputting control data using a predetermined operator,
As a preparation stage of guidance information that is easy to operate with the predetermined operator,
Shooting the movement of the real object by sequentially sending the control data,
When the change in the action of the real object in the captured image exceeds a predetermined size, create a plurality of simulated images for the image frame,
A moving region for each of the simulated images created is determined by a background difference method, a center of gravity of a predetermined region specified by the predetermined operator in the moving region determined for each of the simulated images is determined, and a plurality of determined center of gravity motions are obtained The guide information is created by expanding the obtained trajectory in the horizontal and vertical directions and converting it into image information.
Several indicators are provided in the guidance information, and a coordinate range that is an area for selection with the predetermined operation element is set for each indicator.
For each indicator, associate the simulated image based on the position of the indicator in the guidance information,
For each simulated image, control data for causing the real object to perform an operation corresponding to the simulated image is associated,
As an operation stage of the real object using the guidance information,
Displaying any one of the simulated images together with the guidance information;
When the coordinates designated by the predetermined operator are within the coordinate range in the guidance information, the simulated image associated with the indicator corresponding to the coordinate range is displayed and the simulated image is supported. Get the attached control data,
An operation input method comprising: outputting control data associated with the simulated image to the real object . The motion input method according to claim 1, wherein the simulated image is obtained by performing CG processing of a captured image. An apparatus that teaches the operation of a real object using a predetermined operator with respect to a real object that performs an operation corresponding to the control data when the control data is input.
As a means for preparing guidance information that is easy to operate with the predetermined operator,
Means for photographing the movement of the real object by sequentially sending the control data;
Means for creating a plurality of simulated images for the image frame when a change in the action of the real object exceeds a predetermined size in the captured image;
A moving region for each of the simulated images created is determined by a background difference method, a center of gravity of a predetermined region specified by the predetermined operator in the moving region determined for each of the simulated images is determined, and a plurality of determined center of gravity motions are obtained Means for creating a guide information obtained by expanding the obtained trajectory in the horizontal and vertical directions and converting it into image information;
Means for providing several indicators in the guidance information, and setting a coordinate range as an area when selecting with the predetermined operator for each indicator;
Means for associating the simulated image for each indicator based on the position of the indicator in the guidance information;
Means for associating control data for causing the real object to perform an operation corresponding to the simulated image for each simulated image;
As means for operating the real object using the guidance information,
Means for displaying any one of the simulated images together with the guidance information;
When the coordinates designated by the predetermined operator are within the coordinate range in the guidance information, the simulated image associated with the indicator corresponding to the coordinate range is displayed and the simulated image is supported. Means for acquiring attached control data;
Means for outputting control data associated with the simulated image to the real object ;
An operation input device comprising: The motion input device according to claim 3, wherein the simulated image is obtained by performing CG processing of a captured image. To a control computer that controls a device that teaches the operation of the real object using a predetermined operator with respect to a real object that performs an operation corresponding to the control data when the control data is input.
As a step of the preparation stage of guidance information that is easy to operate with the predetermined operator,
Photographing the movement of the real object by sequentially sending the control data;
Creating a plurality of simulated images for the image frame when a change in motion of the real object exceeds a predetermined magnitude in the captured image;
A moving area for each of the created simulated images is determined by a background difference method, a center of gravity of a predetermined area specified by the predetermined operator in the moving area determined for each of the simulated images is obtained, and movements of the obtained plurality of center of gravity are obtained. Creating a guide information obtained by expanding the obtained trajectory in the horizontal and vertical directions and converting it into image information;
Providing a number of indicators in the guidance information, and setting a coordinate range to be an area when selecting with the predetermined operator for each indicator;
For each indicator, associating the simulated image based on the position of the indicator in the guidance information;
Associating control data for causing the real object to perform an operation corresponding to the simulated image for each simulated image;
As a step of the operation stage of the real object using the guidance information,
Displaying any one of the simulated images together with the guidance information;
When the coordinates specified by the predetermined operator are within the coordinate range in the guidance information, the simulated image associated with the indicator corresponding to the coordinate range is displayed and the simulated image is supported. Obtaining attached control data;
Outputting control data associated with the simulated image and acquired to the real object ;
A program for running The program according to claim 5, wherein the simulated image is obtained by performing CG processing of a captured image. A computer-readable recording medium on which the program according to claim 5 or 6 is recorded.

Images