JP7750072B2 - Image processing device, robot control system and control method - Google Patents

Description

This disclosure relates to an image processing device, a robot control system, and a control method.

Conventionally, robots that pick multiple items one by one from a container are known at production sites. It is desirable for the robot to avoid interfering with the container when picking an object. JP 2019-188516 A (Patent Document 1) discloses a system that detects the position and orientation of a parts box based on sensory information acquired by capturing an image of the container, and then uses the detection results to perform operations to avoid interference between the container and the robot.

Japanese Patent Application Laid-Open No. 2019-188516

At production sites, containers may move due to unintended forces from the external environment, such as workers or robots. If a robot continues to operate without noticing changes in the container's position and orientation, interference between the container and robot may occur. Patent Document 1 does not take into account the possibility that the container's position and orientation may change.

This disclosure was made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device, robot control system, and control method that can reduce interference between containers and robots.

According to one example of the present disclosure, an image processing device processes an image from a camera. The camera is mounted on a robot having an end effector so that the relative position of the camera to the end effector that picks and places an object from a first area to a second area is constant. The image processing device includes a first acquisition unit, a second acquisition unit, and a detection unit. The first acquisition unit acquires initial information indicating the initial position and orientation of a container that holds the object in the first area or the second area. The second acquisition unit acquires an object image obtained by capturing an image of the camera after the start of pick-and-place, with at least a portion of the container having the initial position and orientation included in its field of view. The detection unit detects changes in the position and orientation of the container based on the object image and the initial information.

According to this disclosure, even if a container is moved due to an unintended force from the external environment, the change in the container's position and posture can be recognized. This allows appropriate measures to be taken, such as changing or stopping the robot's operation, in response to the detected change in the container's position and posture. As a result, interference between the container and the robot can be reduced.

In the above disclosure, the image processing device further includes a measurement unit and a setting unit. The measurement unit measures the position and orientation of an object present in the first area using an image from the camera. The setting unit sets a path for the end effector to pick up the object from the home position, place the object in the second area, and return to the home position based on the position and orientation of the object. The path includes waypoints that correspond to the position of the end effector when a characteristic portion of the container having the initial position and orientation is included in the field of view of the camera. The object image is obtained by capturing an image with the camera at the time the end effector passes through the waypoint.

According to the above disclosure, target images are acquired during the pick-and-place operation. Therefore, there is no need to set up an operation for acquiring target images separately from the pick-and-place operation.

In the above disclosure, the waypoint is set between placing the object and returning to the home position.

According to the above disclosure, an image of a target is captured when the end effector is not holding an object. As a result, the target image does not include the object being held by the end effector.

In the above disclosure, the container is a rectangular box-like container in a plan view. The characteristic portion includes at least one of the corners of the container, the edges of the top and side surfaces of the container, and a mark formed on the top surface.

According to the above disclosure, changes in the position and orientation of a container can be easily detected by identifying the position of characteristic parts based on the target image.

In the above disclosure, the first acquisition unit re-acquires the initial information in response to a change in the position and orientation of the container detected by the detection unit.

Alternatively, the image processing device may further include an update unit that identifies the position and orientation of the container based on the target image and updates the initial information according to the identified position and orientation.

According to the above disclosure, when the position and orientation of a container changes, the initial information is updated according to the changed position and orientation. As a result, interference between the container and the robot can be reduced.

According to one example of the present disclosure, a robot control system includes a robot, a camera, a first acquisition unit, a second acquisition unit, and a detection unit. The robot has an end effector that picks and places an object from a first area to a second area. The camera is mounted on the robot so that its relative position with the end effector is constant. The first acquisition unit acquires initial information indicating the initial position and orientation of a container that holds the object in the first area or the second area. The second acquisition unit acquires an object image obtained by capturing an image of the container with the initial position and orientation after the start of pick-and-place, with at least a portion of the container included in its field of view. The detection unit detects changes in the position and orientation of the container based on the object image and the initial information.

According to one example of the present disclosure, a control method for an image processing device that processes camera images comprises first to third steps. The first step is a step of acquiring initial information indicating the initial position and orientation of a container that holds an object in a first area or a second area. The second step is a step of acquiring an object image obtained by capturing an image with the camera after pick-and-place has started, with at least a portion of the container having the initial position and orientation included in the field of view. The third step is a step of detecting changes in the position and orientation of the container based on the object image and the initial information.

These disclosures will also help prevent interference between containers and robots.

This disclosure makes it possible to reduce interference between containers and robots.

1 is a schematic diagram illustrating an overall configuration of a robot control system according to an embodiment. FIG. 2 is a schematic diagram showing a concept of a specific structure of the camera shown in FIG. 1 . FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of the image processing apparatus illustrated in FIG. 1 . FIG. 1 is a diagram illustrating an example of a coordinate system used in a robot control system. FIG. 2 is a block diagram showing an example of a functional configuration of the image processing apparatus shown in FIG. 1 . FIG. 1 illustrates an example of a container. 10A and 10B are diagrams illustrating an example of a method for identifying the position and orientation of a container. 10A and 10B are diagrams illustrating another example of a method for identifying the position and orientation of a container. FIG. 10 is a diagram illustrating an example of a search process for a camera position. FIG. 10 is a diagram showing an example of a target image when the position and orientation of the container are not changed. FIG. 10 is a diagram showing an example of a target image when the position and orientation of a container are changing. FIG. 10 is a diagram showing another example of a target image when the position and orientation of the container have not changed. FIG. 10 is a diagram showing another example of the target image when the position and orientation of the container are changing. FIG. 10 is a diagram showing yet another example of a target image when the position and orientation of the container have not changed. FIG. 10 is a diagram showing yet another example of a target image when the position and orientation of a container are changing. 10 is a flowchart illustrating an example of the overall flow of processing by the image processing device. 17 is a flowchart showing the flow of processing in a subroutine of step S1 shown in FIG. 16. 17 is a flowchart showing an example of the process flow of a subroutine of step S5 shown in FIG. 16. 17 is a flowchart showing another example of the process flow of the subroutine of step S5 shown in FIG. 16. 17 is a flowchart showing an example of the process flow of a subroutine of step S6 shown in FIG. 16. FIG. 10 is a block diagram showing an example of a functional configuration of an image processing device according to a first modified example. FIG. 10 is a diagram illustrating the relationship between a target image and an auxiliary image. 10 is a flowchart showing the flow of processing in a subroutine of step S6 according to Modification 1.

Embodiments of the present invention will be described in detail with reference to the drawings. Note that identical or equivalent parts in the drawings will be designated by the same reference numerals and their description will not be repeated.

§1 Application Example An example of a situation in which the present invention is applied will be described with reference to Figure 1. Figure 1 is a schematic diagram showing the overall configuration of a robot control system according to an embodiment. The robot control system 1 illustrated in Figure 1 is incorporated into, for example, a production line, and is used to control the movement of an object from a first area in a container 5 to a second area in a container 6. The object is selected from one or more articles 2 stored in the container 5.

As shown in FIG. 1, the robot control system 1 includes an image processing device 100, a camera 200, a robot 300, and a robot control device 400.

The camera 200 captures images of subjects including the item 2, containers 5 and 6, etc. The camera 200 outputs image data (hereinafter simply referred to as "image") obtained by capturing the image to the image processing device 100.

The camera 200 has a first mode in which it operates as a 3D camera and a second mode in which it operates as a 2D camera, and captures an image of a subject in a mode that corresponds to an image capture command from the image processing device 100. For example, when the first mode is specified, the camera 200 captures an image of the subject while projecting a projection pattern of light onto the subject. The projection pattern is, for example, a pattern in which the brightness changes periodically along a predetermined direction within the irradiation surface. When the second mode is specified, the camera 200 captures an image of the subject while irradiating the subject with uniform light.

The first mode is specified when measuring the position and orientation of one or more items 2 in a container 5. The range R in which the position and orientation can be measured by imaging in the first mode is determined according to the viewing angle and depth of field of the camera 200.

The second mode is specified when determining the initial position and orientation of the containers 5 and 6. The containers 5 and 6 are placed on a predetermined surface (e.g., the top of a desk, the floor, etc.). Therefore, the height of the top surfaces of the containers 5 and 6 is known. Therefore, the initial position and orientation of the containers 5 and 6 can be determined even using images obtained by imaging in the second mode (hereinafter referred to as "2D images").

Robot 300 picks and places objects. Robot 300 is, for example, a vertical articulated robot. The robot 300 illustrated in FIG. 1 includes a base 301, an articulated arm 302, a flange plate 303, a support member 304, and an end effector 305. Note that FIG. 1 illustrates the robot 300 in a schematic manner, and the shapes of the individual components are not limited to the example shown in FIG. 1.

The base 301 is installed at a fixed position in the production site. One end of the articulated arm 302 is connected to the base 301. A flange plate 303 is provided at the other end of the articulated arm 302. An end effector 305 is attached to the flange plate 303 via a support member 304.

The end effector 305 picks and places an object from a first area in the container 5 to a second area in the container 6. That is, the end effector 305 picks an object from one or more items 2 in the container 5 and places the object in the container 6. The end effector 305 includes, for example, a two-fingered hand, a multi-fingered hand, a suction pad, etc.

The camera 200 is mounted on the robot 300 so that its relative position with respect to the end effector 305 is constant. Specifically, the camera 200 is attached to the support member 304 of the robot 300.

The robot control device 400 controls the articulated arm 302 in accordance with a movement command from the image processing device 100. The movement command indicates, for example, the position and orientation of the end effector 305 (which can also be said to be the position and orientation of the surface of the flange plate 303 to which the end effector 305 is attached). The robot control device 400 controls the articulated arm 302 so that the position and orientation of the end effector 305 matches the movement command.

Furthermore, the robot control device 400 controls the end effector 305 in accordance with the grip command and release command from the image processing device 100. The grip command instructs the end effector 305 to grip an object. The release command instructs the end effector 305 to release the object.

The image processing device 100 determines the position and orientation that the end effector 305 should assume based on the image received from the camera 200. The image processing device 100 generates a movement command to move the end effector 305 to the determined position and orientation, and outputs the generated movement command to the robot control device 400. The image processing device 100 outputs a grip command or release command to the robot control device 400 at the timing when the end effector 305 has moved to the determined position and orientation.

The image processing device 100 may also determine the position and orientation that the camera 200 should assume. The image processing device 100 generates a movement command for moving the camera 200 to the determined position and orientation, and outputs the generated movement command to the robot control device 400. The image processing device 100 outputs an image capture command to the camera 200 at the timing when the camera 200 has moved to the determined position and orientation.

As shown in FIG. 1, the image processing device 100 includes an information acquisition unit 11, an image acquisition unit 12, and a detection unit 13.

The information acquisition unit 11 acquires initial information indicating the initial positions and orientations of the containers 5 and 6. For example, the information acquisition unit 11 identifies the initial positions and orientations of the containers 5 and 6 based on the positions of characteristic parts (e.g., corners) of the containers 5 and 6 shown in the 2D image and the shape data of the containers 5 and 6. The information acquisition unit 11 generates initial information indicating the identified initial positions and orientations.

After the end effector 305 starts picking and placing the object, the image acquisition unit 12 acquires from the camera 200 a first object image showing at least a portion of the container 5 and a second object image showing at least a portion of the container 6.

The detection unit 13 detects changes in the position and orientation of the container 5 based on the first target image and the initial information. Furthermore, the detection unit 13 detects changes in the position and orientation of the container 6 based on the second target image and the initial information.

According to this embodiment, changes in the position and orientation of the containers 5, 6 are detected. Therefore, even if the containers 5, 6 are moved due to unintended forces from the external environment, the change in the position and orientation of the containers 5, 6 is recognized. As a result, appropriate measures can be taken, such as changing or stopping the operation of the robot 300, in response to the detected change in the position and orientation of the containers 5, 6. As a result, interference between the containers 5, 6 and the robot 300 can be reduced.

§2 Specific examples <Camera configuration>
Fig. 2 is a schematic diagram showing the concept of the specific structure of the camera shown in Fig. 1. As shown in Fig. 2, the camera 200 has a projection unit 21 and an imaging unit 22. The projection unit 21 projects an arbitrary projection pattern of light onto a subject in accordance with instructions from the image processing device 100. The imaging unit 22, in accordance with instructions from the image processing device 100, images the subject with the projection pattern of light projected onto it.

The projection unit 21 has, as its main components, a light source 210, such as an LED (Light Emitting Diode) or a halogen lamp, a filter 211, and an optical system 212. The projection unit 21 is formed by assembling together the light source 210, filter 211, optical system 212, lens barrel, and support structure (not shown).

The light source 210 emits light of a predetermined wavelength toward the filter 211. The filter 211 arbitrarily changes the in-plane light transmittance in accordance with instructions from the image processing device 100. The filter 211 includes, for example, a photomask, liquid crystal, or a DMD (Digital Mirror Device). When the filter 211 receives an instruction to capture images in a first mode from the image processing device 100, it generates the projection pattern light required to measure a three-dimensional shape. When the filter 211 receives an instruction to capture images in a second mode from the image processing device 100, it makes the in-plane transmittance uniform. The optical system 212 includes one or more lenses. The light that passes through the filter 211 is irradiated externally via the optical system 212. As a result, projection pattern light according to the state of the filter 211 is irradiated into external space.

The imaging unit 22 has an imaging element 220 as an optical device, an optical system 221, and a lens barrel and support structure (not shown). That is, the imaging unit 22 is composed of an assembly formed by assembling the imaging element 220, optical system 221, lens barrel, and support structure together. The imaging unit 22 captures an image of an object with light projected by the projection unit 21. More specifically, an image is obtained when the imaging element 220 receives light that has passed through the optical system 221. In the first mode, the imaging unit 22 captures an object with a projection pattern light required for measuring a three-dimensional shape projected onto the object, and in the second mode, it captures an object with uniform light projected onto the object. In this way, the imaging unit 22 is shared in both the first and second modes.

The field of view of camera 200 is determined by optical system 221. In this embodiment, camera 200 has an imaging unit 22 that is shared by both the first mode and the second mode. Therefore, the field of view of camera 200 is the same in both the first mode and the second mode. Increasing the field of view of camera 200 reduces the resolution. Camera 200 is primarily used to measure the position and orientation of one or more objects 2. Therefore, the field of view of camera 200 is set so as to obtain the resolution required when performing 3D measurement based on images obtained in the first mode.

Note that by changing the specifications of the optical system 221, it is possible to obtain a wide field of view while maintaining high resolution. However, in this case, the size and weight of the optical system 221 will increase. As described above, the camera 200 is preferably small and lightweight because it is mounted on the robot 300 and moves. Therefore, the field of view of the camera 200 is limited to a range that does not affect the movement of the camera 200. As a result, if the size of the container 5 is changed significantly, there is a possibility that the top surface 501 of the container 5 will not be included in the field of view of the camera 200 at the position of the camera 200 when the range R in which the position and orientation can be measured by first mode imaging includes the item 2 inside the container 5 (see Figure 1).

<Hardware configuration of image processing device>
The image processing device 100 is typically a computer having a general-purpose architecture, and executes the processing according to the present embodiment by executing a pre-installed program (instruction code). Such a program is typically distributed in a state stored on various recording media, or is installed in the image processing device 100 via a network, etc.

When using such a general-purpose computer, an OS (Operating System) for executing basic computer processing may be installed in addition to an application for executing the processing according to this embodiment. In this case, the program according to this embodiment may execute processing by calling the necessary modules, among the program modules provided as part of the OS, in a predetermined sequence at a predetermined timing. In other words, the program according to this embodiment itself may not include the modules described above, and may execute processing in cooperation with the OS. The program according to this embodiment may also be in a form that does not include some of these modules.

Furthermore, the program according to this embodiment may be provided as part of another program. In this case, the program itself does not include the modules included in the other program with which it is combined, and processing is performed in cooperation with the other program. In other words, the program according to this embodiment may be in a form that is incorporated into such other program. Note that some or all of the functions provided by the execution of the program may be implemented as dedicated hardware circuits.

Figure 3 is a schematic diagram showing an example of the hardware configuration of the image processing device shown in Figure 1. As shown in Figure 3, the image processing device 100 includes a CPU (Central Processing Unit) 101, which is an arithmetic processing unit, a main memory 102 and a hard disk 103, which serve as storage units, a camera interface 104, an input interface 105, a display controller 106, a communication interface 107, and a data reader/writer 108. These units are connected to each other via a bus 109 so that they can communicate data with each other.

The CPU 101 performs various calculations by loading programs (code) installed on the hard disk 103 into the main memory 102 and executing them in a predetermined order. The main memory 102 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory), and in addition to programs read from the hard disk 103, it also stores images captured by the camera 200. The hard disk 103 also stores various data, as described below. Note that a semiconductor storage device such as a flash memory may be used in addition to or instead of the hard disk 103.

Camera interface 104 mediates data transmission between CPU 101 and camera 200. In other words, camera interface 104 is connected to camera 200. Camera interface 104 issues image capture commands to camera 200 in accordance with internal commands generated by CPU 101. The image capture command includes projection conditions and image capture conditions. Camera interface 104 includes image buffer 104a for temporarily storing images from camera 200. When a predetermined number of images have been stored in image buffer 104a, camera interface 104 transfers the stored images to main memory 102.

The input interface 105 mediates data transmission between the CPU 101 and the input device 150. That is, the input interface 105 accepts input information entered by the user into the input device 150. The input device 150 includes a keyboard, mouse, touch panel, etc.

The display controller 106 is connected to the display 160 and controls the screen of the display 160 to notify the user of processing results from the CPU 101, etc.

The communication interface 107 mediates data transmission between the CPU 101 and external devices such as the robot control device 400. The communication interface 107 typically comprises Ethernet (registered trademark) or USB (Universal Serial Bus).

Data reader/writer 108 mediates data transmission between CPU 101 and memory card 700, which is a recording medium. That is, memory card 700 stores programs to be executed by image processing device 100, and data reader/writer 108 reads the programs from memory card 700. In response to internal commands from CPU 101, data reader/writer 108 also writes images captured by camera 200 and/or processing results from image processing device 100 to memory card 700. Memory card 700 may be a general-purpose semiconductor storage device such as an SD (Secure Digital), a magnetic storage medium such as a flexible disk, or an optical storage medium such as a CD-ROM (Compact Disk Read Only Memory).

Furthermore, the image processing device 100 may be equipped with an interface that can be connected to at least one of an RFID (Radio Frequency Identification) reader and a barcode reader.

<Coordinate system>
The CPU 101 calculates the positions and orientations of the article 2, the camera 200, and the robot 300 using coordinate values of multiple coordinate systems.

Figure 4 shows an example of a coordinate system used in a robot control system. As shown in Figure 4, the robot control system 1 uses a robot base coordinate system, a flange coordinate system (also called a tool coordinate system), a camera coordinate system, and a workpiece coordinate system.

The robot base coordinate system is a coordinate system based on the base 301 of the robot 300. The robot base coordinate system is defined by the origin Ob set on the base 301 and the base vectors Xb, Yb, and Zb.

The flange coordinate system is a coordinate system based on the flange plate 303 attached to the other end of the articulated arm 302 of the robot 300. The end effector 305 and camera 200 are attached to the flange plate 303 via a support member 304. The flange coordinate system is defined by an origin Of set on the flange surface of the flange plate 303 and basis vectors Xf, Yf, and Zf.

The camera coordinate system is a coordinate system based on the imaging unit 22 of the camera 200. The camera coordinate system is defined by an origin Oc set in the imaging unit 22 and base vectors Xc, Yc, and Zc. The base vector Zc is on the optical axis of the imaging unit 22.

The work coordinate system is a coordinate system based on item 2. The work coordinate system is defined by the origin Ow set on item 2 and the base vectors Xw, Yw, and Zw.

The position and orientation of the article 2 detected based on the image received from the camera 200 is indicated by, for example _, a coordinate transformation matrix ^CHW _that transforms the camera coordinate system into the work coordinate system. The coordinate transformation matrix ^CHW represents the position of the origin Ow of the work coordinate system in the camera coordinate system and the base vectors Xw, Yw, and Zw.

The camera 200 is fixed to the flange plate 303 via a support member 304. Therefore, the relative positional relationship between the camera 200 and the flange plate 303 is constant. This relative positional relationship is represented, for example _, by a coordinate transformation matrix ^FHC _that transforms the flange coordinate system into the camera coordinate system. The coordinate transformation matrix ^FHC represents the position of the origin Oc of the camera coordinate system in the flange coordinate system and the basis vectors Xc, Yc, and _Zc . The coordinate transformation ^{matrix FHC} is represented by fixed values and is determined by calibration that is performed in advance. For example, the calibration may employ known hand-eye calibration in which the imaging unit 22 of the camera 200 captures an image of a marker installed at a fixed position while the robot 300 is operating.

The movement command output from the image processing device 100 to the robot control device 400 is represented by, for example, a coordinate transformation matrix ^BH _F that transforms the robot base coordinate system into the flange coordinate system. The coordinate transformation matrix ^BH _F represents the origin Of and the base vectors Xf, Yf, and Zf of the flange coordinate system in the robot base coordinate system.

The position and orientation of the camera 200 in the robot base coordinate system is indicated by, for example, a coordinate transformation matrix ^B H _C that transforms the robot base coordinate system into the camera coordinate system. The coordinate transformation matrix ^B H _C is expressed as follows:
^B H _C = ^B H _F・^F H _C
As described above, the coordinate transformation matrix ^F H _C is expressed by fixed values. Therefore, the CPU 101 can calculate the movement command to be output to the robot control device 400 from the position and orientation of the camera 200 at the movement destination.

The position and orientation of the article 2 in the robot base coordinate system is represented by, for example _, a coordinate transformation matrix ^BHW that transforms the robot base coordinate system into the workpiece coordinate system. The coordinate transformation _matrix ^BHW is expressed as follows:
^B H _W = ^B H _F・^F H _C・^C H _W
As described above, the coordinate transformation matrix ^C H _W indicates the position and orientation of the article 2 in the camera coordinate system, and is detected based on the image received from the camera 200. Therefore, the CPU 101 can calculate the position and orientation of the article 2 in the robot base coordinate system using the movement command output to the robot control device 400 and the measurement results based on the image.

<Example of functional configuration of image processing device>
Fig. 5 is a block diagram showing an example of the functional configuration of the image processing device shown in Fig. 1. As shown in Fig. 5, the image processing device 100 includes a storage unit 10, an information acquisition unit 11, an image acquisition unit 12, a detection unit 13, a shape data acquisition unit 14, a measurement unit 15, a setting unit 16, and a command unit 17. The storage unit 10 is realized by the main memory 102 and the hard disk 103 shown in Fig. 3. The information acquisition unit 11, the image acquisition unit 12, the detection unit 13, the shape data acquisition unit 14, the measurement unit 15, the setting unit 16, and the command unit 17 are realized by the CPU 101 shown in Fig. 3 executing a program stored on the hard disk 103.

(Storage part)
The memory unit 10 stores calibration data 10a, multiple template data 10b, container shape data 10c, first initial information 10d, second initial information 10e, at least one gripping point data 10f, and home position data 10g.

The calibration data 10a is data that defines the relative positional relationship between the camera 200 and the flange plate 303. The calibration data 10a is generated in advance, for example, by known hand-eye calibration, and stored in the memory unit 10.

The multiple template data 10b indicate the coordinates of points on the surface of a model having the same shape as the item 2 when viewed from multiple virtual viewpoints in virtual space. The multiple template data 10b are created in advance based on data indicating the three-dimensional shape of the item 2 and stored in the memory unit 10.

Container shape data 10c indicates the three-dimensional shape of containers 5 and 6. Container shape data 10c is acquired by the shape data acquisition unit 14 and stored in the memory unit 10.

Figure 6 is a diagram showing an example of a container. As shown in Figure 6, the container 5 is a rectangular box in plan view, and has a rectangular frame-shaped top surface 501. The container shape data 10c indicates the width W, depth D, height H, side wall thickness T, depth H2, etc. of the container 5.

One or more marks 502 may be formed on the top surface 501. In the example shown in FIG. 6, the marks 502 are formed at two positions symmetrical with respect to the center of the container 5. When one or more marks 502 are formed on the top surface 501, the container shape data 10c may further indicate the relative relationship between the positions of the corners 504-507 of the top surface 501 and the equation of a straight line including the position of the mark 502 and the edge 508 (or edge 510) near the mark 502.

Container 6 may have the same shape as container 5, or may have a different shape from container 5. If container 6 has the same shape as container 5, container shape data 10c indicates the three-dimensional shape common to containers 5 and 6. If container 6 has a different shape from container 5, container shape data 10c indicates the three-dimensional shapes of each of containers 5 and 6.

The first initial information 10d indicates the initial position and orientation of the container 5. Specifically, the first initial information 10d indicates the range occupied by the container 5 in its initial position and orientation, for example, in the space of the robot base coordinate system. The first initial information 10d is generated by the information acquisition unit 11 and stored in the memory unit 10.

The second initial information 10e indicates the initial position and orientation of the container 6. Specifically, the second initial information 10e indicates the range occupied by the container 6 in its initial position and orientation, for example, in the space of the robot base coordinate system. The second initial information 10e is generated by the information acquisition unit 11 and stored in the memory unit 10.

The gripping point data 10f indicates the relative position and orientation of the end effector 305 with respect to the item 2 when picking up the item 2. The gripping point data 10f is created in advance using, for example, a model of the item 2 and a model of the end effector 305, and is stored in the memory unit 10.

The home position data 10g indicates the home position of the end effector 305. The home position is expressed, for example, in a robot base coordinate system. The home position is, for example, above the center of the container 5 when the container 5 is placed in an ideal location, and is a position within the container 5 where the range R within which the position and orientation can be measured using first mode imaging is within the container 5. The home position data 10g is created in advance according to the ideal location of the container 5 and stored in the memory unit 10.

(shape data acquisition unit)
The shape data acquisition unit 14 acquires the container shape data 10c according to, for example, one of the following acquisition methods a to e.

[Acquisition method a] The shape data acquisition unit 14 displays a screen on the display 160 prompting input of the shape of the containers 5 and 6 (e.g., width W, depth D, height H, side wall thickness T, depth H2, etc.), and generates container shape data 10c in response to the input to the input device 150.

[Acquisition method b] The shape data acquisition unit 14 acquires CAD data indicating the three-dimensional shape of the containers 5 and 6 from an external device, analyzes the CAD data, and generates container shape data 10c.

[Acquisition method c] An RFID tag that stores container shape data 10c is attached to the containers 5 and 6. The shape data acquisition unit 14 acquires the container shape data 10c from the RFID tag via an RFID reader.

[Acquisition method d] A barcode associated with container shape data 10c is printed on the containers 5 and 6. The shape data acquisition unit 14 stores the barcode and container shape data 10c associated with each type of container 5 and 6. The shape data acquisition unit 14 acquires the container shape data 10c corresponding to the barcode read by the barcode reader.

[Acquisition method e] The position of the camera 200 is manually adjusted so that the outer surface of the side wall of the container 5 is included in the field of view. The height H of the container 5 is obtained by 3D measurement using the image obtained by imaging in the first mode. The position of the camera 200 is also manually adjusted so that the inner surface of the side wall of the container 5 is included in the field of view. The depth H2 of the container 5 is obtained by 3D measurement using the image obtained by imaging in the first mode. The position of the camera 200 is also manually adjusted so that the top surface 501 of the container 5 is included in the field of view. The width W, depth D, and side wall thickness T of the container 5 are obtained by 2D images obtained by imaging in the second mode. Similarly, the height H, width W, depth D, side wall thickness T, and depth H2 of the container 6 are obtained.

(Image acquisition unit)
The image acquisition unit 12 outputs an image capture instruction to the camera 200 and acquires an image from the camera 200. The image acquisition unit 12 outputs an image capture instruction in the second mode when identifying the initial positions and orientations of the containers 5 and 6. The image acquisition unit 12 outputs an image capture instruction in the first mode when measuring the position and orientation of the item 2.

Furthermore, after pick-and-place begins, the image acquisition unit 12 outputs an image capture instruction for the second mode in response to an instruction from the detection unit 13, and acquires a first target image and a second target image that each show at least a portion of the containers 5 and 6.

(Information acquisition department)
The information acquisition unit 11 acquires first initial information 10d indicating the initial position and orientation of the container 5 and second initial information 10e indicating the initial position and orientation of the container 6. As shown in Fig. 5, the information acquisition unit 11 includes an identification unit 11a and a search unit 11b.

(Specific section)
The identification unit 11a uses the 2D image obtained by imaging in the second mode and the container shape data 10c to identify the position and orientation of the container 5. As will be described later, it is preferable that the 2D image be obtained by imaging when the optical axis of the imaging unit 22 of the camera 200 is parallel to the vertical direction (in other words, the optical axis of the imaging unit 22 is perpendicular to the top surface 501 of the container 5).

Figure 7 is a diagram illustrating an example of a method for identifying the position and orientation of a container. The identification unit 11a identifies a virtual surface 90 including the top surface 501 of the container 5 based on the height H indicated by the container shape data 10c. The virtual surface 90 is expressed in the robot base coordinate system. The identification unit 11a stores in advance the position of the surface on which the container 5 is placed (such as the top surface of a desk or floor), and identifies the virtual surface 90 based on the position of that surface and the height H of the container 5.

The identification unit 11a calculates the coordinates of points on the virtual surface 90 that appear in the upper right pixel 81, the lower right pixel 82, the lower left pixel 83, and the upper left pixel 84 of the 2D image 80, based on the position and orientation of the camera 200 and the viewing angle ^{θ of the camera 200. The identification unit 11a may calculate the position and orientation of the camera 200 using the position and orientation of the flange plate 303 when the 2D image 80 is captured and the coordinate transformation matrix FHC} _indicated by the calibration data 10a. The identification unit 11a stores the viewing angle θ that is set in advance in accordance with the specifications of the camera 200. The coordinates of the points on the virtual surface 90 that appear in the upper right pixel 81, the lower right pixel 82, the lower left pixel 83, and the upper left pixel 84 are expressed, for example, in the robot base coordinate system.

The identification unit 11a identifies multiple pixels in the 2D image 80 that depict multiple characteristic portions of the container 5. For example, the identification unit 11a identifies pixels 85-88 in the 2D image 80 that depict corners (characteristic portions) of the outer contour of the container 5. Note that, because the 2D image 80 may be blurred, the identification unit 11a may identify pixels 85-88 after performing known image processing such as contour enhancement. Pixels 85-88 depict one of the corners 504-507 (see Figure 6) of the top surface 501 of the container 5. In other words, pixels 85-88 depict one of the corners 504-507 of the container 5 located on the virtual surface 90. For each of pixels 85-88, the identification unit 11a identifies the coordinates of the point on the virtual surface 90 that is depicted in that pixel based on its positional relationship with the upper right pixel 81, lower right pixel 82, lower left pixel 83, and upper left pixel 84 in the 2D image 80. The coordinates are expressed, for example, in the robot base coordinate system. The coordinates of the four identified points indicate the positions of the four corners of the top surface 501 of the container 5.

The identification unit 11a generates first initial information 10d indicating the range occupied by the container 5 in the space of the robot base coordinate system based on the coordinates of the four identified points and the container shape data 10c, and stores the generated first initial information 10d in the memory unit 10. In this way, the identification unit 11a identifies the position and orientation of the container 5.

When the container shape data 10c indicates the width W and depth D of the container 5, the identification unit 11a can identify the position and orientation of the container 5 based on at least two pixels in the 2D image 80 that capture the corners of the outer contour of the container 5.

For example, the position and orientation of the container 5 is determined based on pixels 85 and 86 as follows: The determination unit 11a calculates the distance between two points on the virtual surface 90 that are reflected in pixels 85 and 86, respectively, based on the coordinates of the two points (robot base coordinate system).

If the calculated distance is within W ± the measurement error range, the identification unit 11a determines that the line connecting the two points on the virtual surface 90 that are captured by pixels 85 and 86 is parallel to the width direction of the container 5. The identification unit 11a identifies a line on the virtual surface 90 that passes through one of the two points and is perpendicular to the line connecting the two points. The identification unit 11a determines a point on the virtual surface 90 that is on the identified line and is a distance to the left of the one point equal to the depth D as one corner of the top surface 501 of the container 5. The identification unit 11a then identifies a line on the virtual surface 90 that passes through the other of the two points and is perpendicular to the line connecting the two points. The identification unit 11a determines a point on the virtual surface 90 that is on the identified line and is a distance to the left of the other point equal to the width W as one corner of the top surface 501 of the container 5.

In this way, the identification unit 11a can identify the position and orientation of the container 5 based on at least two pixels in the 2D image 80 that show the corners of the container 5's outer contour. Therefore, by using a 2D image that shows at least two of the corners 504 to 507 (see Figure 6) of the container 5's top surface 501, the identification unit 11a can successfully identify the position and orientation of the container 5.

Figure 8 is a diagram illustrating another example of a method for identifying the position and orientation of a container. The identification unit 11a identifies a pixel group 89a in the 2D image 80 that reflects a mark 502 (characteristic portion) formed on the container 5. The mark 502 is formed on the top surface 501 of the container 5, and is therefore located on the virtual surface 90. Therefore, the identification unit 11a can identify the center coordinates (robot base coordinate system) of the mark 502 reflected in the pixel group 89a from the positional relationship of the pixel group 89a with the upper right pixel 81, lower right pixel 82, lower left pixel 83, and upper left pixel 84 in the 2D image 80.

Furthermore, the identification unit 11a identifies pixel group 89b in the 2D image 80, which is near pixel group 89a and which captures an edge line (characteristic portion) that forms the boundary between the container 5 and the background. Referring to Figure 6, pixel group 89b captures an edge 510 near mark 502. Edge 510 is a line at the edge of the top surface 501 of the container 5, and is therefore located on the virtual surface 90. Therefore, the identification unit 11a can identify the equation (expressed in the robot base coordinate system) of the straight line that includes edge 510 captured in pixel group 89a, based on the positional relationship of pixel group 89b with respect to the upper right pixel 81, lower right pixel 82, lower left pixel 83, and upper left pixel 84 in the 2D image 80.

The identification unit 11a references the container shape data 10c and reads out the relative relationship between the positions of the corners 504 to 507 of the top surface 501 and the position of the mark 502 and the equation of a straight line including the edge 510 near the mark 502. The identification unit 11a applies the identified position of the mark 502 and the equation of a straight line including the edge 510 to the read-out correlation, thereby identifying the coordinates of the corners 504 to 507 of the container 5. These coordinates are expressed, for example, in the robot base coordinate system.

The identification unit 11a generates first initial information 10d indicating the range occupied by the container 5 in the space of the robot base coordinate system based on the coordinates of the identified four corners 504 to 507 and the container shape data 10c, and stores the generated first initial information 10d in the storage unit 10. In this way, the identification unit 11a identifies the position and orientation of the container 5.

In this way, the position and orientation of the container 5 is determined based on the mark 502 of the container 5 and the edge 510 near the mark 502, which appear in the 2D image 80. Therefore, by using a 2D image that shows the mark 502 of the container 5 and the edge 510 (or edge 508) near it, the determination unit 11a can successfully determine the position and orientation of the container 5.

Similarly, the identification unit 11a identifies the position and orientation of the container 6 using the 2D image obtained by imaging in the second mode and the container shape data 10c. The identification unit 11a generates second initial information 10e indicating the range occupied by the container 6 in the identified position and orientation, and stores the generated second initial information 10e in the memory unit 10.

(Search Department)
The search unit 11b searches for the position of the camera 200 when the identification unit 11a successfully identifies the position and orientation of each of the containers 5 and 6.

The search unit 11b initiates a search in response to a failure to identify the position and orientation of a container based on 2D images acquired from the camera 200 positioned at the reference position. A reference position is determined in advance for each of the containers 5 and 6. For example, the first reference position corresponding to container 5 is determined above the center of container 5 in its ideal position and orientation. The second reference position corresponding to container 6 is determined above the center of container 6 in its ideal position and orientation. The reference positions are determined so that the optical axis of the imaging unit 22 of the camera 200 is vertical.

Figure 9 is a diagram illustrating an example of the camera position search process. Figure 9 shows the method of the camera 200 position search process when the position and orientation of the container 5 are successfully identified.

As shown in the lower left of Figure 9, the search unit 11b moves the camera 200 upward from the first reference position P0 to search for the position and orientation of the camera 200 when identification by the identification unit 11a is successful. Specifically, after moving the camera 200 upward a predetermined distance, the search unit 11b outputs an image capture instruction for the second mode to the camera 200 and executes a first control process to confirm whether or not the position and orientation of the container 5 has been successfully identified based on the 2D image received from the camera 200. If the position and orientation of the container 5 has been successfully identified, the search unit 11b ends the search. If the position and orientation of the container 5 has not been successfully identified, the search unit 11b repeats the first control process until the position and orientation of the container 5 has been successfully identified.

Also, as shown in the lower right of FIG. 9 , the search unit 11b may search for the position and orientation of the camera 200 when the identification by the identification unit 11a is successful by moving the camera 200 horizontally from the first reference position P0. Specifically, the search unit 11b moves the camera 200 horizontally a predetermined distance, then outputs an image capture instruction for the second mode to the camera 200 and executes a second control process to confirm whether the position and orientation of the container 5 have been successfully identified based on the 2D image received from the camera 200. If the position and orientation of the container 5 have been successfully identified, the search unit 11b terminates the search. If the position and orientation of the container 5 have not been successfully identified, the search unit 11b repeats the second control process until the position and orientation of the container 5 have been successfully identified.

If the position and orientation of the container 5 are not successfully identified even after the camera 200 reaches its limit, the search unit 11b will issue an error notification.

Using a similar method, the search unit 11b searches for the position of the camera 200 when the position and orientation of the container 6 are successfully identified.

(Measurement section)
The measurement unit 15 measures the position and orientation of the item 2 by performing a three-dimensional measurement process that measures the three-dimensional shape of the subject based on the image acquired from the camera 200 and a detection process that detects the shape of the item 2 from the measured three-dimensional shape. As shown in Fig. 5, the measurement unit 15 includes a point cloud data generation unit 15a that performs the three-dimensional measurement process and a workpiece detection unit 15b that performs the detection process.

The point cloud data generator 15a performs three-dimensional measurement of the field of view of the camera 200 based on the image, and generates three-dimensional point cloud data. The three-dimensional point cloud data indicates the three-dimensional coordinates of each point on the subject surface (measurement surface) that exists in the field of view of the camera 200. In this embodiment, the point cloud data generator 15a uses, for example, structured illumination as the three-dimensional measurement process.

The workpiece detection unit 15b detects the position and orientation of the item 2 by a three-dimensional search. Specifically, the workpiece detection unit 15b compares multiple template data 10b stored in the memory unit 10 with the three-dimensional point cloud data, and searches the three-dimensional point cloud data for data similar to the templates. The searched data corresponds to data for the portion where the item 2 is located. The workpiece detection unit 15b detects the position and orientation of the item 2 based on the data similar to the template that has been searched for in the three-dimensional point cloud data. The position and orientation of the item 2 detected by the workpiece detection unit 15b is expressed in the camera coordinate system.

The workpiece detection unit 15b may detect the position and orientation of the item 2 using a known detection algorithm. Specifically, the workpiece detection unit 15b calculates the correlation value between the three-dimensional point cloud data and the template data 10b, and determines that the item 2 is present if the calculated correlation value is equal to or greater than a predetermined threshold. The workpiece detection unit 15b then detects the position and orientation of the item 2 according to the template data 10b with the highest correlation value.

(Settings section)
The setting unit 16 determines one of the one or more articles 2 whose position and orientation have been measured by the measurement unit 15 as the target, and sets a path for the end effector 305 to pick and place the target. The paths set by the setting unit 16 include a first path from the home position to a position where the target is picked (hereinafter referred to as the "picking position"), a second path from the picking position to a position where the target is placed (hereinafter referred to as the "place position"), and a third path returning from the place position to the home position.

The setting unit 16 selects one item 2 from one or more items 2 whose position and orientation have been identified by the measurement unit 15. The setting unit 16 calculates the position and orientation of the end effector 305 when grasping the item 2 based on the position and orientation of the selected item 2 and the gripping point data 10f. The setting unit 16 determines whether the end effector 305 and the articulated arm 302 supporting the end effector 305 in the calculated position and orientation will interfere with the container 5 based on the first initial information 10d. At this time, the setting unit 16 may also determine whether the end effector 305 and the articulated arm 302 will interfere with another item 2. If interference occurs, the setting unit 16 selects another item 2 from one or more items 2 whose position and orientation have been identified by the measurement unit 15. If no interference occurs, the setting unit 16 determines the selected item 2 as the object to be picked. The setting unit 16 then determines the position of the end effector 305 when grasping the object as the picking position.

The setting unit 16 determines a first path from the home position to the picking position. At this time, the setting unit 16 determines the first path based on the first initial information 10d so as to avoid interference between the container 5 and the end effector 305.

The setting unit 16 stores the position and orientation of each item 2 placed in the empty container 6 since the container 6 was installed. The setting unit 16 determines the position and orientation of the object to be placed in the container 6 based on the second initial information 10e and the position and orientation of each item 2 placed in the container 6. For example, when placing an item 2 in the container 6 according to predetermined rules, the setting unit 16 determines, as the position and orientation of the object to be placed, a position and orientation that differs by a predetermined offset amount from the position and orientation of the item 2 placed previously. The setting unit 16 calculates the position and orientation of the end effector 305 for placing the object with the determined position and orientation. The setting unit 16 then determines, as the placement position, the position of the end effector 305 when releasing the object.

The setting unit 16 determines a second path from the end point of the first path to the place position. At this time, the setting unit 16 determines the second path based on the first initial information 10d and the second initial information 10e so as to avoid interference between the containers 5 and 6 and the end effector 305.

The setting unit 16 determines a third path from the end point of the second path to the home position. At this time, the setting unit 16 determines the third path based on the first initial information 10d and the second initial information 10e so as to avoid interference between the containers 5 and 6 and the end effector 305.

Furthermore, the setting unit 16 sets the second route or the third route so as to satisfy the following conditions A and B.
Condition A: The characteristic portion of the container 5 having the initial position and orientation indicated by the first initial information 10d includes a first passing point corresponding to the position of the end effector when it is included within the field of view of the camera 200.
Condition B: The characteristic portion of the container 6 having the initial position and orientation indicated by the second initial information 10e includes a second passing point corresponding to the position of the end effector when it is included within the field of view of the camera 200.

Characteristic portions of container 5 include, for example, at least one of the following: a corner of container 5, an edge between the top surface and side surface of container 5, and a mark formed on the top surface of container 5. Characteristic portions of container 6 include, for example, at least one of the following: a corner of container 6, an edge between the top surface and side surface of container 6, and a mark formed on the top surface of container 6.

(Command Department)
The command unit 17 generates a movement command for each control period in accordance with the first path set by the setting unit 16, and outputs the generated movement command to the robot control device 400. As a result, the end effector 305 moves from the home position to the picking position.

The command unit 17 outputs a grip command to the robot control device 400 when the end effector 305 reaches the end point of the first path. This causes the end effector 305 to grip the target object at the picking position.

The command unit 17 generates a movement command for each control cycle according to the second path set by the setting unit 16, and outputs the generated movement command to the robot control device 400. As a result, the end effector 305 moves from the picking position to the placing position while gripping the target object.

The command unit 17 outputs a release command to the robot control device 400 when the end effector 305 reaches the end point of the second path. This causes the end effector 305 to release the target object at the place position.

The command unit 17 generates a movement command for each control cycle according to the third path set by the setting unit 16, and outputs the generated movement command to the robot control device 400. As a result, the end effector 305 moves from the place position to the home position.

(Detection unit)
The detection unit 13 detects changes in the position and orientation of the containers 5 and 6 after pick-and-place is started.

When the end effector 305, which is performing a series of pick-and-place operations, reaches the first passing point, the detection unit 13 outputs an instruction to acquire a first target image to the image acquisition unit 12. In response to receiving the acquisition instruction, the image acquisition unit 12 outputs an instruction to capture an image in the second mode to the camera 200 and acquires the first target image. The detection unit 13 detects changes in the position and orientation of the container 5 based on the first target image and the first initial information 10d.

As described above, the first passing point corresponds to the position of the end effector when a characteristic portion of the container 5 having the initial position and orientation indicated by the first initial information 10d is included within the field of view of the camera 200. The characteristic portion of the container 5 includes, for example, at least one of the corners of the container 5, the edges between the top and side surfaces of the container 5, and a mark formed on the top surface of the container 5.

Figure 10 is a diagram showing an example of a target image when the position and orientation of the container has not changed. Figure 11 is a diagram showing an example of a target image when the position and orientation of the container has changed. Figures 10 and 11 show a first target image 70 when the edge between the top surface and side surface of the container 5 has been set as a characteristic part. As shown in Figures 10 and 11, the edge 511 between the top surface and side surface of the container 5 is captured in the first target image 70.

The detection unit 13 extracts a group of pixels in which the edge 511 appears from the first target image 70. For example, the detection unit 13 extracts a group of pixels in which the edge 511 appears by performing edge extraction processing.

Furthermore, the detection unit 13 places a container model 5M in the virtual space in the same position and orientation as the initial position and orientation indicated by the first initial information 10d. The container model 5M is created based on the shape of the container 5 indicated by the container shape data 10c. Furthermore, the detection unit 13 places a virtual camera in the virtual space in the same position and orientation as the camera 200 when the end effector 305 passes through the first pass point. The detection unit 13 creates a virtual image 70M when the container model is captured by the virtual camera. The imaging conditions of the virtual camera (field of view, resolution, etc.) are the same as those of the camera 200. The detection unit 13 extracts a group of pixels in the virtual image 70M that captures the edge 511M of the container model 5M.

The detection unit 13 calculates the difference between the coordinates of the pixel group in which the edge 511 appears in the first object image 70 and the coordinates of the pixel group in which the edge 511M appears in the virtual image 70M. For example, the difference is calculated as the sum of the differences in the Y coordinates of pixels having the same X coordinate. If the calculated difference is equal to or less than a threshold, the detection unit 13 determines that the position and orientation of the container 5 has not changed (see FIG. 10), and if the calculated difference exceeds the threshold, the detection unit 13 determines that the position and orientation of the container 5 has changed (see FIG. 11).

Figure 12 is a diagram showing another example of a target image when the position and orientation of the container has not changed. Figure 13 is a diagram showing another example of a target image when the position and orientation of the container has changed. Figures 12 and 13 show a first target image 70 when the corner 506 of the container 5 has been set as a characteristic part. As shown in Figures 12 and 13, the corner 506 of the container 5 appears in the first target image 70.

The detection unit 13 extracts pixels in which the corner 506 appears from the first target image 70. For example, the detection unit 13 extracts pixels in which the corner 506 appears by performing edge extraction processing. Furthermore, the detection unit 13 identifies two unit vectors v1 and v2 that are respectively along two edges 509 and 510 that include the corner 506.

The detection unit 13 places a container model 5M in virtual space with the same position and orientation as the initial position and orientation indicated by the first initial information 10d. The container model 5M is created based on the shape of the container 5 indicated by the container shape data 10c. Furthermore, the detection unit 13 places a virtual camera in the virtual space with the same position and orientation as the camera 200 when the end effector 305 passes through the first pass point. The detection unit 13 creates a virtual image 70M when the container model is captured by the virtual camera. The imaging conditions of the virtual camera (field of view, resolution, etc.) are the same as those of the camera 200. In the virtual image 70M, the detection unit 13 identifies the pixel that captures the corner 506M of the container model 5M and two unit vectors v1M and v2M that are respectively along the two edges 509M and 510M that include the corner 506M.

The detection unit 13 calculates a first difference between the coordinates of the pixel where the corner 506 appears in the first object image 70 and the coordinates of the pixel where the corner 506M appears in the virtual image 70M. The first difference indicates the sum of the square of the difference in the X coordinates and the square of the difference in the Y coordinates. Furthermore, the detection unit 13 calculates a second difference between the unit vectors v1 and v2 and the unit vectors v1M and v2M. The second difference indicates, for example, a representative value (average value, maximum value, etc.) of the angle between the unit vectors v1 and v1M and the angle between the unit vectors v2 and v2M.

The detection unit 13 determines that the position and orientation of the container 5 have not changed if the first difference is less than the first threshold and the second difference is less than the second threshold (see FIG. 12). The detection unit 13 determines that the position and orientation of the container 5 have changed if the first difference exceeds the first threshold or if the second difference exceeds the second threshold (see FIG. 13). Note that the detection unit 13 may determine that the change in the position and orientation of the container 5 has been caused by translational movement if the first difference exceeds the first threshold, and may determine that the change in the position and orientation of the container 5 has been caused by rotational movement if the second difference exceeds the second threshold.

Figure 14 is a diagram showing yet another example of a target image when the position and orientation of the container has not changed. Figure 15 is a diagram showing yet another example of a target image when the position and orientation of the container has changed. Figures 14 and 15 show a first target image 70 when a mark 502 formed on the top surface of the container 5 is set as a characteristic part. As shown in Figures 14 and 15, the mark 502 on the container 5 appears in the first target image 70.

The detection unit 13 extracts pixels in which the mark 502 appears from the first target image 70. For example, the detection unit 13 extracts pixels in which the mark 502 appears by performing a known matching process. Furthermore, the detection unit 13 identifies a unit vector v3 along the edge 510 near the mark 502.

The detection unit 13 places a container model 5M in the virtual space with the same position and orientation as the initial position and orientation indicated by the first initial information 10d. The container model 5M is created based on the shape of the container 5 indicated by the container shape data 10c. Furthermore, the detection unit 13 places a virtual camera in the virtual space with the same position and orientation as the camera 200 when the end effector 305 passes through the first pass point. The detection unit 13 creates a virtual image 70M when the container model 5M is imaged by the virtual camera. The imaging conditions of the virtual camera (field of view, resolution, etc.) are the same as those of the camera 200. The detection unit 13 identifies, in the virtual image 70M, the pixel where the mark 502M of the container model 5M appears and the unit vector v3M along the edge 510M near the mark 502M.

The detection unit 13 calculates a third difference between the coordinates of the pixel where the mark 502 appears in the first object image 70 and the coordinates of the pixel where the mark 502M appears in the virtual image 70M. The third difference indicates, for example, the sum of the square of the difference in the X coordinates and the square of the difference in the Y coordinates. Furthermore, the detection unit 13 calculates a fourth difference between the unit vector v3 and the unit vector v3M. The fourth difference indicates, for example, the angle between the unit vector v3 and the unit vector v3M.

The detection unit 13 determines that the position and orientation of the container 5 have not changed if the third difference is less than the third threshold and the fourth difference is less than the fourth threshold (see FIG. 14). The detection unit 13 determines that the position and orientation of the container 5 have changed if the third difference exceeds the third threshold or if the fourth difference exceeds the fourth threshold (see FIG. 15). Note that the detection unit 13 may determine that the change in the position and orientation of the container 5 has been caused by translational movement if the third difference exceeds the third threshold, and may determine that the change in the position and orientation of the container 5 has been caused by rotational movement if the fourth difference exceeds the fourth threshold.

The detection unit 13 can detect changes in the position and orientation of container 6 based on the second target image using the same method as for container 5.

<Processing flow of image processing device>
16 is a flowchart showing an example of the overall flow of processing by the image processing device. First, the CPU 101 of the image processing device 100 acquires first initial information 10d and second initial information 10e indicating the initial positions and orientations of the containers 5 and 6, respectively (step S1).

Next, the CPU 101 determines the first area within the container 5 as the search range for items to be picked (step S2).

Next, the CPU 101 outputs a command to move to the home position to the robot control device 400, and then outputs an image capture command in the first mode to the camera 200. Based on the image received from the camera 200, the CPU 101 performs 3D measurement of the search range for the picking target and detects the position and orientation of one or more items 2 present in the container 5 (step S3).

Next, based on the 3D measurement results, the CPU 101 sets a path for the end effector 305 to pick up a target item from one or more items 2 in the container 5 and place it in a second area in the container 6 (step S4).

Next, the CPU 101 generates commands for the pick-and-place operation according to the set path and outputs the generated commands to the robot control device 400 (step S5).

Next, the CPU 101 detects changes in the position and orientation of the containers 5 and 6 based on the first and second target images obtained during the pick-and-place operation (step S6).

Next, the CPU 101 determines whether there are any remaining items 2 in the container 5 (step S7). For example, if the positions and orientations of two or more items 2 were detected in the most recent step S3, the CPU 101 determines YES in step S7.

If the answer is YES in step S7, the CPU 101 returns the process to step S3. If the answer is NO in step S7, the CPU 101 ends the process.

<Subroutine of Step S1>
FIG. 17 is a flowchart showing the flow of the subroutine processing of step S1 shown in FIG.

The CPU 101 generates a movement command to move the camera 200 to the first reference position, outputs the generated movement command to the robot control device 400, and then outputs an instruction to capture images in the second mode to the camera 200 (step S11). This causes the camera 200 to capture images in the second mode at the first reference position and output the 2D images obtained by the capture to the image processing device 100.

The CPU 101 determines whether the container 5 is visible in the 2D image to the extent that the position and orientation of the container 5 can be determined (step S12). For example, the CPU 101 determines YES in step S12 if it can identify pixels from the 2D image that show at least two of the corners 504 to 507 of the top surface 501 of the container 5. Alternatively, the CPU 101 determines YES in step S12 if it can identify the mark 502 and its nearby edge from the 2D image.

If the result of step S12 is NO, the CPU 101 determines whether the camera 200 has been moved to its upper limit (step S13). If the result of step S13 is NO, the CPU 101 executes first movement control, generating a movement command to move the camera 200 upward by a predetermined distance and outputting the generated movement command to the robot control device 400 (step S4).

If step S13 is YES, the CPU 101 determines whether the camera 200 has moved to its lateral limit (step S15). If step S15 is NO, the CPU 101 generates a movement command to move the camera 200 a predetermined distance in the lateral direction (any direction parallel to the horizontal plane) and executes second movement control to output the generated movement command to the robot control device 400 (step S16). That is, the CPU 101 executes second movement control if the position and orientation of the container 5 are not successfully identified in the first movement control. Note that in the first execution of step S16, the CPU 101 first moves the camera 200 to the first reference position and then executes second movement control.

After step S14 or step S16, the CPU 101 outputs a second mode imaging command to the camera 200 (step S17). This causes the camera 200 to perform second mode imaging and output the 2D image obtained by this imaging to the image processing device 100. After step S17, the CPU 101 returns the process to step S12.

If the answer is YES in step S15, the CPU 101 issues an error notification (step S18). After step S18, the CPU 101 terminates processing.

If the answer is YES in step S12, the CPU 101 determines the initial position and orientation of the container 5 based on the 2D image and the container shape data 10c (step S19). After step S19, the process returns to step S2 in FIG. 16.

The CPU 101 also performs processing according to the flowchart shown in Figure 17 for container 6 to determine the initial position and orientation of container 6.

<Subroutine of Step S5>
Fig. 18 is a flowchart showing an example of the processing flow of the subroutine of step S5 shown in Fig. 16. Fig. 18 shows the processing flow when the first pass point and the second pass point are included on the second route from the picking position to the placing position.

First, the CPU 101 outputs a movement command to the robot control device 400 to move the end effector 305 from the home position to a position (picking position) where the target object will be picked up (step S21).

When the end effector 305 reaches the picking position, the CPU 101 outputs a gripping command to the robot control device 400 (step S22). This causes the end effector 305 to pick up the target object.

Next, the CPU 101 outputs a movement command to the robot control device 400 to move the end effector 305 from the picking position to the position where the target object is to be placed (placement position) (step S23).

Next, the CPU 101 determines whether the end effector 305 has reached the first passing point (step S31). If the end effector 305 has not reached the first passing point (NO in step S31), the CPU 101 returns the process to step S31.

If the end effector 305 reaches the first passing point (YES in step S31), the CPU 101 outputs an image capture instruction in the second mode to the camera 200 and acquires the first target image (step S32).

Next, the CPU 101 determines whether the end effector 305 has reached the second passing point (step S33). If the end effector 305 has not reached the second passing point (NO in step S33), the CPU 101 returns the process to step S33.

If the end effector 305 reaches the second passing point (YES in step S33), the CPU 101 outputs an image capture instruction in the second mode to the camera 200 and acquires a second target image (step S34).

Next, when the end effector 305 reaches the placement position, the CPU 101 outputs a release command to the robot control device 400 (step S24). This causes the end effector 305 to place the object into the container 6.

Next, the CPU 101 outputs a movement command to the robot control device 400 to move the end effector 305 from the place position to the home position (step S25). After step S25, the process returns to step S6 in Figure 16.

In the processing flow shown in Figure 18, the first and second target images are captured when the end effector 305 is gripping an object. Therefore, the object being gripped by the end effector 305 may appear in the first and second target images. As a result, it may not be possible to accurately detect changes in the position and orientation of the containers 5, 6 using the first and second target images. Therefore, step S5 may be performed according to the flowchart shown in Figure 19 instead of the flowchart shown in Figure 18.

Figure 19 is a flowchart showing another example of the processing flow of the subroutine of step S5 shown in Figure 16. Figure 19 shows the processing flow when a first pass point and a second pass point are included on the third route from the place position to the home position. The flowchart shown in Figure 19 differs from the flowchart shown in Figure 18 in that the processing of steps S31 to S34 is performed after step S25 rather than after step S23. As a result, the first and second target images are captured when the end effector 305 is not gripping an object. Therefore, changes in the position and orientation of the containers 5 and 6 can be detected with high accuracy using the first and second target images.

<Subroutine of Step S6>
FIG. 20 is a flowchart showing an example of the process flow of the subroutine of step S6 shown in FIG.

The CPU 101 determines whether a characteristic part of the container 5 is shown in the first target image (step S41). For example, if the characteristic part is the edge 508 between the top and side surfaces of the container 6, the CPU 101 determines whether the edge 508 of the container 5 is shown in the first target image depending on whether an edge line of a predetermined length or more is extracted from the first target image.

If the answer is YES in step S41, the CPU 101 determines whether the position and orientation of the container 5 has changed based on the first target image and the first initial information 10d (step S42).

If the answer is NO in step S41 or YES in step S42, the CPU 101 performs step S1a to acquire the initial position and orientation of the container 5. The specific processing method of step S1b is the same as that of step S1. As a result, if the position and orientation of the container 5 changes, the initial position and orientation of the container 5 is acquired again, and the first initial information 10d is updated. As a result, interference between the container 5 and the end effector 305 is avoided.

If the answer is NO in step S42, the CPU 101 determines whether a characteristic part of the container 6 is shown in the second target image (step S43). If the answer is YES in step S43, the CPU 101 determines whether the position and orientation of the container 6 have changed based on the second target image and the second initial information 10e (step S44).

If the answer is NO in step S43 or YES in step S44, the CPU 101 performs step S1b to acquire the initial position and orientation of the container 6. The specific processing method of step S1b is the same as that of step S1. As a result, if the position and orientation of the container 6 changes, the initial position and orientation of the container 6 is acquired again, and the second initial information 10e is updated. As a result, interference between the container 6 and the end effector 305 is avoided.

<Modification 1>
In the above description, when a change in the positions and orientations of the containers 5 and 6 is detected, the operation of acquiring the initial positions and orientations of the containers 5 and 6 is executed again, and the first initial information 10d and the second initial information 10e are updated. However, the positions and orientations of the containers 5 and 6 after the change may be measured using the first target image and the second target image, and the first initial information 10d and the second initial information 10e may be updated in accordance with the measured positions and orientations.

Figure 21 is a block diagram showing an example of the functional configuration of an image processing device according to Modification 1. As shown in Figure 21, image processing device 100A according to Modification 1 differs from image processing device 100 shown in Figure 5 in that it further includes an update unit 18. The update unit 18 is realized by CPU 101 shown in Figure 3 executing a program stored on hard disk 103.

The update unit 18 identifies the position and orientation of the container 5 using a first target image obtained by imaging with the camera 200 when the end effector 305 reaches the first passing point, and updates the first initial information 10d according to the identified position and orientation. Similarly, the update unit 18 identifies the position and orientation of the container 6 using a second target image obtained by imaging with the camera 200 when the end effector 305 reaches the second passing point, and updates the second initial information 10e according to the identified position and orientation.

When a target image showing a mark formed on the top surface of a container is acquired (see Figures 14 and 15), the update unit 18 identifies the position and orientation of the container from the target image using a method similar to that used by the identification unit 11a.

Specifically, the update unit 18 identifies the position of the mark 502 and the equation of a line including the edge 510 based on the coordinates of the pixel in which the mark 502 appears and the pixel group in which the edge 510 near the mark 502 appears in the first target image. The update unit 18 references the container shape data 10c and reads the correlation between the positions of the corners 504 to 507 of the container 5 and the equation of a line including the position of the mark 502 and the edge 510 near the mark 502. The update unit 18 identifies the coordinates of the corners 504 to 507 of the container 5 by applying the identified position of the mark 502 and the equation of a line including the edge 510 to the read correlation. These coordinates are expressed, for example, in the robot base coordinate system. The update unit 18 calculates the range occupied by the container 5 in the space of the robot base coordinate system based on the coordinates of the identified four corners 504 to 507 and the container shape data 10c, and updates the first initial information 10d to indicate this range.

The update unit 18 uses a similar method to identify the position and orientation of the container 6 from a second target image that shows the mark formed on the top surface of the container 6, and updates the second initial information 10e according to the identified position and orientation.

When a target image showing the edges of a container is acquired (see Figures 10 and 11), the position and orientation of the container cannot be measured from the target image alone. Therefore, the update unit 18 outputs an instruction to acquire auxiliary images to the image acquisition unit 12, and acquires auxiliary images (first auxiliary image and second auxiliary image). The auxiliary images show edges of the container that are perpendicular to the edges shown in the target image.

For example, the first auxiliary image 75 is acquired by capturing an image with the camera 200 at a position where the edge 508 of the container 5 having the initial position and orientation indicated by the first initial information 10d is included in the field of view. Therefore, the image acquisition unit 12 generates a movement command for the end effector 305 to the position when the edge 508 of the container 5 having the initial position and orientation is included in the field of view of the camera 200, and outputs the generated movement command to the robot control device 400. After the robot 300 has moved, the image acquisition unit 12 outputs an image capture command in the second mode to the camera 200 and acquires the first auxiliary image.

Figure 22 shows the relationship between the target image and the auxiliary image. As shown in Figure 22, the first target image 70 shows the edge 511 of the container 5, and the first auxiliary image 75 shows the edge 508 of the container 5.

The update unit 18 simply measures the changed position and orientation of the container 5 based on the first target image 70 and the first auxiliary image 75.

Specifically, the update unit 18 extracts a group of pixels in which the edge 511 appears in the first target image 70, and calculates an equation (hereinafter referred to as the "first line equation") of a straight line (hereinafter referred to as the "first line 520") containing the edge 511 based on the coordinates of the extracted pixel group. The first line equation indicates the relationship between the X and Y coordinates on a virtual surface 90 (see FIG. 7) containing the top surface of the container 5 in the robot base coordinate system. Similarly, the update unit 18 extracts a group of pixels in which the edge 508 appears in the first auxiliary image 75, and calculates an equation (hereinafter referred to as the "second line equation") of a straight line (hereinafter referred to as the "second line 521") containing the edge 508 based on the coordinates of the extracted pixel group.

The update unit 18 calculates the coordinates of the intersection 76 between the first line 520 and the second line 521 based on the first line equation and the second line equation. The update unit 18 determines that a corner 504 of the container 5 exists at the intersection 76, and specifies the coordinates of the corner 504. Furthermore, the update unit 18 calculates the coordinates of a point 77 on the first line 520 that is a width W away from the intersection 76. The width W is indicated by the container shape data 10c. The update unit 18 determines that a corner 507 of the container 5 exists at the point 77, and specifies the coordinates of the corner 507. The update unit 18 calculates the coordinates of a point 78 on the second line 521 that is a depth D away from the intersection 76. The depth D is indicated by the container shape data 10c. The update unit 18 determines that a corner 505 of the container 5 exists at the point 78, and specifies the coordinates of the corner 505. The update unit 18 determines the coordinates of the remaining corners 506 based on the container shape data 10c.

The update unit 18 calculates the range occupied by the container 5 in the space of the robot base coordinate system based on the coordinates of the identified four corners 504 to 507 and the container shape data 10c, and updates the first initial information 10d to indicate this range.

In a similar manner, the update unit 18 acquires a second auxiliary image of the container 6, which captures edges of the container 6 that are perpendicular to the edges captured in the second target image. The update unit 18 identifies the coordinates of the four corners of the container 6 based on the second target image and the second auxiliary image. The update unit 18 calculates the range occupied by the container 6 in the space of the robot base coordinate system based on the coordinates of the identified four corners and the container shape data 10c, and updates the second initial information 10e to indicate this range.

Image processing device 100A according to variant 1 executes steps S1 to S7 shown in FIG. 16, similar to image processing device 100. However, image processing device 100A executes processing according to the flowchart shown in FIG. 23 as a subroutine of step S6, instead of the flowchart shown in FIG. 20.

Figure 23 is a flowchart showing the processing flow of the subroutine of step S6 in Variation 1. The flowchart shown in Figure 23 differs from the flowchart shown in Figure 20 in that steps S51 and S52 are executed when the answer to steps S42 and S44 is YES, respectively.

If the answer is YES in step S42, the CPU 101 of the image processing device 100A determines the position and orientation of the container 5 based on the first target image, and updates the first initial information 10d according to the determined position and orientation (step S51).

If the answer is YES in step S44, the CPU 101 of the image processing device 100A determines the position and orientation of the container 6 based on the second target image, and updates the second initial information 10e according to the determined position and orientation (step S52).

<Modification 2>
In the above description, the detection unit 13 detects changes in the positions and orientations of the containers 5 and 6 based on the first and second object images obtained by imaging in the second mode. However, the detection unit 13 may also detect changes in the positions and orientations of the containers 5 and 6 based on the first and second object images obtained by imaging in the first mode.

For example, the detection unit 13 generates three-dimensional point cloud data of the container 5 based on a first target image obtained by imaging in the first mode. The detection unit 13 identifies the surface position of the container 5 in its initial position and orientation based on the first initial information 10d. The detection unit 13 calculates the difference between the surface position indicated by the generated three-dimensional point cloud data and the surface position identified based on the first initial information 10d. The detection unit 13 determines that the position and orientation of the container 5 have not changed if the calculated difference is equal to or less than a threshold, and determines that the position and orientation of the container 5 have changed if the calculated difference exceeds the threshold.

<Modification 3>
In the above description, the first and second target images are assumed to be acquired during the pick-and-place operation. However, the image acquisition unit 12 may acquire the first and second target images at any timing after the start of the pick-and-place operation. For example, the image acquisition unit 12 may move the camera 200 to the first and second pass points and acquire the first and second target images each time a predetermined number of pick-and-place operations are completed. Alternatively, the image acquisition unit 12 may move the camera 200 to the first and second pass points and acquire the first and second target images each time a predetermined time has elapsed after the start of the pick-and-place operation.

<Modification 4>
Camera 200 is not limited to the configuration shown in FIG. 2 . For example, camera 200 may be a twin-lens stereo camera. In this case, when camera 200 receives an instruction to capture images in the first mode, it captures images according to a stereo system. Camera 200 may also be a twin-lens stereo camera with an attached projector, as disclosed in Japanese Patent Application Laid-Open No. 2019-138822, for example. In this way, camera 200 only needs to have a first mode in which it operates as a 3D camera and a second mode in which it operates as a 2D camera, and various systems may be adopted as the system for operating as a 3D camera.

§3 Supplementary Note As described above, the present embodiment includes the following disclosure.

(Configuration 1)
An image processing device (100, 100A) that processes an image of a camera (200), wherein the camera (200) is mounted on a robot (300) having an end effector (305) that picks and places an object from a first area to a second area so that the relative position of the camera (200) with respect to the end effector (305) is constant, and the image processing device (100, 100A)
a first acquisition unit (11, 101) that acquires initial information (10d, 10e) indicating an initial position and orientation of a container (5, 6) that accommodates the object in the first area or the second area;
a second acquisition unit (12, 101) that acquires a target image obtained by capturing an image of the camera (200) in a state in which at least a portion of the container (5, 6) having the initial position and orientation is included in a field of view after the start of the pick-and-place;
and a detection unit (13, 101) that detects a change in the position and orientation of the container (5, 6) based on the target image and the initial information.

(Configuration 2)
a measurement unit (15, 101) that measures the position and orientation of the object present in the first area using an image from the camera (200);
a setting unit (16) that sets a path for the end effector (305) to pick up the object from a home position, place the object in the second area, and return to the home position based on the position and orientation of the object,
the path includes waypoints corresponding to positions of the end effector (305) when a feature of the container (5, 6) having the initial position and orientation is within the field of view of the camera (200);
The image processing device (100, 100A) according to configuration 1, wherein the target image is obtained by capturing an image with the camera (200) at the timing when the end effector (305) passes through the passing point.

(Configuration 3)
The image processing device (100, 100A) according to claim 2, wherein the passing point is set during a period from when the object is placed until when the object returns to the home position.

(Configuration 4)
The containers (5, 6) are rectangular box-shaped in plan view,
The image processing device (100, 100A) according to configuration 2 or 3, wherein the characteristic portion includes at least one of a corner of the container (5, 6), a ridge between the top surface and the side surface of the container (5, 6), and a mark (502) formed on the top surface.

(Configuration 5)
The image processing device (100, 100A) described in any one of configurations 1 to 4, wherein the first acquisition unit (11, 101) re-acquires the initial information in response to a change in the position and orientation of the container (5, 6) being detected by the detection unit (13, 101).

(Configuration 6)
The image processing device (100A) according to any one of configurations 1 to 4, further comprising an update unit (18, 101) that identifies a position and orientation of the container (5, 6) based on the target image and updates the initial information in accordance with the identified position and orientation.

(Configuration 7)
A robot control system (1), comprising:
a robot (300) having an end effector (305) for picking and placing an object from a first area to a second area;
a camera (200) mounted on the robot (300) so that its relative position with respect to the end effector (305) is constant;
a first acquisition unit (11, 101) that acquires initial information indicating an initial position and orientation of a container (5, 6) that accommodates the object in the first area or the second area;
a second acquisition unit (12, 101) that acquires a target image obtained by capturing an image of the camera (200) in a state in which at least a portion of the container having the initial position and orientation is included in a field of view after the start of the pick-and-place;
A robot control system comprising: a detection unit (13, 101) that detects a change in the position and orientation of the container (5, 6) based on the target image and the initial information.

(Configuration 8)
A control method for an image processing device (100, 100A) that processes an image of a camera (200), wherein the camera (200) is mounted on a robot (300) having an end effector (305) that picks and places an object from a first area to a second area so that the relative position of the camera (200) with respect to the end effector (305) is constant, and the control method includes:
acquiring initial information indicating an initial position and orientation of a container (5, 6) that accommodates the object in the first area or the second area;
After the pick-and-place operation is started, an object image is acquired by capturing an image of the camera (200) in a state in which at least a portion of the container (5, 6) having the initial position and orientation is included in the field of view;
and detecting a change in the position and orientation of the container (5, 6) based on the target image and the initial information.

Although the embodiments of the present invention have been described, the embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the meaning and scope of the claims.

1 Robot control system, 2 Item, 5, 6 Container, 5M Container model, 10 Memory unit, 10a Calibration data, 10b Template data, 10c Container shape data, 10d First initial information, 10e Second initial information, 10f Grasping point data, 10g Home position data, 11 Information acquisition unit, 11a Identification unit, 11b Search unit, 12 Image acquisition unit, 13 Detection unit, 14 Shape data acquisition unit, 15 Measurement unit, 15a Point cloud data generation unit, 15b Workpiece detection unit, 16 Setting unit, 17 Command unit, 18 Update unit, 21 Projection unit, 22 Imaging unit, 70 First object image, 70M Virtual image, 75 First auxiliary image, 76 Intersection, 77, 78 Point, 80 2D image, 81 Upper right pixel, 82 Lower right pixel, 83 Lower left pixel, 84 Upper left pixel, 85, 86, 88 pixels, 89a, 89b pixel group, 90 virtual surface, 100, 100A image processing device, 101 CPU, 102 main memory, 103 hard disk, 104 camera interface, 104a image buffer, 105 input interface, 106 display controller, 107 communication interface, 108 data reader/writer, 109 bus, 150 input device, 160 display, 200 camera, 210 light source, 211 filter, 212, 221 optical system, 220 imaging element, 300 robot, 301 base, 302 articulated arm, 303 flange plate, 304 support member, 305 end effector, 400 robot control device, 501 upper surface, 502, 502M marks, 504 to 507, 506M Corner, 508-511, 509M-511M Edge, 520 First straight line, 700 Memory card.

Claims (7)

An image processing device that processes an image of a camera, the camera being mounted on a robot having an end effector that picks and places an object from a first area to a second area so that a relative position of the camera with respect to the end effector is constant, the image processing device comprising:
a first acquisition unit that acquires initial information indicating an initial position and orientation of a container that accommodates the object in the first area or the second area;
a second acquisition unit that acquires, after the start of the pick-and-place operation, a target image obtained by capturing an image of the camera in a state in which at least a portion of the container having the initial position and orientation is included in a field of view;
a detection unit that detects a change in the position and orientation of the container based on the target image and the initial information;
a measurement unit that measures the position and orientation of the object present in the first area using an image from the camera;
a setting unit that sets a path for the end effector to pick up the object from a home position, place the object in the second area, and return to the home position based on the position and orientation of the object,
the path includes waypoints corresponding to positions of the end effector when a feature of the container having the initial position and orientation is within a field of view of the camera;
The target image is obtained by capturing an image with the camera at the timing when the end effector passes through the passing point . The image processing device according to claim 1 , wherein the passing point is set between when the object is placed and when the object returns to the home position. The container has a rectangular box shape in a plan view,
The image processing device according to claim 1 or 2 , wherein the characteristic portion includes at least one of a corner of the container, a ridge between the top surface and a side surface of the container, and a mark formed on the top surface. The image processing device according to claim 1 , wherein the first acquisition unit reacquires the initial information in response to a change in the position and orientation of the container being detected by the detection unit. The image processing device according to claim 1 , further comprising an update unit that identifies a position and orientation of the container based on the target image, and updates the initial information in accordance with the identified position and orientation. 1. A robot control system, comprising:
a robot having an end effector that picks and places an object from a first area to a second area;
a camera mounted on the robot so that its relative position with respect to the end effector is constant;
a first acquisition unit that acquires initial information indicating an initial position and orientation of a container that accommodates the object in the first area or the second area;
a second acquisition unit that acquires, after the start of the pick-and-place operation, a target image obtained by capturing an image of the camera in a state in which at least a portion of the container having the initial position and orientation is included in a field of view;
a detection unit that detects a change in the position and orientation of the container based on the target image and the initial information;
a measurement unit that measures the position and orientation of the object present in the first area using an image from the camera;
a setting unit that sets a path for the end effector to pick up the object from a home position, place the object in the second area, and return to the home position based on the position and orientation of the object,
the path includes waypoints corresponding to positions of the end effector when a feature of the container having the initial position and orientation is within a field of view of the camera;
the target image is obtained by capturing an image with the camera at the timing when the end effector passes through the passing point . A control method for an image processing device that processes an image captured by a camera, the camera being mounted on a robot having an end effector that picks and places an object from a first area to a second area so that the relative position of the camera to the end effector is constant, the control method comprising:
acquiring initial information indicating an initial position and orientation of a container that accommodates the object in the first area or the second area;
acquiring an object image obtained by capturing an image of the object with the camera after the start of the pick-and-place operation, in a state in which at least a portion of the container having the initial position and orientation is included in a field of view;
detecting a change in the position and orientation of the container based on the target image and the initial information;
measuring the position and orientation of the object present in the first area using an image from the camera;
and a step of picking up the object from a home position, placing the object in the second area, and setting a path for the end effector to return to the home position based on the position and orientation of the object;
the path includes waypoints corresponding to positions of the end effector when a feature of the container having the initial position and orientation is within a field of view of the camera;
A control method , wherein the target image is obtained by capturing an image with the camera at the timing when the end effector passes through the passing point .