JP6022331B2 - Camera system - Google Patents

Description

  The present invention automatically recognizes an irradiation point formed by irradiation from a light source from an image taken by a camera as a reference point used for calibration of various camera parameters between a plurality of cameras and recognition of the shape of a photographed object. More particularly, the present invention relates to a camera system that enables an operator to easily set an irradiation point in a real space.

  A camera system for tracking a person using a surveillance camera has been studied. When constructing such a camera system, it is necessary to obtain camera parameters such as the position and orientation of the camera and to adjust the correspondence between the virtual space recognized by the camera system and the monitoring space that is the real space. This adjustment work is called calibration.

  For example, in order to perform calibration, a plurality of reference points serving as reference points for position measurement are set in a monitoring space within the camera field of view, and the reference points are photographed by the camera. Then, each reference point is detected from the photographed image, and camera parameters are measured from the geometric relationship between the detection position of the reference point on the image and the reference point position in the monitoring space.

  Camera parameters such as camera position and orientation can be measured with a single camera if the 3D coordinates of the reference point are given, but can be measured without giving the 3D coordinates of the reference point if a common field of view of multiple cameras is used. it can. That is, a geometrical feature in which a plurality of reference points are set in a common field of view and the same reference point is detected from each image captured by a plurality of cameras sharing the field of view, and the detection positions of these reference points are the same position. It is possible to measure the camera parameters of each camera by using the relationship.

  In addition to such calibration, the reference point is also used for measuring a three-dimensional position / shape of a fixture or the like installed in the monitoring space. For example, a plurality of reference points are set on the surface of a fixture, etc., each reference point is detected from an image captured by a camera, and the geometric relationship between each reference point and the camera position / posture is applied to the surface of the fixture, etc. By obtaining the three-dimensional position of each installed reference point, the position or shape of a fixture or the like may be measured.

  Conventionally, in order to install such a reference point, there has been proposed an image processing apparatus that irradiates spot light from a light emitting device onto a monitoring space and realizes camera calibration using the irradiated point as a reference point (Patent Document 1). .

JP 2004-46772 A

  By the way, in response to the recent drop in camera prices, a camera system has been proposed in which a large number of surveillance cameras are installed to monitor a wide surveillance space. In such a case, it becomes difficult to irradiate the spot light by using the light emitting device from outside the monitoring space as in the prior art and to set the irradiation point. For this reason, it is conceivable that an operator holds a portable laser pointer or the like and irradiates a spot light while moving in the field of view of the camera to set an irradiation point.

  However, there are cases where some irradiation points are not detected due to the reasons described below even though the operator has performed irradiation. That is, a detection failure occurs when the illumination environment at the irradiation point is bright and a sufficient luminance difference cannot be obtained between the irradiation point and the background. Alternatively, detection failure occurs when the background color of the irradiation point is bright and a sufficient luminance difference cannot be obtained between the irradiation point and the background. Alternatively, a detection omission occurs when the irradiation point is far from the irradiator and the luminance at the irradiation point decreases and a sufficient luminance difference cannot be obtained between the irradiation point and the background. Alternatively, since the irradiation point is far from the camera, the image of the irradiation point is small, and the irradiation point is excluded as noise, resulting in detection omission. Alternatively, a plurality of causes among the above-described causes are combined to cause detection failure.

  If a shortage of irradiation points occurs due to a detection omission, the operator resets the irradiation points. At this time, it is difficult for an operator who sets a large number of irradiation points in a wide range to determine a position to be reset or to determine a light source position that reliably detects the irradiation point to be reset, In particular, there is a problem that it becomes more difficult when there are many cameras or when multiple causes are combined.

  A camera system according to the present invention includes one or a plurality of cameras that capture an image of a predetermined space, a control device that processes an image captured by the camera, and an irradiation point that can irradiate the space and can be captured by the camera. A movable light source, and a storage unit that stores at least a plurality of irradiation request positions in the space where the irradiation points need to be formed; A recommended light source position determination unit that determines a position where the sum of the distances from the plurality of irradiation request positions is a minimum as a recommended light source position when irradiating the plurality of irradiation request positions, and outputs the recommended light source position And an output unit.

  In the camera system according to the present invention, the output unit may output the irradiation request position.

  In another camera system according to the present invention, the recommended light source position determination unit extracts a luminance value of the irradiation request position in the image, and weights the distance from the irradiation request position according to the luminance value. And the sum is calculated.

  In another camera system according to the present invention, the control device further includes a request position setting unit that sets the irradiation request position and stores the irradiation request position in units of segmented areas into which the image is divided, and the recommended unit When the plurality of irradiation request positions are adjacent areas, the light source position determination unit increases the weight for the distance as the luminance value is lower.

  In still another camera system according to the present invention, the control device further includes a request position setting unit that sets the irradiation request position and stores the irradiation request position in units of a segmented area obtained by dividing the image, The recommended light source position determination unit increases the weight with respect to the distance as the luminance value is higher when the plurality of irradiation request positions are separated from each other.

  In these camera systems according to the present invention, a plurality of the cameras are installed so as to have a common field of view in the space, and the storage unit stores camera parameters including the position and orientation of the camera in the space. The control device further includes an irradiation point detection unit that detects the irradiation point from an image captured by the camera, and a calibration unit that calibrates the camera parameter based on the irradiation point. The calibration unit detects the irradiation point image at the same position in the space when the irradiation point detection unit detects the irradiation point image as the irradiation point in each of the images simultaneously captured by the plurality of cameras. The camera parameters can be calibrated by applying a constraint condition that exists.

  According to the present invention, since the operator knows the light source position (standing position) for detecting the irradiation point more reliably, the irradiation point can be efficiently detected with less redo and trial and error, and calibration work and measurement work are easy.・ It becomes quick.

1 is a schematic diagram illustrating a schematic configuration of a camera system according to an embodiment of the present invention. It is a typical top view of monitoring space. It is the block diagram which showed the structure of the outline of a control apparatus. It is a schematic diagram which shows the data structure of a camera parameter in a table format. It is a schematic diagram which shows the example of the request | requirement position set to the camera pair which consists of the camera 2B and the camera 2C. It is a schematic diagram which shows an example when it fails to detect an irradiation point. It is a flowchart which shows the outline of operation | movement of the image process part in the calibration process of the camera system which concerns on embodiment of this invention. It is a flowchart which shows the outline of the presentation process of the recommended light source position which the camera system which concerns on embodiment of this invention performs by a calibration process. It is a schematic diagram which shows the data structure of irradiation point information in a table format. It is a schematic diagram which shows the data structure of request information in a table format. It is a schematic diagram which shows the data structure of a chain determination table in a table format. It is a schematic diagram of the image area | region which shows the example in case the weighted average position weighted so largely that the brightness | luminance of each pixel is made into a recommended light source position. It is a schematic diagram of the image area | region which shows the example in case the weighted average position weighted so heavily that the brightness | luminance of each pixel is high is made into a recommended light source position. It is an example of the display screen in the display part corresponding to FIG. It is an example of the display screen in the display part corresponding to FIG. It is a flowchart which shows the outline of the operation | movement in the measurement process of the camera system which concerns on embodiment of this invention. It is a schematic diagram of the image area | region which shows the example of a setting of the irradiation request information in a measurement process.

  Hereinafter, a camera system according to an embodiment of the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings.

[Configuration of Camera System 1]
FIG. 1 is a schematic diagram illustrating a schematic configuration of a camera system 1 according to the embodiment. The camera system 1 includes a plurality of cameras 2 (2A, 2B, 2C, 2D, 2E, 2F, 2G, 2H), an irradiator 3, a control device 4, and a display unit 5. FIG. 1 shows a situation where an operator 10 carrying the irradiator 3 and the display unit 5 irradiates the floor surface (reference surface 11) in the monitoring space to form a light image on the floor surface. The object 12 is installed on the floor.

  In the present embodiment, the center of gravity of the optical image is the irradiation point 13. The worker 10 moves in the monitoring space and installs irradiation points 13 at various places. The camera 2 captures the irradiation point 13, and the control device 4 detects the irradiation point 13 from the captured image, and performs calibration of the camera 2 and three-dimensional measurement of the object 12. In addition, the control device 4 outputs to the display unit 5 a light source position (recommended light source position) that can form an easily detected irradiation point 13 at a required position (irradiation request position) where the irradiation point 13 is required to be installed. To do. The operator 10 can perform the work efficiently by reducing the failure by performing irradiation from the displayed standing position.

  The camera 2 is a surveillance camera, and each camera 2 is connected to the control device 4. Each camera 2 is arranged in a chain with at least one other camera providing a common field of view, and the sum of the fields of view of the plurality of cameras 2 becomes a monitoring space. The camera 2 captures the irradiation point 13 that moves together with the worker 10 every predetermined time, and sequentially outputs the captured images to the control device 4.

  In the example shown in FIG. 1, the camera system 1 includes eight cameras 2A, 2B, 2C, 2D, 2E, 2F, 2G, and 2H, and the control device 4 includes A, B, C,. A camera ID of H is assigned and managed. The plurality of cameras 2 are arranged in a grid pattern on the ceiling with their optical axes oriented vertically downward. FIG. 2 is a schematic plan view of the monitoring space 20, and the position of the object 12 within the reference plane 11 and the arrangement of the eight cameras 2 (2A to 2H) are indicated by “☆”. The distance between the cameras 2 is set so that adjacent cameras have a common field of view in consideration of the angle of view of the cameras. In this example, camera 2A has at least a common field of view with camera 2B, camera 2B has at least a common field of view with cameras 2A and 2C, and camera 2C has at least a common field of view with cameras 2B, 2D, 2G, and 2H. , Camera 2D has a common field of view with at least cameras 2C, 2G, 2H, camera 2E has a common field of view with at least camera 2F, camera 2F has a common field of view with at least cameras 2E, 2G, and camera 2G has at least The cameras 2C, 2D, 2F, and 2H have a common visual field, and the camera 2H has at least a common visual field with the cameras 2C, 2D, and 2G. In the figure, the field of view of the camera 2B is indicated by 100b, and the field of view of the camera 2C is indicated by 100c. The other visual fields of the camera 2 are not shown. A region 200 indicates a common field of view of the field of view 100b of the camera 2B and the field of view 100c of the camera 2C. Note that the number of cameras 2 is arbitrary as long as it is two or more, and the position and orientation of the camera 2 are arbitrary within a range where at least common fields of view are chained.

  The irradiator 3 is a light source that outputs spot light. For example, a laser pointer or a collimated LED light can be used. The irradiator 3 is carried by the operator 10 and irradiates light to the object 12 such as the reference plane 11 or the fixture by the operation of the operator 10. By this irradiation, an irradiation point 13 is set on the reference surface 11 or the surface of the object 12. Incidentally, the operator 10 irradiates the reference plane 11 in the common visual field in the calibration work of the position and orientation of the camera 2, and irradiates the object 12 existing in the common visual field in the three-dimensional measurement work of the object 12. The irradiator 3 is preferably one that can be easily obtained and used without requiring a license.

  The control device 4 is connected to each camera 2 via a LAN such as Ethernet (registered trademark) or a wiring such as a coaxial cable. The display unit 5 is connected by a wireless line such as Bluetooth. The control device 4 acquires an image taken by the camera 2 and detects the irradiation point 13 from the image. The control device 4 calculates a worker standing position that does not hinder the detection of the irradiation point 13 at the requested position, and outputs information including the requested position and the worker standing position to the display unit 5. The detected irradiation point 13 is used to calibrate the camera 2 or measure the object 12.

  The display unit 5 includes a wireless communication interface and is connected to the control device 4 through a wireless line. The display unit 5 includes display means including an image display device such as a liquid crystal touch panel display and an arithmetic device, and displays information input from the control device 4 so as to be visible.

  FIG. 3 is a block diagram showing a schematic configuration of the control device 4. The control device 4 includes an image acquisition unit 40, a storage unit 41, an image processing unit 42, and a display communication unit (output unit) 43.

  The image acquisition unit 40 includes a camera communication unit 400 and a synchronization control unit 401, and synchronizes images from the cameras 2 and outputs the images to the image processing unit 42.

  The camera communication unit 400 is a communication interface that communicates with the camera 2 and sequentially receives images captured by each camera 2. The camera ID of the camera 2 that captured the image is assigned to the received image and the synchronization control unit 401 receives the image. Output. Here, if the camera 2 is an analog camera, the camera ID can be specified from the correspondence with a preset connection terminal, and if it is an IP camera, the camera ID can be specified from the correspondence with a preset transmission source address.

  The synchronization control unit 401 synchronizes the image capturing times received by the camera communication unit 400 from the respective cameras 2. Specifically, each camera 2 captures and transmits an image with a capturing period sufficiently shorter than the reference period, and the synchronization control unit 401 synchronizes the received image with less than an allowable time difference set to be considered as simultaneous. It is assumed that the taken simultaneous image is taken. The synchronization control unit 401 extracts a simultaneously shot image for each reference period, assigns a common frame number to the simultaneously shot images, and outputs the same to the image processing unit 42. The synchronization control unit 401 generates a buffer (not shown) that accumulates images from each camera 2 for at least the time of the reference period, and generates time information to determine the coincidence, in order to extract simultaneously shot images for each reference period. Timing means (not shown). For example, the reference period may be 3 fps (frame per second), the imaging period may be 15 fps, and the allowable time difference may be 66 ms (mili second), for example. Note that the synchronization timing may be directly controlled by outputting a synchronization pulse from the synchronization control unit 401 to each camera 2 for each reference period.

  The storage unit 41 is a memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 41 stores various parameters, a program of the image processing unit 42, and the like, and inputs / outputs such information to / from the image processing unit 42. The various parameters include camera parameters 410, irradiation request information 411, irradiation limit values 412, irradiation point information 413, and object measurement information 414.

  The camera parameters 410 are internal parameters and external parameters of each camera 2 and are associated with the camera ID of the camera 2. FIG. 4 is a schematic diagram showing the data structure of the camera parameter 410 in a table format.

Internal parameters include focal length and optical center position of the lens. In FIG. 4, for example, the focal length of the camera 2A is represented as f _A , and the optical center position of the lens is represented as c _A. Accurate values are set in advance for the focal length and the optical center position of the lens, and do not change during calibration of the camera 2.

  The external parameters include at least information on the position and orientation of the camera 2 in the monitoring space. Specifically, the installation height and rotation matrix of the camera 2 expressed in a common coordinate system (world coordinate system) common to the plurality of cameras 2. , Translation vector, photographing plane normal vector, and inter-camera distance.

  Here, the world coordinate system is a right-handed orthogonal coordinate system XYZ that imitates a monitoring space. In this embodiment, the Z axis is set vertically upward, the floor surface that is the reference surface is set to height Z = 0, and the reference surface The point below the camera A at the vertical is the coordinate origin. In addition, the horizontal direction and the vertical direction of the image of the camera A are the x-axis and y-axis of the photographing plane, and the x-axis and y-axis projected onto the reference plane are the X-axis and Y-axis, respectively.

For example, the rotation matrix R _B of the camera 2B defines the rotation around the optical axis of the camera 2B. The normal vector n _{B on the} imaging plane defines the direction of the optical axis of the camera 2B. The translation vector is a two-dimensional vector that defines the relative positional relationship between the two cameras 2 in the XY plane. Translation vector in the present embodiment is defined the position of the camera 2A starting, for example, the translation vector t _B of the camera 2B is starting from a position of the camera 2A, 2B in each XY plane, and the end point. The inter-camera distance is the distance of a certain camera 2 from the reference camera 2 and is the distance in the XY plane of both cameras 2. In the present embodiment, the inter-camera distance is defined with reference to the camera 2A. For example, in FIG. 4, the inter-camera distance d _AB indicates the distance between the cameras 2A and 2B. The inter-camera distance may be a distance between the three-dimensional coordinates of the camera 2.

  Among these external parameters, the installation height is set to an accurate value in advance and does not change during the calibration work of the camera 2. The rotation matrix and the normal vector of the imaging plane are set in advance as initial values that indicate a vertically downward value, which is a value in the installation plan, and are used as a calibration work target of the camera 2. Initial values are not set for the translation vector and the inter-camera distance, but are set by the calibration operation of the camera 2.

  The irradiation request information 411 can simultaneously photograph a request position that is a position in a monitoring space where it is necessary to request the worker 10 to set the irradiation point 13 for calibration and three-dimensional measurement, and the request position. Contains information associating possible camera pairs. In the irradiation request information 411, coordinate data representing the requested position is set in association with the camera ID of the camera pair by a request setting unit 422, which will be described later. The requested position can be a point or a line, but is preferably an area. By setting the required position as an area, a request from the camera system 1 to the worker 10 becomes a request to set the irradiation point 13 at an arbitrary position in the area. Accordingly, the burden on the operator 10 is reduced because the target of irradiation increases. In addition, by setting the area, when a part of a specific requested position is too bright, the remaining part excluding the part can be continued as the requested position. Therefore, in this embodiment, the requested position is an area, and the area of each requested position is also referred to as an area hereinafter. FIG. 10 used in the description of the subsequent processing schematically shows the data structure of the irradiation request information 411. In FIG. 10, symbols such as “A5” indicating the request position indicate areas as described later. Further, the irradiation request information 411 includes a requirement condition that defines the installation requirement of the irradiation point 13 at the required position.

  The irradiation limit value 412 is an irradiation limit luminance value and an irradiation limit distance value set in advance in consideration of the characteristics of the irradiator 3. When an excessively bright place is irradiated, a sufficient luminance difference cannot be obtained between the irradiation point 13 and the background, and there is a high possibility that the irradiation point 13 cannot be detected. The irradiation limit luminance value is a threshold value for determining whether or not the required position is too bright to detect the irradiation point 13. Further, at a place too far from the light source, the luminance per unit area decreases due to an increase in the diameter of the optical image, and there is a high possibility that the irradiation point 13 cannot be detected. The irradiation distance limit value is a threshold value for determining whether or not the requested position is so far that the irradiation point 13 cannot be detected.

  The irradiation point information 413 is generated by an irradiation point detection unit 420 described later, and is used by the general calibration unit 421, the precision calibration unit 424, and the object measurement unit 425. The irradiation point information 413 is information on the irradiation point 13 detected from the captured images of the plurality of cameras 2 captured in synchronization with the reference period, and an image of the irradiation point 13 (irradiation point image) detected in the captured image. Are associated with the camera ID and the frame number assigned to the image from which the irradiation point image is detected. FIG. 9 used in the description of the subsequent processing schematically shows the data structure of the irradiation point information 413. In FIG. 9, (57, 90) indicating the detection position indicates the x coordinate and y of the irradiation point image in the photographed image. It represents a pair with coordinates.

  The object measurement information 414 is information obtained by measuring the three-dimensional shape of the object 12 existing in the monitoring space, and is generated by the object measurement unit 425 based on the irradiation point information 413. The object measurement information 414 is used as background information when the camera system 1 detects or tracks a person in the monitoring space.

  The chain determination table 415 is information used for managing the progress status of the precision calibration process, which will be described later. As shown in FIG. 11 used in the explanation of the subsequent process, the camera pair to be subjected to precision calibration and the completion of the precision calibration are completed. The data structure is associated with / incomplete.

  The image processing unit 42 is configured using an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit), and detects an irradiation point by reading and executing a program from the storage unit 41. Functions as a unit 420, a general calibration unit 421, a request setting unit 422, a recommended light source position calculation unit 423, a precision calibration unit 424, an object measurement unit 425, and the like.

  The irradiation point detection unit 420 detects the irradiation point 13 from the image captured by the camera 2. Specifically, irradiation point image detection processing is performed on images captured by each camera 2 at a reference period, and the detection position of each irradiation point image is assigned to the image from which the irradiation point image is detected. The irradiation point information 413 of the storage unit 41 is stored in association with the ID and the frame number. In the detection process, a difference area is extracted based on a background difference or an inter-frame difference, a labeling process is performed on the extracted difference area, and an area in a size range set in advance based on the size of the light image of the irradiator 3 is emitted. The image is selected as an image, and the center of gravity of the selected light image is detected as the irradiation point 13. By providing a restriction on the size range, the difference area caused by the worker 10 is excluded.

  Here, the irradiation point 13 set in the common visual field of the plurality of cameras 2 is basically photographed simultaneously by each of the plurality of cameras 2, and one irradiation point 13 is set at a temporary point by the operator 10. It is. Therefore, by searching for the irradiation point 13 associated with the same frame number in the irradiation point information 413, a camera pair having a common field of view can be detected. The irradiation point detection unit 420 extracts an irradiation point pair having the same frame number from the irradiation point information 413, and stores the camera ID of the extracted irradiation point 13 in the irradiation request information 411 as a camera pair having a common field of view.

  In this embodiment, the calibration operation of the camera 2 is performed in two stages: rough calibration (rough calibration) and higher-precision calibration (precision calibration). The general calibration process is performed by the general calibration unit 421, and the precise calibration process is performed by the precise calibration unit 424. The general calibration unit 421 and the precision calibration unit 424 perform respective calibration processes based on the irradiation point information 413 generated by the irradiation point detection unit 420. Here, the precision calibration unit 424 performs calibration with higher accuracy than the general calibration by using the irradiation points 13 at more required positions than the general calibration unit 421.

  The general calibration unit 421 and the precision calibration unit 424 extract from the irradiation point information 413 the irradiation point images that are commonly detected in the simultaneously captured images, and restrain that these irradiation point images correspond to the same position in the monitoring space. The position / orientation of the camera 2 is calculated by applying the conditions, and the camera parameter 410 of each camera 2 is updated with the position / orientation. Specifically, the position of the irradiation point image detected by the camera pair to be calibrated is corrected for distortion with the internal parameters of the camera 2, and the detection position of the irradiation point image and the installation height of the camera 2 are used. A homography matrix is derived. Then, a rotation matrix, a translation vector, a normal vector of the imaging plane, and an inter-camera distance of each camera constituting the camera pair are calculated from the homography matrix for each camera pair, and these are converted into a common coordinate system to be used for each camera 2. Find the position / posture. How to find the position and orientation using the homography matrix is known from “Study on Optimal Triangulation of Planar Scene” Information Processing Society of Japan, CVIM, 2010-CVIM-171 (4), 1-8, 2010-08-11) It is. In addition to the method using a homography matrix, various calibration methods based on epipolar geometry can be applied.

  The object measurement unit 425 applies a constraint condition that the irradiation point images detected in common from the simultaneously shot images by the irradiation point detection unit 420 correspond to the same position in the monitoring space, and the three-dimensional object 12 existing in the common visual field. Measure the position. The three-dimensional measurement function by the object measurement unit 425 is used after the precision calibration unit 424 has finished the precision calibration process. That is, the three-dimensional measurement process (object measurement process) is executed in a state where the precise position / orientation of the camera is stored in the camera parameter 410 of the storage unit 41.

  More specifically, the object measurement unit 425 determines the triangulation principle at the position / posture of the camera pair determined by the camera parameter 410 and the detection position of the irradiation point 13 captured by the camera pair stored in the irradiation point information 413. Is applied to derive the three-dimensional position of each irradiation point 13. Further, the object measuring unit 425 derives a plane for each requested position by regarding a plurality of irradiated points 13 having the same corresponding requested position among the irradiated points 13 as points on the same plane. Further, the object measuring unit 425 derives a tangent line between the derived planes, and sets the tangent line as a side of each plane. The monitoring space is a solid surface composed of a reference surface (floor surface), a virtual ceiling having the same height as the camera installation height, and a virtual wall defined by a finite range of XY coordinates, and a tangent line is defined as a reference surface, a virtual ceiling, and a virtual wall. Derived even between.

  The object measurement unit 425 stores the data of each plane in which the edge is derived and the contour is defined in the object measurement information 414 of the storage unit 41 as the three-dimensional shape data representing the surface of the object 12.

  As described above, the irradiation point 13 detected by the irradiation point detector 420 is used in the rough calibration process, the precision calibration process, and the object measurement process. The request setting unit (requested position setting unit) 422 sets a plurality of required positions for which installation of the irradiation point 13 is required within the common visual field for each camera 2. Specifically, the request setting unit 422 generates the above-described irradiation request information 411 and stores it in the storage unit 41. In addition, the request setting unit 422 stores the request information 411 and then sequentially refers to the reference point information 413 and the irradiation request information 411 so that the irradiation point detection unit 420 exists at each request position in the common visual field. Is detected, and only when the irradiation point 13 is detected, the requested position where the irradiation point 13 was photographed is deleted from the storage unit 41. In addition, the setting criteria of the request | requirement position in each process differ, and each is demonstrated below.

(1) Required position in rough calibration process In the calibration process (rough calibration process and precision calibration process) of a camera pair having a common field of view, it is necessary to detect a plurality of irradiation points 13 from the reference plane of the common field of view. In this embodiment, as described above, calibration using a homography matrix is applied to each camera pair. In this case, four or more irradiation points 13 that are not on the same straight line for each camera pair are detected from the simultaneously captured image, and these irradiation points 13 are dispersed in order to ensure the accuracy of calibration. Is desirable. For this reason, a total of 8 or more irradiation points 13 are required to be detected by the two cameras 2 in a pair. Therefore, the request setting unit 422 sets a plurality of request positions in units of segmented areas obtained by segmenting the images of the cameras 2. An example of a suitable division method is to divide the image radially at predetermined angles around the intersection of the optical axis of the camera 2 and the imaging plane.

  Further, apart from the plurality of radial areas described above, it is preferable that the image area of the camera 2 is additionally set as a single area where the intersection of the optical axis of the camera 2 and the imaging plane is located inside. . Thereby, it is possible to avoid the crowding of a plurality of radial areas in the central portion of the radial section, and it is possible to eliminate the situation where the irradiation points 13 are detected in a concentrated manner in a narrow region of the central portion. That is, it is possible to prevent the degree of dispersion of the detected irradiation points 13 from being lowered, and to ensure the accuracy of calibration. Therefore, in the present embodiment, the request setting unit 422 includes 9 areas including a circular area corresponding to the vicinity of the bottom of each camera 2 and 8 areas obtained by radially dividing the area remaining outside the camera 2 in 45 degree increments. Is set as the requested position.

  FIG. 5 is a schematic diagram showing the required positions set as described above, and FIG. 5 shows an example of the required positions set for the camera pair composed of the camera 2B and the camera 2C. FIGS. 5A and 5B show image areas 21b and 21c photographed by the cameras 2B and 2C arranged in the monitoring space 20 shown in a plan view in FIG. Incidentally, the range shown in the image areas 21b and 21c in the monitoring space 20 is represented by the fields of view 100b and 100c of the cameras 2B and 2C, respectively, in the plan view of the monitoring space 20 shown in FIG. Requested positions B1 to B9 are shown in the image area 21b of the camera 2B shown in FIG. 5A, and required positions C1 to C9 are shown in the image area 21c of the camera 2C shown in FIG. 5B. Yes. Specifically, an area B1 having a radius R on the reference plane 11 is set as an image area 21b as a circular area corresponding to the vicinity of the camera 2B vertically below, and lines connecting the cameras 2B and 2C vertically below each other are set. The image area around the area B1 is divided into 8 areas in increments of 45 degrees as 0 degrees, and areas B2 to B9 are set in order. Similarly, an area C1 which is one circular area with respect to the camera 2C and areas C2 to C9 obtained by radially dividing the periphery are set.

  The circular area in the center is set for the purpose of dispersing the radial area, and the size thereof is determined so that the object can be achieved. For this purpose, the central area is not limited to a circle but may be a polygon such as a quadrangle.

  For each camera 2, the request setting unit 422 reads the initial value of the camera parameter 410 from the storage unit 41, classifies the image area of the camera 2 based on this, and sets nine areas as the required position in the rough calibration process. To do. Then, for each camera 2 of the camera pair, the irradiation point 13 (irradiation point image) is detected in any four areas of the nine areas, and the irradiation point 13 (irradiation point image) is detected in a total of eight areas in the camera pair. Is set as a requirement. Two or more detections are allowed in one area.

  The requested position in the image area is not limited to the radial section, and the requested position can be dispersed in the image area by a block (lattice) section.

(2) Required Position in Precision Calibration Process The description generally given as the calibration process in (1) above is common to the rough calibration process and the precision calibration process, and the request setting unit 422 in this embodiment is a precision calibration process. The above nine areas are also set as required positions. For each camera 2, the request setting unit 422 reads the camera parameters 410 roughly calibrated from the storage unit 41, classifies the above-described image area of the camera 2 based on the camera parameters 410, and sets nine areas as required positions in the precision calibration process. Set. For each camera 2 of the camera pair, the irradiation point 13 (irradiation point image) is detected in all nine areas, and the camera pair requires that the irradiation point 13 (irradiation point image) is detected in a total of 18 areas. Set as a condition. Two or more detections are allowed in one area.

  That is, the difference between the required position in the precision calibration process and the required position in the rough calibration process is used for setting the area and the number of irradiation points 13 required by the required position in the precision calibration process and in the area setting. The camera parameter 410 is an initial value in rough calibration, whereas it is a value that is roughly calibrated in precision calibration.

(3) Required position in measurement processing The required position in measurement processing is a position where detection of the irradiation point 13 necessary for three-dimensional measurement of the object 12 is required, and is based on images taken by each camera 2. Generated.

  In the present embodiment, it is assumed that the object surface is a collection of planes, and the request setting unit 422 sets an area estimated as a plane constituting the object surface as a required position. In addition, in order to specify a plane, the request setting unit 422 sets detection of three or more irradiation points 13 that are not on the same straight line as the required condition for each region.

  Specifically, the request setting unit 422 divides the image of each camera 2 into a group of adjacent pixels that are similar in color to each other, and sets each of the divided areas as a required position. For example, an image can be divided by performing clustering using the pixel color and the pixel position as parameters.

  In another embodiment, the request setting unit 422 generates an edge image by filtering the image of each camera 2 using a Sobel filter or the like, and divides the edge image into closed regions each surrounded by an edge. Each can be set to a required position.

  Here, even if the operator 10 performs irradiation, the irradiation point detection unit 420 sometimes fails to detect the irradiation point. FIG. 6 is a schematic diagram showing an example when the irradiation point is missed. FIG. 6A shows a case where an area 14 that is too bright is generated in the common visual field due to direct sunlight or the like inserted through a window. In this case, even if this region 14 is irradiated, a significant luminance difference does not occur between the irradiation point 15 and the surrounding background, so that the irradiation point 15 cannot be detected.

  FIG. 6B shows a case where the distance from the light source position to the irradiation point is too far. In this case, since there is no significant luminance difference between the irradiation point 16 and the surrounding background due to a decrease in luminance per unit area due to light attenuation and an increase in the diameter of the optical image, the irradiation point 16 is It cannot be detected.

  Also, when the distance from the camera to the irradiation point is long, the optical image becomes too small to be distinguished from noise, and the irradiation point cannot be detected from the optical image.

  Furthermore, a combination of a decrease in luminance due to the distance from the light source position to the irradiation point and a bright shared visual field may fail to be detected. In this case, there are various combinations of the degree of brightness reduction and the degree of brightness that cause detection failure. Further, the detection failure due to the brightness may be caused by the fact that the background color itself is bright. Similarly, in the case where the light image size reduction due to the distance from the camera to the irradiation point and the brightness of the common field of view are combined and fail to detect, the irradiation point detection failure is a combination of two or more of the above factors. Can occur.

  These missed detections are basically improved by irradiating near the required position, but the operator 10 grasps all the missed detections with many cameras 2 installed. Have difficulty. Further, it is inefficient to perform irradiation near each required position.

  Therefore, the camera system 1 calculates a light source position suitable for irradiation of the requested position by a recommended light source position calculation unit 423 described below and presents it to the operator 10. In other words, the worker 10 is presented with certainty where irradiation is reliable and efficient.

  The recommended light source position calculation unit 423 is a recommended light source position determination unit that determines a recommended light source position that is a position of a light source suitable for detecting the irradiation point 13 from the requested position of interest. The position where the sum of the distances is the minimum is determined as the recommended light source position when irradiating the plurality of required positions. In the present embodiment, a light source position suitable for irradiation of the required position for each camera pair that performs precision calibration processing or measurement processing, in particular, a light source position whose distances from a plurality of required positions are close in average is calculated as a recommended light source position. A composite image in which the requested position and the recommended light source position are superimposed on the captured image is output to the display communication unit 43.

  As a result, it is possible to present the operator 10 with a light source position that can reliably detect the irradiation point 13 for a plurality of required positions from one place. As a result, the operator 10 can efficiently set the irradiation point 13 while irradiating a plurality of required positions from the light source position while reducing the number of irradiation failures, so the work load is significantly reduced.

  The recommended light source position can be obtained as the center of gravity of a plurality of required positions. Specifically, the recommended light source position calculation unit 423 calculates the center of gravity (average position) of the pixel positions included in each of the plurality of required positions on the captured image as the recommended light source position. Furthermore, by calculating the center of gravity by weighting the distance from each requested position to the light source position with the brightness of the requested position, the work load can be further reduced. Specifically, the recommended light source position calculation unit 423 extracts the luminance value of the pixel included in the requested position in the captured image, and calculates the center of gravity by weighting the position of each pixel with the luminance value of the pixel. There are two types of weighting: a case where a greater weight should be applied to a darker one and a case where a greater weight should be applied to a brighter one. Details of the recommended light source position calculation process will be described later.

  The display communication unit 43 is a communication interface that performs communication between the control device 4 and the display unit 5. The display communication unit 43 functions as an output unit that outputs information on the requested position input from the recommended light source position calculating unit 423 and information on the recommended light source position with respect to the requested position to the display unit 5.

[Operation in Calibration Processing of Camera System 1]
FIG. 7 is a flowchart showing an outline of the operation in the calibration process of the camera system 1. FIG. 8 is a flowchart showing an outline of a recommended light source position presentation process performed by the camera system 1 in the calibration process. The operation in the calibration process of the camera system 1 will be described with reference to FIGS.

  When the camera 2 is installed and the control device 4 and the display unit 5 are turned on, the calibration operation starts. Immediately after startup, the camera parameter 410 is an initial value and is in a state before rough calibration. Each camera 2 sequentially captures its field of view and transmits the captured image to the control device 4.

  The image acquisition unit 40 of the control device 4 sequentially inputs the simultaneously captured images to the image processing unit 42 of the control device 4. In this state, the worker 10 irradiates the floor with the irradiator 3 while moving in the monitoring space.

  Every time a simultaneously captured image is input from the image acquisition unit 40 to the image processing unit 42, the processing from step S1 to step S14 is repeated. Hereinafter, the repeating time unit is referred to as a frame.

  When the image processing unit 42 acquires the simultaneously captured image (S1), the image processing unit 42 operates as the irradiation point detection unit 420, performs a difference process on each simultaneously captured image, and detects the irradiation point 13 (irradiation point image) from the simultaneously captured image. (S2). The irradiation point detection unit 420 associates the detected coordinates on the image of the irradiation point 13 with the camera ID of the image and the frame number of the image and stores them in the irradiation point information 413 of the storage unit 41.

  Next, the image processing unit 42 operates as the general calibration unit 421 to detect a new camera pair (S3). That is, the general calibration unit 421 searches the irradiation point information 413, extracts a pair of camera IDs associated with the irradiation point 13 having the same frame number as a camera pair, and extracts it with reference to the camera parameter 410. It is confirmed whether or not the camera pair is a new camera pair that is not set as a precision calibration target. The detected camera pair is a camera sharing at least a part of the field of view. In addition, it may be added to the extraction condition that there are four or more pairs of irradiation point images having the same frame number.

  When one or more new camera pairs are detected (in the case of “YES” in S3), the image processing unit 42 operates as the request setting unit 422, and generates irradiation request information 411 for rough calibration for each camera pair. Is stored in the storage unit 41 (S4). Specifically, the request setting unit 422 reads out the camera parameters 410 of each camera 2 constituting the new camera pair, sets nine areas for each camera, and requests position where the irradiation point 13 is detected in the camera pair. The required condition is set so that the number of cameras is any 4 areas for each camera and 8 areas in total for the camera pairs.

  Subsequently, the request setting unit 422 confirms whether or not each new camera pair satisfies a rough calibration requirement condition (S5). Specifically, the request setting unit 422 refers to the irradiation point information 413 and the irradiation request information 411 of the new camera pair, and the area (to which the coordinates of the irradiation point image associated with the camera ID of the camera pair belong ( (Required position) is deleted, and it is confirmed from the number of remaining requested positions whether the detected areas are 4 areas or more for each camera, that is, a total of 8 areas or more.

  If the new camera pair includes a camera pair that satisfies the required conditions (in the case of “YES” in S5), the image processing unit 42 operates as the general calibration unit 421 and uses the irradiation point information 413 of the camera pair. Approximate calibration is performed (S6), the approximate position / orientation values as calibration results are written in the camera parameter 410, and the camera pair is set as a precision calibration target (S7).

  When a new camera pair is set as a precision calibration target, the image processing unit 42 operates as the request setting unit 422, generates irradiation request information 411 for precise calibration for the camera pair, and stores it in the storage unit 41 (S8). . That is, the request setting unit 422 reads the camera parameters 410 of each camera 2 constituting the new camera pair, sets nine areas for each camera, and sets the number of requested positions where the irradiation point 13 is detected in each camera pair. The required conditions are set to 9 areas for the camera and 18 areas in total for the camera pair.

  Since the area set in step S8 is calculated using the camera parameter 410 roughly calibrated, the area is more suitable for the actual position and orientation of the camera 2 than the area set in step S4.

  When a new camera pair is not detected (in the case of “NO” in S3), the image processing unit 42 skips steps S4 to S8 for the current frame, and proceeds to step S9. If a new camera pair is detected but does not satisfy the rough calibration requirement (in the case of “NO” in S5), the image processing unit 42 skips steps S6 to S8 for the current frame and skips step S9. Proceed to

  Through the processing so far, the camera pairs irradiated by the operator 10 with a common field of view of four or more areas are sequentially listed as precision calibration targets by the image processing unit 42, and the irradiation request information 411 for each camera pair to be precisely calibrated is set. (S8).

  Next, the image processing unit 42 operates as the request setting unit 422 to check whether there is a camera pair that has not been subjected to precision calibration among the camera pairs that are subject to precision calibration (S9). After updating the required position based on the detection result of the irradiation point 13 in step S2 (S10), it is confirmed whether or not the camera pair satisfies the required conditions for precision calibration (S11). Specifically, the request setting unit 422 refers to the camera parameter 410 to check whether there is a camera pair for which precision calibration has not been completed (S9), and irradiation point information 413 and irradiation request information 411 for a camera pair for which precision calibration has not been completed. With reference to the above, after deleting the area (required position) to which the coordinates of the irradiation point 13 associated with the camera ID of the camera pair belong (S10), has the remaining required position reached zero? That is, it is confirmed whether the irradiation point 13 is detected in 18 areas (S11).

  When a camera pair that satisfies the required conditions is included in the camera pair that has not been subjected to the precision calibration (in the case of “YES” in S11), the image processing unit 42 operates as the precision calibration unit 424 and the irradiation point of the camera pair. Precision calibration is performed using the information 413 (S13), the value of the precise position / orientation as the calibration result is written in the camera parameter 410, and information indicating that the camera has been precision calibrated is stored in the chain determination table 415. It is written and it is confirmed whether or not the precision calibration of all cameras is completed (S14).

  In step S14, the precision calibration unit 424 searches the camera parameter 410 to count the number of cameras that have been precision calibrated, and confirms the connection relationship of the camera pair that has been precision calibrated in the chain determination table 415. If the count matches the number of all cameras and there are no isolated camera pairs, it is determined that the precision calibration of all the cameras has been completed. When the precision calibration of all the cameras is completed (in the case of “YES” in S14), the calibration process is terminated (S15). By the way, if redundant precision calibration targets are set, calibration may be completed even if there are still uncalibrated camera pairs.

  On the other hand, if the count is smaller than the total number of cameras or if there is an isolated camera pair, the process is returned to step S1 as incomplete, and the process for the simultaneously shot image of the next frame of the next frame is performed (S14). "NO" → S1).

  With reference to FIG. 9 to FIG. 11, a processing example when the required condition is satisfied (in the case of “YES” in S11) will be described. 9, 10, and 11 schematically show the data structure of the irradiation point information 413, the irradiation request information 411, and the chain determination table 415, as described above, and their changes between two consecutive frames. Is shown. FIGS. 9 (a), 10 (a), and 11 (a) show states in the 41st frame, respectively, and FIGS. 9 (b), 10 (b), and 11 (b) show the states in the 41st frame, respectively. The state in 42 frames is shown.

  In the irradiation point information 413 in the 41st frame, information on the irradiation point 13 detected by the irradiation point detection unit 420 in the imaging in the 1st to 41st frames is stored (FIG. 9A). From the 37th frame to the 41st frame, the irradiation point 13 is detected in the common visual field of the cameras B and C.

  In addition, the rough calibration unit 421 roughly calibrates the camera pairs “A, B”, “A, C”, “B, C”, and “C, D” until the 41st frame, and as a result, the chain determination table. In 415, it is recorded that these camera pairs have been set as precision calibration targets (FIG. 11A). Further, the chain determination table 415 records that the precision calibration by the precision calibration unit 424 has been completed for the camera pair “A, B” and the precision calibration has not been completed for the other camera pairs (FIG. 11 (a)).

  Of the 18 areas of areas B1 to B9 and areas C1 to C9 set in the irradiation request information 411 as the initial request position for the camera pair “B, C” by the request setting unit 422 at the time of the 41st frame, the area B3 2 areas of C4 and C4 remain as required positions (FIG. 10A). That is, when the irradiation point 13 is detected in the areas B3 and C4 in the subsequent frames, the camera pair “B, C” can be precisely calibrated. It should be noted that the required position for the camera pair “A, B” that has completed precision calibration has already been deleted from the irradiation request information 411.

  Next, in the 42nd frame, the irradiation point detection unit 420 newly detects an irradiation point image from each of the images of the cameras B and C, and the detection result is added to the irradiation point information 413 (FIG. 9B). The request setting unit 422 confirms that the irradiation point image newly detected on the camera B side is in the area B3 and that the irradiation point image newly detected on the camera C side is in the area C4. The area B3 and the area C4 that were the request positions of the camera pair “B, C” were deleted from the irradiation request information 411 (step S10, FIG. 10B). As a result of the update of the required position, it is determined that the irradiation point 13 has been detected at all the required positions necessary for the precision calibration of the camera pair “B, C” (“YES” determination in step S11), and the precision calibration unit 424 performs precision calibration using the irradiation point information 413 of the camera pair “B, C”, and records that the precision calibration is completed in the chain determination table 415 (step S13, FIG. 11B).

  When the precise calibration of the camera pair “B, C” is completed, there are two camera pairs, “A, B” and “B, C”, for which the precision calibration has been completed (FIG. 11B). . That is, the precision calibration is completed for the three cameras 2 with the camera IDs “A”, “B”, and “C”. Here, the cameras “A” and “C” are connected via the camera “B”, and it is determined based on the chain determination table 415 that these three cameras 2 are connected.

  At this time, since the five cameras 2 having the camera IDs “D” to “H” remain as incomplete precision calibration, it is not yet determined in step S14 that all calibrations have been completed. Since it can be determined from the chain determination table 415 that the cameras “A”, “B”, and “C” are in a connection relationship, precise calibration of the camera pair “A, C” has not been completed in the chain determination table 415. Even if it is recorded, it is determined that all calibrations have been completed. In other words, in this example, the precise calibration of the camera pair “A, C” is not an indispensable condition for the completion of the entire calibration when determining the completion of the entire calibration.

  If there is no camera pair that has not been subjected to precision calibration in step S9 (in the case of “NO” in S9), the precision calibration unit 424 skips steps S10 to S13 and confirms completion of step S14.

  If there is no camera pair that satisfies the irradiation request information 411 in step S11 (in the case of “NO” in S11), the image processing unit 42 operates as the recommended standing position calculation unit 423 to irradiate the required position. The standing position of the worker 10 suitable for point setting is calculated and presented to the worker 10 (S12).

  The recommended light source position presentation process (S12) will be described with reference to FIG.

  First, the recommended light source position calculation unit 423 selects one presentation camera pair that presents the recommended light source position on the display unit 5 from the camera pairs for which precision calibration has not been completed (S20). Specifically, the recommended light source position calculation unit 423 suppresses presentation flutter by selecting a presentation camera pair selected one frame before. As a result, the camera pair that the worker 10 is currently irradiating the common field of view is selected, so that unnecessary movement of the worker 10 can be reduced. The recommended light source position calculation unit 423 stores the presentation camera pair selected in the current frame in the storage unit 41 in preparation for selection in the next frame.

  On the other hand, when a new irradiation point 13 is not detected for a certain period from the pair of presentation cameras selected one frame before, the pair of presentation cameras is changed. That is, in this case, the recommended light source position calculation unit 423 refers to the irradiation point information 413 to select a camera ID that has not been precisely calibrated for the camera ID associated with the irradiation point image with the latest frame number.

  Next, the recommended light source position calculation unit 423 refers to the irradiation request information 411 of the presentation camera pair, and uses the camera with the more requested position among the cameras constituting the camera pair as the presentation camera that presents the recommended light source position. Select (S21). In preparation for selection in the next frame, the recommended light source position calculation unit 423 stores the presentation camera selected in the current frame in the storage unit 41. When the number of requested positions is the same, fluttering of presentation is suppressed by selecting the presentation camera selected one frame before. If one of the paired camera pairs cannot be selected as the presentation camera even if the number of requested positions and the conditions of the previous presentation camera are imposed, the camera with the younger camera ID is selected.

  Subsequently, the recommended light source position calculation unit 423 counts high-luminance pixels that are equal to or greater than the irradiation limit luminance value Ti for each requested position in the captured image of the presentation camera, and requests that a predetermined number or more (for example, 90% or more) be high-luminance pixels. The presence or absence of the position is confirmed (S22).

  If there is at least one corresponding request position (in the case of “YES” in S22), the recommended light source position calculation unit 423 determines that there is an undetectable position that is too bright to detect the irradiation point 13, and the recommended light source position calculation unit 423 performs “light shielding Please output "to the display communication unit 43 (S23), and the process returns to step S1 in FIG.

  On the other hand, if there is no undetectable position (in the case of “NO” in S22), the recommended light source position calculation unit 423 performs processes S24 to S29 and outputs a composite image. Processes S24 to S29 will be described with reference to specific examples shown in FIGS. FIG. 12 shows an example of the image areas 22b and 22c of the cameras 2B and 2C when the weighted average position weighted more as the luminance of each pixel is lower is used as the recommended light source position. FIG. 14 shows a display corresponding to FIG. 6 is an example of a display screen in a unit 5. FIG. 13 shows an example of the image areas 25b and 25c of the cameras 2B and 2C when the weighted average position weighted more as the luminance of each pixel is higher is the recommended light source position. FIG. 15 corresponds to FIG. It is an example of the display screen in the display part 5 to perform.

  When there is no undetectable position (in the case of “NO” in S22), the recommended light source position calculation unit 423 integrates the requested positions adjacent to each other among the requested positions (S24), and counts the number of requested positions after the integration. To do.

  In each image area of FIGS. 12 and 13, the difference in luminance value is expressed in three stages, and the brightness increases in the order of the vertical line area, horizontal line area, and white area. In the examples of FIGS. 12 and 13, the white outline is a window and high-brightness pixels formed by light inserted from the window, and it is expressed that the periphery of the high-brightness pixel is slightly brighter. The x mark represents the detected irradiation point 13, and an area where no x mark exists is the required position. The thick line is an integrated area where the required positions are integrated. In the example of FIG. 12, the required positions C1, C2, C3, C4, C5, C7, C8, and C9 of the camera 2C that is the presentation camera are integrated, and the count value of the integrated area becomes 1. In the example of FIG. The requested positions C4 and C8 of the camera 2C to be used are not integrated and the count value is 2.

  As shown in the example of FIG. 12, when there are many request positions adjacent to each other, the progress of the work is prioritized, and the recommended light source positions are detected so that the irradiation points 13 can be reliably detected at many request positions by irradiation from one light source position Should be calculated. Therefore, the recommended light source position calculation unit 423 determines whether or not the number of request positions after integration is one by integrating adjacent request positions (S25), and when the request positions are integrated into one, In other words, when the requested positions are adjacent to each other and form a continuous integrated area (in the case of “YES” in S25), the weighting that weights the pixel positions included in the requested positions more heavily as the brightness of each pixel is lower The average position is calculated as the recommended light source position (S26). As a result, the light source can be guided to a dark place that is close to a plurality of required positions and in which the irradiation point 13 can be easily detected, thereby improving work efficiency.

Specifically, when the required positions are integrated into one (in the case of “YES” in S25), the recommended light source position calculation unit 423 indicates that the luminance value of the pixels included in the integrated required positions is the irradiation limit luminance. A weighted average position obtained by weighting the pixel position P (x _p , y _p ) of each pixel that is less than the value Ti as the lower the luminance value I (x _p , y _p ) of the pixel is calculated by the following formula: A position Q (x _q , y _q ) is set (S26).

Here, f (I) is a monotonically increasing function whose value range is positive in the range of luminance I set in advance through experiments, and S is a set of P (x _p , y _p ).

  In the example of FIG. 12, the required positions are integrated into one for both the cameras 2B and 2C, but the camera 2C has more required positions. Therefore, as described above, the camera 2C is used as the presentation camera, and the recommended light source position is obtained on the camera 2C side having many required positions. A solid double circle 220 is a recommended light source position given by a weighted average position weighted by luminance. Incidentally, a dotted double circle 221 indicates a simple average position that is not weighted by luminance. The weighted average position is farther from the high-luminance pixel and the surrounding bright area than the simple average position.

  On the other hand, when there are few request positions adjacent to each other as shown in FIG. 13, that is, when a plurality of request positions are isolated, a bright request is obtained by guiding the light source position closer to the request position brighter than the simple average position. It is preferable to reduce the detection failure of the irradiation point 13 at the position. Therefore, the recommended light source position calculation unit 423 integrates the adjacent request positions and has a plurality of request positions after integration, that is, if the request positions are formed as one unseparated area (in S25). In the case of “NO”, a weighted average position weighted more as the luminance of each pixel is higher is calculated as the recommended light source position at the pixel position included in each requested position (S27). As a result, the light source can be guided near a bright place that is close to a plurality of required positions and where the irradiation point 13 is difficult to detect, thereby reducing failures and improving work efficiency.

Specifically, when the requested positions are not integrated into one (in the case of “NO” in S25), the recommended light source position calculation unit 423 determines that the luminance value of the pixels included in each requested position is the irradiation limit luminance value. pixel position P of each pixel is less than Ti (x _{p, y p)} intensity value I (x _{p, y p)} of the pixel in the weighted average position was higher significantly weighted calculated by the following equation, the recommended light source position Let Q (x _q , y _q ) (S27).

Here, g (I) is a monotonically increasing function having a positive value in the range of luminance I set in advance through experiments, and S is a set of P (x _p , y _p ).

  In the example of FIG. 13, the camera 2C has more required positions than the camera 2B. Therefore, as described above, the camera 2C is used as the presentation camera, and the recommended light source position is obtained on the camera 2C side having many required positions. A solid double circle 250 is a recommended light source position given by a weighted average position weighted by luminance. A dotted double circle 251 indicates a simple average position that is not weighted by luminance. Compared with the simple average position, the weighted average position is closer to the high-luminance pixel and the surrounding bright area.

  Note that, in step S25, cases may be classified according to the number of integrated request positions instead of depending on the number of areas after integration. For example, the process is divided into steps S26 and S27 depending on whether or not three or more request positions are integrated.

  As described above, the recommended light source position is calculated by using the center of gravity (weighted average position) of the pixel positions included in each of the plurality of required positions on the captured image as the recommended light source position. Here, since the possibility that the irradiation point 13 is detected is extremely low in the luminance region exceeding the irradiation limit luminance value, even if the operator 10 irradiates, it is a wasteful work. Therefore, when calculating the recommended light source position, it is preferable to exclude high luminance pixels whose luminance exceeds the irradiation limit luminance value on the captured image from the original data for calculating the average position. Thereby, when calculating the weighted average position that is weighted more as the luminance value is higher, it is avoided that the irradiation point 13 is guided to the high luminance region where the detection possibility is low, and the work efficiency of the operator 10 is improved. Can be raised. In addition, in the case of calculating a weighted average position that is weighted more as the luminance value is lower, the irradiation point 13 is guided to a darker place and is easily detected. When the average position calculation is performed by excluding the high luminance pixels exceeding the irradiation limit luminance value, the high luminance pixels are also excluded from the required position to be presented to the operator 10 on the display unit 5.

  The recommended light source position calculation unit 423 superimposes the recommended light source position and message calculated in step S26 or S27 on the image of the presentation camera, excluding high luminance pixels whose luminance value is equal to or greater than the irradiation limit luminance value Ti from the requested position. Then, a composite image is generated (S28), and the composite image is output to the display communication unit 43 (S29).

The display communication unit 43 to which the composite image is input transmits the composite image to the display unit 5, and the display unit 5 that has received the composite image displays the composite image on the display.
In the example described above, the composite image of FIG. 14 or FIG. 15 is displayed.

  FIGS. 14 and 15 are examples of display screens in the display unit 5 based on the results of FIGS. 12 and 13, respectively, and a composite image obtained by combining the recommended light source position, the requested position, and the message with the image captured by the camera 2C is schematically shown. Show. Incidentally, the recommended light source position 23 and the required position 24 are preferably displayed so as to be easily identified by color coding or the like. For example, in FIGS. 14 and 15, the recommended light source position 23 is red (horizontal line region) and the required position 24 Is displayed in green (hatched area). The request position 24 output to the display unit 5 is obtained by excluding high-luminance pixels as described above.

  Thus, the required position and the recommended light source position are presented to the worker 10, and the worker 10 irradiates the required position with the irradiator 3 from the recommended light source position. Since the irradiation point 13 formed from the recommended light source position is detected more reliably, the operator 10 can avoid the failure of not being able to detect the irradiation point 13 from the required position, and can complete the calibration work with a small work load. Become. Furthermore, the operator 10 can determine the position where the light source is actually installed or the position where the operator 10 holds the irradiator 3 by superimposing the recommended light source position on the captured image in the synthesis in step S28.

  When the recommended light source position is presented, the image processing unit 42 returns the process to step S1 in FIG. 7 to perform the process on the simultaneously captured image of the next frame.

[Operation in 3D Measurement Processing of Camera System 1]
After completing the calibration process, the camera system 1 shifts to a measurement process for three-dimensional measurement of the monitoring space.

  FIG. 16 is a flowchart showing an outline of the operation in the measurement process of the camera system 1. The operation | movement in the measurement process of the camera system 1 is demonstrated with reference to FIG.

  When the three-dimensional measurement process is executed, each camera 2 sequentially captures its own field of view and transmits the captured image to the control device 4 as in the calibration process described above. The image acquisition unit 40 of the control device 4 sequentially inputs the simultaneously captured images to the image processing unit 42 of the control device 4. In this state, the worker 10 irradiates an object such as a fixture existing in the monitoring space with the irradiator 3 while moving in the monitoring space.

  When the image processing unit 42 acquires the simultaneously shot image (S50), the image processing unit 42 operates as the request setting unit 422, divides each image into a group of adjacent pixels having similar colors, and generates divided regions (S51). Irradiation request information 411 is generated and stored in the storage unit 41 for each region as a request position for requesting detection of three or more irradiation points 13 (S52). FIG. 17 is a schematic diagram of an image area showing a setting example of the irradiation request information 411. FIG. 17 shows an example of irradiation request information 411 for the cameras 2B and 2C. As shown in the image area 26b, for the camera 2B, the required positions b3 and b4 on the surface of the fixture and the required position b1 on the wall surface. , B5, and the required position b2 are set on the floor. Further, as shown in the image area 26c, for the camera 2C, required positions c2, c3, and c4 are set on the surface of the furniture, required positions c5 and c6 are set on the wall surface, and a required position c1 is set on the floor surface. .

  After the generation of the irradiation request information 411, the image processing unit 42 repeats the processes in steps S53 to S62 every time a simultaneously shot image is input.

  First, when the image processing unit 42 obtains a simultaneous captured image (S53), the image processing unit 42 operates as an irradiation point detection unit 420, performs difference processing on each simultaneous captured image, detects the irradiation point 13 from the simultaneous captured image, and performs the irradiation. The camera ID of the image and the frame number of the image are associated with the coordinates on the image of the point 13 and stored in the irradiation point information 413 of the storage unit 41 (S54).

  Next, the image processing unit 42 operates as the request setting unit 422, and refers to the irradiation point information 413 and the irradiation request information 411 to check whether there is a request position where three or more irradiation points 13 are detected. (S55).

  If there is a corresponding required position (“YES” in S55), the image processing unit 42 operates as the object measuring unit 425, and applies the principle of triangulation to the irradiation point 13 detected at the required position. Then, three-dimensional measurement is performed (S57), and the plane of the divided area that is the required position is derived (S58). Then, the derived plane information is stored in the object measurement information 414 of the storage unit 41 (S59). At this time, the request setting unit 422 deletes the request position where the three-dimensional measurement is performed from the irradiation request information 411.

  On the other hand, when there is no requested position where three or more irradiation points 13 are detected (in the case of “NO” in S55), the image processing unit 42 operates as the recommended light source position calculation unit 423, and three irradiation points 13 exist. The recommended light source position for the required position that is less than the current position is presented (S56). The recommended light source position presentation processing in step S56 is the same as the processing described with reference to FIG. As a result, as in the case of the calibration process, the required position and the recommended light source position are presented to the operator 10, and the efficiency of the operation of detecting the irradiation point 13 from the required position is improved. In the measurement process that requires detection of a plurality of irradiation points 13 at each required position, the screen display of the display unit 5 is performed so that the operator 10 can know the number of irradiation points 13 that have already been detected for each required position. You can also. For example, the area of the requested position and the number of detections in the area can be displayed on the screen, or the color intensity can be changed for each area according to the number of detections. Thereby, not only the area displayed as the requested position is no longer the requested position but is no longer displayed on the display unit 5, the worker 10 can be presented with a more detailed work progress status, thereby improving work efficiency. I can plan.

  Here, the measurement process may include a required position where it is difficult to install the irradiation point 13. For example, when the height of the object 12 is high, it is difficult for the worker 10 standing on the floor to irradiate the required positions b3 and c2 that are the upper surface of the object 12. In such a case, the operator 10 designates a requested position that is difficult to install in the presentation image of the display unit 5 and inputs a request cancellation. In step S60, the request setting unit 422 confirms the request cancel input, and if there is an input, deletes the corresponding request position from the irradiation request information 411 (“YES” in S60 → S61). On the other hand, if there is no request release input, step S61 is skipped (in the case of “NO” in S60).

  Subsequently, the object measuring unit 425 confirms the irradiation request information 411 (S62), and ends the measurement process if the requested position information is lost (“YES” in S62 → S63). On the other hand, if the requested position remains (“NO” in S62), the image processing unit 42 returns the process to step S53 and performs the process on the simultaneously captured image of the next frame.

[Modification]
(1) In the above-described embodiment, an example in which the XY coordinates of the camera and the approximate calibration value in the optical axis direction are set using the irradiation point 13 is shown. It may be set and rough calibration may be omitted.

  (2) The irradiator 3 may be a light source that outputs pattern light such as a predetermined cross pattern instead of the above-described output of spot light. In this case, the irradiation point detection unit 420 may detect a predetermined shape point in the pattern, such as an intersection in the cross pattern, as the irradiation point 13 instead of the center of gravity of the optical image.

  (3) In the recommended light source position presentation process (S12) of the above embodiment, one of the camera pairs having a common field of view is selected as the presentation camera, and a light source position suitable for irradiation to the requested position of the presentation camera is calculated. . In this regard, a light source position suitable for irradiation to the required position of both cameras in the camera pair may be calculated and presented. As a result, it is possible to further improve the efficiency of the calibration processing and the three-dimensional measurement work that require the detection of the irradiation point 13 in common with the camera pair in the simultaneously shot image.

  For example, the recommended light source position calculation unit 423 calculates the recommended light source position Q for each of both cameras of the presentation camera pair using the formula (1) or (3), and further displays based on the recommended light source position Q of both cameras. The recommended light source position Q presented in 5 is calculated. Specifically, the midpoint or the weighted average position of the recommended light source positions Q of both cameras can be presented. When performing the weighted average, the number of required positions in each camera and the number of pixels included in the required positions can be used as weights.

  (4) As described in the above embodiment, the calibration process of the camera 2 generally requires that four or more irradiation points 13 that are not on the same straight line are detected from the simultaneously captured image. However, when some of the degrees of freedom of the position / orientation of the camera 2 are known and calibration is not necessary, the number of irradiation points 13 detected at least required for the calibration process is less than four. You can also.

  (5) In the above embodiment, the output of the recommended light source position is an image output from the display communication unit 43 to the display of the display unit 5, but the form of the output is not limited to this. For example, it can be voice output or printing on a medium such as paper.

  (6) In the above embodiment, the recommended light source position calculation unit 423 simply calculates the recommended light source position in the coordinate system on the captured image, but may calculate it more strictly in the world coordinate system. In this case, the recommended light source position calculation unit 423 converts each requested position of the presentation camera into coordinates on the reference plane 11 using the camera parameters 410 of the camera, and recommends the center position or weighted center position of the converted coordinates. Calculated as the light source position. This form is useful in a camera system in which the camera 2 is installed in a tilted posture.

  1 camera system, 2A, 2B, 2C, 2D, 2E, 2F, 2G, 2H camera, 3 irradiator, 4 control device, 5 display unit, 10 operator, 11 reference plane, 12 object, 40 image acquisition unit , 41 storage unit, 42 image processing unit, 43 display communication unit, 400 camera communication unit, 401 synchronization control unit, 410 camera parameter, 411 irradiation request information, 412 irradiation limit value, 413 irradiation point information, 414 object measurement information, 420 Irradiation point detection unit, 421 rough calibration unit, 422 requirement setting unit, 423 recommended light source position calculation unit, 424 precision calibration unit, 425 object measurement unit.

Claims (6)

One or a plurality of cameras that capture an image of a predetermined space; a control device that processes an image captured by the camera; a movable light source that irradiates the space and forms an irradiation point that can be captured by the camera; A camera system comprising:
The control device includes:
A storage unit that stores at least a plurality of irradiation request positions in the space where the irradiation points need to be formed;
A recommended light source position determination unit that determines a position where the sum of the distances from the plurality of irradiation request positions is a minimum as a recommended light source position when irradiating the plurality of irradiation request positions;
An output unit for outputting the recommended light source position;
A camera system comprising:   The camera system according to claim 1, wherein the output unit outputs the irradiation request position.   The recommended light source position determination unit extracts a luminance value of the irradiation request position in the image, and calculates the sum by weighting the distance from the irradiation request position according to the luminance value. The camera system according to claim 1 or 2. The control device further includes a request position setting unit that sets the irradiation request position and stores the irradiation request position in units of a segmented area obtained by dividing the image,
The recommended light source position determination unit, when the plurality of irradiation request positions are adjacent areas, increasing the weight for the distance as the luminance value is low,
The camera system according to claim 3. The control device further includes a request position setting unit that sets the irradiation request position and stores the irradiation request position in units of a segmented area obtained by dividing the image,
When the recommended light source position determination unit is an area where the plurality of irradiation request positions are separated, the higher the luminance value, the larger the weight for the distance,
The camera system according to claim 3. A plurality of the cameras are installed so as to have a common field of view within the space,
The storage unit stores camera parameters including the position and orientation of the camera in the space,
The control device includes:
An irradiation point detector for detecting the irradiation point from an image captured by the camera;
A calibration unit that calibrates the camera parameters based on the irradiation point; and
When the irradiation point detection unit detects an irradiation point image as the irradiation point in each of the images simultaneously captured by the plurality of cameras, the plurality of irradiation point images are at the same position in the space. Calibrating the camera parameters by applying a constraint condition with
The camera system according to any one of claims 1 to 5, wherein:

Images