JP7793333B2 - Imaging system, control method, program, and storage medium - Google Patents

Description

Related to imaging systems, control methods, programs, and storage media.

For surveillance cameras and other applications, it is desirable to expand the dynamic range to capture images in environments with large differences in brightness or different lighting conditions.

Patent document 1 discloses a technology that expands the dynamic range by controlling the exposure time for each of multiple pixel blocks on the imaging surface.

WO2017/018188

The problem that this invention aims to solve is to improve the visibility of pixel blocks for which exposure parameters such as exposure time can be set.

In order to solve the above problem, an imaging system according to one aspect of the present invention includes an imaging unit in which pixel blocks consisting of a plurality of pixels are arranged two-dimensionally on an imaging surface, and which is capable of individually setting exposure parameters for the pixel blocks; a determination unit that determines a difference between a first exposure parameter and a second exposure parameter so that the difference is equal to or greater than a predetermined value; an exposure control unit that controls imaging by alternately setting the first exposure parameter and the second exposure parameter determined by the determination unit in the pixel blocks two-dimensionally ; and a display unit that displays an image captured by the imaging unit.

This invention improves the visibility of pixel blocks for which exposure parameters such as exposure time can be set.

FIG. 1 is a diagram showing an example of the configuration of an imaging system according to a first embodiment. FIG. 1 is a diagram showing an example of the configuration of an imaging apparatus according to a first embodiment. FIG. 2 is a diagram showing an example of the configuration of a client apparatus according to the first embodiment. FIG. 1 is a diagram showing an example of the functional configuration of an imaging system according to a first embodiment. 1A and 1B are diagrams showing an example of an imaging surface and a captured image in an imaging device according to a first embodiment. FIG. 2 is a diagram showing an example of the operation of the imaging system according to the first embodiment. FIG. 2 is a diagram schematically showing an example of a pixel block on an imaging surface of the imaging device according to the first embodiment. FIG. 2 is a view showing an example of an image captured by the imaging system according to the first embodiment. FIG. 2 is a diagram schematically showing an example of a pixel block on an imaging surface of the imaging device according to the first embodiment. 3A and 3B are views showing an image captured by the imaging system according to the first embodiment and a designated area.

The following describes in detail the modes for implementing the present invention, with reference to the accompanying drawings. The embodiment described below is one example of a means for realizing the present invention, and should be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and various conditions. The present invention is not limited to the following embodiment. Furthermore, it is also possible to configure the present invention by appropriately combining parts of each of the embodiments described below.

<Embodiment 1>
(System configuration)
1 is a diagram showing an example of hardware constituting an image capturing system according to this embodiment. The image capturing system 100 includes an image capturing apparatus 101, a network 102, a client apparatus 103, a display unit 104, and an input unit 105.

The imaging device 101 can communicate with the client device 103 via the network 102. The imaging device 101 captures an image of a subject to generate an image, and transmits the captured image to the client device 103 via the network 102. A display unit 104 and an input unit 105 are connected to the client device 103. Images sent from the imaging device 101 are output to and displayed on the display unit 104 via the client device 103. The input unit 105 includes a keyboard, mouse, etc., and is used to operate the client device 103. Operations on the client device 103 include setting the imaging conditions of the imaging device 101 via the network 102 and operating PTZ (Pan Tilt Zoom).

In this embodiment, the client device 103, display unit 104, and input unit 105 are separate entities, but the client device 103, display unit 104, and input device 105 may be integrated into one configuration, such as a notebook PC with a touch panel display. Furthermore, they do not need to be connected via the network 102, and the surveillance camera 101 and client device 103 may be directly connected. Furthermore, the imaging device 101, client device 103, display unit 104, and input unit 105 may all be integrated into one configuration, such as a consumer camera with a touch panel display.

(Device configuration)
2 is a diagram showing an example of the device configuration of the imaging device according to this embodiment. The imaging device 101 has an imaging unit 201, a network I/F 202, a CPU (Central Processing Unit) 203, a RAM (Random Access Memory) 204, and a ROM (Read Only Memory) 205. The imaging optical system 200 is detachably provided on the surveillance camera 101.

The imaging optical system 200 is a lens that focuses light from a subject onto the imaging surface of the imaging element 201a (described later), and is composed of, for example, a zoom lens, a focus lens, and a blur correction lens.

In this embodiment, the imaging device 101 and the imaging optical system 200 are provided as separate, detachable units, but the imaging device 101 and the imaging optical system 200 may also be integrated into one unit, as in a lens-integrated camera.

The imaging unit 201 captures an image of a subject using the imaging optical system 200 and generates an image. The imaging unit 201 has an image sensor 201a, an amplifier 201b, and an A/D converter 201c.

The image sensor 201a photoelectrically converts light from the subject collected by the imaging optical system 200 and outputs an image signal. Pixel blocks consisting of multiple pixels (e.g., 128 x 128 pixels) are arranged two-dimensionally on the imaging surface of the image sensor 201a (image sensor 201). Exposure parameters can be set individually for each pixel block. Here, exposure parameters are parameters related to exposure, and include at least one of exposure time, analog gain (described below), and exposure value. The image sensor 201a is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

The amplifier 201b amplifies and outputs the electrical signal output from the image sensor 201a. An amplifier 201b is provided for each pixel, and its signal amplification rate (analog gain) can be set and changed for each pixel block of the image sensor 201a.

The A/D conversion unit 201c converts the analog image signal output from the amplifier 201b into a digital signal.

The network I/F 202 is an interface for transmitting image data, which is a digital signal output from the A/D conversion unit 201c, to the client device 103 via the network 102. The image data transmitted thereby may be image data that has been encoded using image processing into a specified file format such as Motion JPEG, H264, or H265. The image data may be stored in an internal storage device such as the RAM 204 or ROM 205 described below, or in a removable storage medium such as an SD card.

The CPU 203 is a central processing unit that controls the imaging device 101. The RAM 204 temporarily stores computer programs executed by the CPU 203. The RAM 204 provides a work area used by the CPU 203 when executing processing. The RAM 205 also functions as a frame memory and a buffer memory.

The ROM 205 stores programs and other data used by the CPU 203 to control the imaging device 101.

Figure 3 shows an example of the device configuration of a client device according to this embodiment. The client device 103 is an information processing device having a CPU 301, RAM 302, ROM 303, input I/F 304, output I/F 305, and network I/F 406.

CPU 301 is a central processing unit that controls the client device 103.

RAM 302 temporarily stores programs and the like used by CPU 301 to control client device 103. RAM 302 provides a work area used by CPU 301 when executing processing. RAM 302 also functions as frame memory and buffer memory.

ROM 303 stores programs and other data that the CPU 301 uses to control the client device 103.

The input I/F 304 is an interface for connecting the input unit 105 and the client device 103, and accepts operations for the client device 103 input by the user via the input unit 105.

The output I/F 306 is an interface for connecting to the display unit 104 and outputs the image output from the imaging device 101.

The network I/F 306 is an interface that connects to the surveillance camera 101 via the network 102 and is used to input operation information for the surveillance camera 101 and accept images output from the surveillance camera 101.

(Functional configuration)
Fig. 4 is a diagram showing an example of the functional configuration of the imaging system according to this embodiment. The imaging system 100 has an imaging unit 201, a determination unit 401, an exposure control unit 402, and a display unit 104. The functions of each unit shown in Fig. 4 can be achieved by the CPU 203 (or CPU 301) reading and executing a program stored in the RAM 204 (or 302) or the ROM 205 (or ROM 303).

The imaging unit 201 has pixel blocks consisting of multiple pixels arranged two-dimensionally on the imaging surface, and exposure parameters can be set individually for each pixel block.

The determination unit 401 determines the difference between the first exposure parameter and the second exposure parameter to be equal to or greater than a predetermined value. The first exposure parameter and the second exposure parameter are parameters including at least one of the exposure time, gain, and exposure value. Therefore, the difference between the first exposure parameter and the second exposure parameter is the difference in exposure time, gain, and exposure value, for example. The predetermined value is a threshold, and is a different threshold for each of the exposure time, gain, and exposure value. In other words, at least one of the exposure time threshold, gain threshold, and exposure value threshold is the predetermined value. The predetermined value is set by the user or designer.

The exposure control unit 402 controls the imaging by alternately setting the first exposure parameter and the second exposure parameter determined by the determination unit 401 to multiple pixel blocks.

The display unit 104 displays the image captured by the exposure control unit 402 controlling the imaging unit 201.

(Operation description)
The operation of the imaging system according to this embodiment will be described below with reference to FIGS.

Figure 5 shows an example of the imaging surface of the image sensor 201a and an image captured. Figure 5(a) is a schematic diagram of the imaging surface 501, and Figure 5(b) is a diagram showing an example of an image captured by the imaging device 101. As shown in Figure 5(a), pixel blocks 502 consisting of multiple pixels are arranged two-dimensionally on the imaging surface 501. Figure 5(b) shows an image displayed on the display unit 104, illustrating the state before operation of the imaging system according to this embodiment begins. The position of each pixel block is expressed in a two-dimensional coordinate system, with the block in the i-th column and j-th row from the bottom left being represented as (i, j). Therefore, as shown in Figure 5(a), the coordinates of the bottom leftmost pixel block are represented as (1, 1), the adjacent pixel block to the right is represented as (2, 1), and the adjacent pixel block above is represented as (1, 2).

Figure 6 is a flowchart showing an example of the operation of the imaging system according to this embodiment. The operation of the flowchart shown in Figure 6 begins when the CPU 203 (or CPU 301) reads and executes a program stored in the RAM 204 (or 302) or ROM 205 (or ROM 303). Therefore, each operation of the imaging system 100 may be performed by the imaging device 101 or the client device 103.

In step S601, the area in which the pixel block boundaries are to be displayed is specified. The area in which the pixel block boundaries are to be displayed is specified by a specifying unit (not shown) when an area is input from the input unit 105. If no area is input from the input unit 105, the area in which the pixel blocks of the entire screen are to be displayed is determined. In this embodiment, a case in which pixel block boundaries are displayed across the entire image is described, but it is also possible to display the pixel block boundaries of an area specified by the user via the input unit 105.

In step S602, the determination unit 401 determines the difference between the first exposure time (first exposure parameter) and the second exposure time (second exposure parameter) to be equal to or greater than a predetermined value. Figure 7 shows an example of a screen for setting the difference between the first exposure time and the second exposure time. In Figures 7(a) and 7(b), a predetermined value (threshold value) can be set by dragging the knob 702 on the slider 701 with a finger, a mouse, or the like. The determination unit 401 determines the difference between the first exposure time and the second exposure time to be equal to or greater than the set predetermined value. In this embodiment, the difference between the first exposure time and the second exposure time is determined to be a predetermined value. Note that the first exposure time and the second exposure time can be any value as long as the difference between them is equal to or greater than the predetermined value. For example, if the difference in exposure time is set to 1/60, the first exposure time may be 1/30 and the second exposure time 1/60, or the first exposure time may be 1/60 and the second exposure time 1/30.

In step S603, the exposure control unit 402 controls the imaging by alternately setting the first exposure parameter and the second exposure parameter to pixel blocks on the imaging surface. Figure 7(a) shows a schematic example of a case where the predetermined value is set small, and Figure 7(b) shows a schematic example of a case where the predetermined value is set large.

In step S604, the display unit 104 displays the image captured by the imaging unit 201 under the control of step S603. Figure 8 shows an example of an image displayed by the display unit 104. Figure 8(a) shows the image before the start of operation, and Figure 8(b) shows the image after the operation of step S604 is completed. In Figure 8(a), it is difficult to see the boundaries of the pixel blocks for which exposure parameters can be set. On the other hand, in Figure 8(b), the visibility of the boundaries of the pixel blocks is improved. This is effective when the user wants to assign an appropriate pixel block to a specific subject (e.g., a subject of interest). If the boundaries of the pixel blocks can be seen, it is possible to change the size of the pixel blocks or adjust the imaging position so that the subject of interest does not straddle the boundaries of the pixel blocks or so that the outline of the subject fits within the pixel blocks.

As described above, the imaging system according to this embodiment can improve the visibility of pixel blocks for which exposure parameters such as exposure time can be set.

A further explanation of a modified example of this embodiment is provided. In this embodiment, the case has been described in which the difference between the first exposure parameter and the second exposure parameter is determined to be equal to or greater than a certain value. However, the determination unit 401 may also determine the difference between the first target luminance and the second target luminance to be equal to or greater than a predetermined value. Here, the target luminance is the target value of the luminance output from the pixel block. Figure 9 shows an example of a screen for setting the luminance difference as a threshold. As shown in Figure 9(a), as the knob 702 is raised, the difference between the first target luminance and the second target luminance increases, making the boundaries of the pixel blocks more clearly visible. On the other hand, as shown in Figure 9(b), lowering the knob can improve the visibility of the pixel blocks without reducing the visibility of the image. The exposure control unit 402 controls the exposure parameters for each pixel block so that the luminance of the pixel block alternates between the first target luminance and the second target luminance.

Figure 10 shows an example of a case in which pixel blocks are visible only in a partial area of an image in the imaging system of this embodiment. As shown in Figure 10, the user can specify the area in which they want pixel blocks to be displayed by dragging using the input unit 105. In step S601, if a specified area 1001 is specified, an indication unit (not shown) designates the specified area 1001 as the area in which pixel blocks are to be displayed. The exposure control unit 402 controls the imaging by alternately setting the first exposure parameter and the second exposure parameter to the pixel blocks corresponding to the area designated by the indication unit. This makes it possible to make visible only the area in which the user has viewed the pixel blocks.

<Other embodiments>
The present invention can be realized by a process of reading and executing a program that realizes one or more functions of the above-described first embodiment. This program is supplied to a system or device via a network or a storage medium, and is read and executed by one or more processors in the computer of the system or device. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

100 Imaging system 104 Display unit 201 Imaging unit 401 Determination unit 402 Exposure control unit

Claims (16)

an imaging unit in which pixel blocks each consisting of a plurality of pixels are arranged two-dimensionally on an imaging surface, and exposure parameters can be set individually for each pixel block;
a determination unit that determines a difference between the first exposure parameter and the second exposure parameter to be equal to or greater than a predetermined value;
an exposure control unit that performs control to capture an image by alternately setting the first exposure parameter and the second exposure parameter determined by the determination unit to the pixel block in the two-dimensional pattern;
a display unit that displays an image captured by the imaging unit;
An imaging system having: The imaging system of claim 1, wherein the exposure parameters include at least one of exposure time, gain, and exposure value. the determination unit determines a difference between a first target luminance and a second target luminance to be equal to or greater than a predetermined value;
the exposure control unit performs control to individually set the exposure parameters for each of the pixel blocks arranged in the two-dimensional shape and capture an image so that the first target luminance and the second target luminance are alternately output from the pixel blocks in the two-dimensional shape.
3. The imaging system according to claim 1, wherein the imaging system includes: a first optical fiber; The imaging system described in any one of claims 1 to 3, further comprising an input unit for inputting the predetermined value. further comprising an indicator for indicating a predetermined area from the image displayed on the display unit;
the exposure control unit performs control to capture an image by alternately setting the first exposure parameter and the second exposure parameter in the two-dimensional pattern to pixel blocks corresponding to the area designated by the designation unit.
5. The imaging system according to claim 1, wherein the imaging system includes: a first lens; The imaging system described in any one of claims 1 to 5, characterized in that the determination unit determines the first exposure parameter and the second exposure parameter. 4. The imaging system according to claim 3 , wherein the determination unit determines the first target luminance and the second target luminance. an imaging step of individually setting exposure parameters for pixel blocks each made up of a plurality of pixels arranged two-dimensionally on an imaging surface and capturing an image;
a determining step of determining a difference between the first exposure parameter and the second exposure parameter to be equal to or greater than a predetermined value;
an exposure control step of performing control to capture an image by alternately setting the first exposure parameter and the second exposure parameter determined in the determination step to the pixel block in the two-dimensional pattern;
a display step of displaying the image captured in the exposure control step;
A control method for an imaging system having the above features. The control method for an imaging system described in claim 8, wherein the exposure parameters include at least one of exposure time, gain, and exposure value. In the determining step, a difference between the first target luminance and the second target luminance is determined to be equal to or greater than a predetermined value;
the exposure control step performs control to capture an image by individually setting the exposure parameters for each of the pixel blocks arranged in the two-dimensional shape so that the first target luminance and the second target luminance are alternately output from the pixel blocks in the two-dimensional shape.
10. The method for controlling an imaging system according to claim 8 or 9. The control method for an imaging system described in any one of claims 8 to 10, further comprising an input step of inputting the predetermined value. The method further includes a designation step of designating a predetermined area from the image displayed in the display step,
In the exposure control step, the first exposure parameter and the second exposure parameter are alternately set in the two-dimensional pattern to pixel blocks corresponding to the area designated in the designation step, and control is performed to capture an image.
12. The method for controlling the imaging system according to claim 8. A control method for an imaging system described in any one of claims 8 to 12, characterized in that the determination step determines the first exposure parameter and the second exposure parameter. 11. The method for controlling an imaging system according to claim 10 , wherein the determining step determines the first target luminance and the second target luminance. A program for causing a computer to execute the imaging system control method described in any one of claims 8 to 14. A computer-readable storage medium storing the program described in claim 15.