JP7631427B2 - Information processing system, information processing device, and information processing program - Google Patents

Description

Embodiments of the present invention relate to an information processing system, an information processing device, and an information processing program.

Systems are known that remotely monitor objects to be checked that are in remote locations. For example, systems are known that monitor objects to be checked by sending captured images of the objects to be checked in a remote location to a management server or the like and checking the captured images on the management server side. Also known are systems that analyze the operational status of production equipment, which is an example of an object to be checked, by analyzing the captured images.

JP 2008-305259 A JP 2002-132324 A

However, in order to check the details of the situation of a remote location, it is necessary to perform analysis using captured images received at high image quality and high frame rate. However, depending on the communication conditions, degradation of image quality and low frame rate due to video delays and dropped frames may occur, making it difficult to provide a highly accurate view of the situation of the target.

The objective of the present invention is to provide an information processing system, an information processing device, and an information processing program that can provide a highly accurate confirmation of the situation of the object to be confirmed.

The information processing system of the embodiment is an information processing system including a first information processing device connected to an image capturing unit that captures a captured image, and a second information processing device that communicates with the first information processing device. The second information processing device includes a receiving unit, a converting unit, and a display control unit. The receiving unit receives the captured image and digitized data obtained by converting average values of pixel values constituting a confirmation target image area in which a confirmation target is captured and included in the captured image in accordance with a conversion rule. The converting unit converts the digitized data converted for each frame of the captured image into a situation image that represents the situation of the confirmation target captured in the confirmation target image area based on the conversion rule. The display control unit displays a superimposed image in which the situation image is superimposed on the captured image. The conversion rule is represented by a single threshold value for binarization and status information of the image area to be confirmed represented by each of the numerical values after binarization of the average value of the pixel values that make up the image area to be confirmed, the quantified data is binary data obtained by binarizing the average value of the pixel values that make up the image area to be confirmed using the threshold value represented by the conversion rule, the object to be confirmed is a flashing light, and the status information represents whether the light is off or on.

FIG. 1 is a schematic diagram of an example of a configuration of an information processing system according to an embodiment. FIG. 2 is a schematic diagram of an example of a captured image. FIG. 3 is an explanatory diagram of an example of transmission control to the second information processing device by the communication control unit. FIG. 4A is a schematic diagram of an example of a superimposed image. FIG. 4B is a schematic diagram of an example of a superimposed image. FIG. 5 is a flowchart showing an example of the flow of information processing executed by the first information processing device. FIG. 6 is a flowchart showing an example of the flow of information processing executed by the second information processing device. FIG. 7 is a diagram showing a hardware configuration.

Below, an embodiment of an information processing system, an information processing device, and an information processing program according to the present invention will be described with reference to the drawings.

Figure 1 is a schematic diagram of an example of the configuration of an information processing system 1 according to this embodiment.

The information processing system 1 includes a first information processing device 10, a second information processing device 20, and an image capturing unit 30. The first information processing device 10 and the second information processing device 20 are connected to each other so as to be able to communicate with each other via a network NW or the like. The first information processing device 10 and the image capturing unit 30 are directly connected to each other so as to be able to communicate with each other.

The first information processing device 10 is an information processing device located at a location that is the subject of remote monitoring by the second information processing device 20. The first information processing device 10 is connected to an imaging unit 30. The first information processing device 10 may be configured to include the imaging unit 30.

The photographing unit 30 is a photographing device that obtains photographed image data by photographing. The photographing unit 30 is a known digital camera or the like. In the following description, the photographed image data may be simply referred to as a photographed image. The photographing unit 30 obtains a photographed image of the confirmation object C.

The confirmation target C is an object to be confirmed by a user who operates a second information processing device 20 that is located remotely from the first information processing device 10. The confirmation target C is an element in the real space that is the object of observation for at least two types of situation changes. In other words, the confirmation target C is an element in the real space where two or more types of situation changes occur.

Specifically, for example, the confirmation target C may be a light such as a flashing LED (Light-Emitting Diode), at least a partial area of a display screen, a rotating component such as a discharge or turbine, a component such as a valve or dial that rotates or turns, water droplets such as water or oil, a specific part of a person or robot that is moving, a specific part of a remote-controlled robot, etc.

In this embodiment, an example will be described in which the confirmation target C is a blinking LED provided on equipment M located at a location that is the subject of remote monitoring. The confirmation target C is set by a user's operational instruction or the like (described in detail later).

The image capturing unit 30 captures images of the confirmation object C continuously in chronological order, and sequentially outputs the captured images obtained by capturing to the first information processing device 10 at the frame rate at which the images were captured. As a result, a captured video consisting of multiple captured images is output from the image capturing unit 30 to the first information processing device 10 at the frame rate at which the images were captured. The frame rate at which the image capturing unit 30 captures images is, for example, 30 fps to 60 fps, but is not limited to this range. It is preferable that the frame rate at which the image capturing unit 30 captures images is 30 fps or higher.

The image capturing unit 30 may be fixed in real space or configured to be movable in real space as long as it is capable of acquiring an image capturing an image of the confirmation target C. In this embodiment, an example will be described in which the image capturing unit 30 is movably arranged by being supported by a user or a moving body such as a mobile robot.

The first information processing device 10 is an information processing device connected to the image capture unit 30, and is located in a remote location relative to the second information processing device 20. A remote location means a location where the radio wave conditions during communication with the second information processing device 20 are not constant, and may include communication periods in which the radio waves are weak and communication is slow, or communication periods in which transmission delays and dropped frames of the captured images 40 being transmitted make real-time video communication difficult.

The first information processing device 10 includes a communication unit 12, a UI (user interface) unit 14, a storage unit 16, and a control unit 18. The communication unit 12, the UI unit 14, the storage unit 16, and the control unit 18 are communicatively connected via a bus or the like. The image capture unit 30 is communicatively connected to the control unit 18.

The communication unit 12 communicates with an external information processing device. In this embodiment, the communication unit 12 communicates with the second information processing device 20 via the network NW.

The UI unit 14 receives operation inputs from the user and outputs various information. The UI unit 14 includes an input unit 14A and a display unit 14B. The input unit 14A is an input device such as a keyboard that receives operation inputs from the user. The display unit 14B is a display that displays various information. The storage unit 16 stores various information. The storage unit 16 is, for example, a hard disk drive (HDD) or a memory.

The control unit 18 is a calculation unit that executes information processing.

The control unit 18 includes a setting unit 18A, an acquisition unit 18B, a generation unit 18C, and a communication control unit 18D. At least one of the setting unit 18A, the acquisition unit 18B, the generation unit 18C, and the communication control unit 18D is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by using a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

The setting unit 18A sets the confirmation target image area in which the confirmation target C is captured in the captured image and sets the irregular rules.

Figure 2 is a schematic diagram of an example of a captured image 40A. The captured image 40A is an example of a captured image 40 captured by the image capture unit 30.

The user operates the UI unit 14 while viewing the captured image 40A to select the area in which the desired confirmation target C appears as the confirmation target image area T. FIG. 2 shows a state in which the areas of two blinking LEDs appearing in the captured image 40A have been selected as the confirmation target image area T1 and the confirmation target image area T2, respectively. The confirmation target image area T1 and the confirmation target image area T2 are examples of the confirmation target image area T.

A specific area included in the captured image 40A is selected as the confirmation target image area T, and the confirmation target image area T is set, and an object captured in the confirmation target image area T is set as the confirmation target C.

Returning to FIG. 1, the explanation will be continued. The setting unit 18A stores the confirmation target image area T included in the captured image 40 in the storage unit 16.

The setting unit 18A also sets the transformation rules for the confirmation target image area T contained in the captured image 40A.

The conversion rule is a conversion rule for converting the confirmation target image area T into a numerical value that represents the state of the confirmation target C reflected in the confirmation target image area T.

The status of the confirmation target C reflected in the confirmation target image area T means the status of the confirmation target C, which is an element reflected in the confirmation target image area T and in which two or more types of status changes occur.

Specifically, if the object to be checked C is a blinking LED, the status of the object to be checked C means, for example, "on" or "off." If the object to be checked C is a component that rotates or turns, such as a valve or dial, the status of the object to be checked C means, for example, the degree of rotation or turning of the component. If the object to be checked C is a discharge, the status of the object to be checked C means, for example, the status of the discharge phenomenon. If the object to be checked C is droplets of water, oil, or the like, the status of the object to be checked C means, for example, the amount of droplets dripping or leaking.

The quantified data is data expressed by digital values that represent the status of the confirmation target C reflected in the confirmation target image area T. The digital value is, for example, a value represented by a color, shape, or a judgment result such as a numerical value, but the numerical value is not limited to an integer. For example, the quantified data is P-value data obtained by P-valueizing the confirmation target image area T in accordance with a conversion rule. P is an integer of 2 or more. For example, P matches the number of types of changing status of the confirmation target C that represent two or more types of status changes.

When the digitized data is P-value data, the conversion rule is represented by P-1 threshold values for P-value conversion and status information of the image area T to be confirmed, which is represented by each of the numerical values after P-value conversion of the image area T to be confirmed.

The setting unit 18A sets the conversion rules that represent the P-1 threshold values for P-value conversion and the status information of the image area T to be confirmed that is represented by each of the numerical values after P-value conversion of the image area T to be confirmed, by accepting instructions from the user operating the input unit 14A, etc.

For example, assume that the status change of the LED, which is the confirmation target C reflected in the confirmation target image area T, represents two types of status changes, "on" and "off." Also assume that the binary data obtained by binarizing the confirmation target image area T to a value of "0" or "1" representing the status of the confirmation target C reflected in the confirmation target image area T is used as the quantified data. In this case, for example, the setting unit 18A sets, as the conversion rule, a threshold value used to binarize the average value of the pixel values constituting the confirmation target image area T, and status information indicating that the binary value "0" represents the status of the confirmation target C "off" and the binary value "1" represents the status of the confirmation target C "on."

For example, the user operates the input unit 14A to input a conversion rule for converting the confirmation target image area T included in the captured image 40A into digitized data. In detail, the user inputs the conversion rule by inputting the situation information of the confirmation target C represented by the threshold value used for binarization and the numerical value after binarization. The setting unit 18A sets the conversion rule by accepting the conversion rule input by the user through an instruction to operate the input unit 14A.

The setting unit 18A stores the set conversion rule in the storage unit 16 in association with the set image area T to be confirmed. The setting unit 18A also transmits the set conversion rule and the image area T to be confirmed to the second information processing device 20. The second information processing device 20 receives the image area T to be confirmed and the conversion rule, stores the received conversion rule and the image area T to be confirmed in association with each other, and uses them for various processes described below.

The setting of the image area T to be checked and the conversion rules may be performed on the second information processing device 20 side, which will be described later. In this embodiment, an example will be described in which the setting of the image area T to be checked and the conversion rules is performed on the second information processing device 20 side, which will be described later.

In this case, the setting unit 18A of the first information processing device 10 transmits the captured image 40 captured by the imaging unit 30 to the first information processing device 10. The setting unit 18A then receives the confirmation target image area T and the conversion rule that have been set using the captured image 40 from the second information processing device 20. The setting unit 18A sets the confirmation target image area T and the conversion rule by correlating the received confirmation target image area T and the conversion rule and storing them in the memory unit 16.

The quantified data may be data expressed by a numerical value that represents the status of the confirmation target C reflected in the confirmation target image area T, and is not limited to P-value data in which the confirmation target image area T is P-valued according to a conversion rule. For example, the quantified data may be data in which the status of the confirmation target C reflected in the confirmation target image area T is quantified for each situation according to a predetermined rule. In this case, the conversion rule may be a conversion rule for converting the status of the confirmation target image area T into the numerical value. In detail, in this case, the conversion rule may be represented by a rule for converting the status of the confirmation target image area T into a numerical value, and the status information of the confirmation target C represented by each of the converted numerical values.

In this embodiment, a case will be described as an example in which the status change of the LED, which is the confirmation target C reflected in the confirmation target image area T, represents two types of status changes, "on" and "off." Also, a case will be described as an example in which the binary data in which the confirmation target image area T is binarized to a numerical value "0" representing the status "off" of the confirmation target C reflected in the confirmation target image area T, or a numerical value "1" representing the status "on," is used as the quantified data.

When the above conversion rules are set in both the first information processing device 10 and the second information processing device 20, the acquisition unit 18B, the generation unit 18C, and the communication control unit 18D execute the following processing.

The acquisition unit 18B acquires the captured images 40 from the image capture unit 30. As described above, the image capture unit 30 sequentially outputs the captured images 40 obtained by image capture to the first information processing device 10 at the frame rate at the time of image capture. Therefore, the acquisition unit 18B sequentially acquires multiple captured images 40 at the frame rate at the time of image capture by the image capture unit 30.

The generation unit 18C generates digitized data by converting the confirmation target image area T, which is included in the captured image 40 and contains the confirmation target C, in accordance with the conversion rules.

The generating unit 18C reads the image area T to be confirmed stored in the memory unit 16. The generating unit 18C then identifies the image area T to be confirmed included in the acquired photographed image 40 by a known image processing technique using pattern matching, feature quantities, etc. The generating unit 18C then converts the identified image area T to be confirmed included in the photographed image 40 into digitized data in accordance with the conversion rules stored in the memory unit 16.

For example, as shown in FIG. 2, assume a scene in which confirmation target image area T1 and confirmation target image area T2 included in the captured image 40 are set as confirmation target image area T. In this case, the generation unit 18C identifies each of the confirmation target image area T1 and confirmation target image area T2 included in the captured image 40 newly acquired by the acquisition unit 18B. Then, the generation unit 18C converts each of the identified confirmation target image area T1 and confirmation target image area T2 into digitized data in accordance with the conversion rules.

For example, consider a case in which the conversion rule includes a threshold value used to binarize the average pixel values that make up the confirmation target image region T, and status information indicating that the binarized value "0" represents the status of the confirmation target C as "off" and the binarized value "1" represents the status of the confirmation target C as "on."

In this case, the generation unit 18C binarizes the average pixel value of the identified confirmation target image area T included in the captured image 40 using a threshold value included in the conversion rule. Through this binarization conversion process, the generation unit 18C converts each of the confirmation target image areas T1 and T2 included in the captured image 40 into a numerical value "0" or "1" that represents the blinking status of the confirmation target C captured in each of the confirmation target image areas T1 and T2. Through this conversion process, the generation unit 18C generates numerical data for each of the confirmation target image areas T1 and T2.

Let's go back to Figure 1 and continue the explanation.

The communication control unit 18D transmits the captured image 40 acquired by the acquisition unit 18B and the digitized data generated by the generation unit 18C to the second information processing device 20.

The communication control unit 18D sequentially transmits the captured images 40 to the second information processing device 20 at a frame rate lower than that at which the captured images 40 were captured, and sequentially transmits the digitized data sequentially generated from each of the multiple captured images 40 to the second information processing device 20 at a frame rate higher than that of the captured images 40. That is, the communication control unit 18D sequentially transmits the captured images 40 to the second information processing device 20 at a frame rate lower than that at which the captured images 40 were captured, and sequentially transmits the digitized data to the second information processing device 20 at a frame rate higher than that of the captured images 40. In this embodiment, the communication control unit 18D sequentially transmits the digitized data to the second information processing device 20 at the frame rate at which the captured images 40 were captured will be described as an example.

In detail, when the communication control unit 18D starts acquiring the captured images 40 sequentially transmitted from the photographing unit 30, it transmits one Nth frame of the captured image 40 to the second information processing device 20. N is an integer equal to or greater than 1. A frame means a frame, that is, one captured image 40. The Nth frame means the Nth captured image 40 (frame, frame) when counting the captured images 40 in chronological order from the first captured image 40 captured when photographing by the photographing unit 30 begins.

The communication control unit 18D also sequentially transmits the quantified data generated by the generation unit 18C from the confirmation target image area T included in each of the captured images 40 from the Nth frame to M frames (M is an integer equal to or greater than 2) to the second information processing device 20 at the frame rate at which the captured images 40 were captured by the imaging unit 30.

Then, when the communication control unit 18D transmits the digitized data generated from the N+Mth frame of the captured image 40 to the second information processing device 20, it sets N+M+1 as the new N, and repeats the transmission of the new Nth frame of the captured image 40 to the second information processing device 20 and the transmission of the digitized data for M frames to the second information processing device 20.

By these processes, the communication control unit 18D sequentially transmits to the second information processing device 20 the quantified data of the confirmation target image area T contained in each of the M frames of the captured images 40 sequentially acquired from the capture unit 30 each time the communication control unit 18D transmits one frame of the captured image 40 acquired from the capture unit 30 to the second information processing device 20. Therefore, the communication control unit 18D transmits the captured image 40 to the second information processing device 20 at a low frame rate, and transmits the quantified data to the second information processing device 20 at a high frame rate. The value of M may be determined in advance so that the captured image 40 is transmitted to the second information processing device 20 at a low frame rate, and the quantified data is transmitted to the second information processing device 20 at a high frame rate.

Figure 3 is an explanatory diagram of an example of transmission control to the second information processing device 20 by the communication control unit 18D. In Figure 3, digitized data 42A is an example of digitized data 42 for M frames of the confirmation target image area T1 included in the captured image 40B. The digitized data 42A includes digitized data 42A1 to digitized data 42AM for M frames. The digitized data 42B is an example of digitized data 42 for M frames of the confirmation target image area T2 included in the captured image 40B. The digitized data 42B includes digitized data 42B1 to digitized data 42BM for M frames.

The communication control unit 18D sequentially transmits M frames of digitized data 42A1 through 42AM and M frames of digitized data 42B1 through 42BM generated from each of the M frames of captured images 40 sequentially acquired from the capture unit 30 to the second information processing device 20 each time it transmits one frame of captured image 40B acquired from the capture unit 30 to the second information processing device 20. For this reason, the communication control unit 18D transmits the captured image 40B to the second information processing device 20 at a low frame rate, and transmits the digitized data 42 to the second information processing device 20 at a high frame rate.

The communication control unit 18D may also reduce the resolution of the captured image 40 acquired from the image capture unit 30 to a value lower than the resolution at the time of capture, and then transmit the image to the second information processing device 20.

Returning to FIG. 1, we will continue the explanation. Next, we will explain the second information processing device 20.

The second information processing device 20 is an information processing device located at a remote location relative to the first information processing device 10 and the confirmation target C. The second information processing device 20 is operated, for example, by a user who monitors or checks the confirmation target C.

The second information processing device 20 includes a communication unit 22, a UI unit 24, a storage unit 26, and a control unit 28. The communication unit 22, the UI unit 24, the storage unit 26, and the control unit 28 are communicatively connected via a bus or the like.

The communication unit 22 communicates with an external information processing device. In this embodiment, the communication unit 22 communicates with the first information processing device 10 via the network NW.

The UI unit 24 receives operation inputs from the user and outputs various information. The UI unit 24 includes an input unit 24A and a display unit 24B. The input unit 24A is an input device such as a keyboard that receives operation inputs from the user. The display unit 24B is a display that displays various information. The storage unit 26 stores various information. The storage unit 26 is, for example, a HDD or a memory.

The control unit 28 is a calculation unit that executes information processing.

The control unit 28 includes a setting unit 28A, a receiving unit 28B, a conversion unit 28C, and a display control unit 28D. At least one of the setting unit 28A, the receiving unit 28B, the conversion unit 28C, and the display control unit 28D is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by using a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

The setting unit 28A sets the confirmation target image area T in which the confirmation target C is captured in the captured image 40, and sets the irregular rules.

The setting unit 28A acquires the captured image 40 from the first information processing device 10. The setting unit 28A displays the captured image 40 acquired from the first information processing device 10 on the display unit 24B.

The user operates the input unit 24A while viewing the displayed captured image 40 to select the area in which the desired confirmation target C appears as the confirmation target image area T. By selecting a specific area included in the captured image 40 as the confirmation target image area T, the confirmation target image area T is set, and the object that appears in the confirmation target image area T is set as the confirmation target C. The setting unit 28A stores the confirmation target image area T included in the captured image 40 in the memory unit 26.

The setting unit 28A also sets a conversion rule for the confirmation target image region T included in the captured image 40. For example, the setting unit 28A sets a conversion rule that represents P-1 threshold values for P-value conversion and status information for the confirmation target image region T represented by each of the P-value converted values of the confirmation target image region T by accepting an instruction from the user to operate the input unit 24A, etc.

As described above, in this embodiment, the setting unit 28A sets, as conversion rules, a threshold value used to binarize the average pixel values constituting the confirmation target image region T, and status information indicating that the binarized value "0" represents the status of the confirmation target C as "off" and that the binarized value "1" represents the status of the confirmation target C as "on."

For example, the user operates the input unit 24A to input a conversion rule for converting an area selected as the confirmation target image area T included in the captured image 40 into digitized data. In detail, the user inputs the conversion rule by inputting the situation information of the confirmation target C represented by the threshold value used for binarization and the numerical value after binarization. The setting unit 28A sets the conversion rule by accepting the conversion rule input by the user through an instruction to operate the input unit 24A.

The setting unit 28A stores the set conversion rule in the memory unit 26 in association with the set image area T to be confirmed. The setting unit 28A also transmits the set conversion rule and the image area T to be confirmed to the first information processing device 10. The first information processing device 10 receives the image area T to be confirmed and the conversion rule, and stores them in association with each other in the memory unit 16.

As described above, the setting of the image area T to be confirmed and the conversion rule may be performed by the setting unit 18A of the first information processing device 10. In this case, the setting unit 28A receives the image area T to be confirmed and the conversion rule set by the setting unit 18A from the first information processing device 10. The setting unit 28A sets the image area T to be confirmed and the conversion rule by associating the received image area T to be confirmed and the conversion rule and storing them in the memory unit 26.

The receiving unit 28B receives the captured image 40 and the digitized data 42 from the first information processing device 10.

In detail, the receiving unit 28B receives the captured image 40 at a frame rate lower than that at which the captured image 40 was captured by the imaging unit 30, and receives the digitized data 42 at a frame rate higher than that of the captured image 40. For example, the receiving unit 28B receives the digitized data 42 at the frame rate at which the captured image 40 was captured by the imaging unit 30.

Specifically, for example, each time the receiving unit 28B receives one frame of the captured image 40 from the first information processing device 10, it receives from the first information processing device 10 digitized data 42 that is obtained by converting the confirmation target image area T contained in each of the M frames of the captured image 40.

Each time the conversion unit 28C receives one frame of digitized data 42, it converts the received digitized data 42 into a situation image based on the conversion rules. The situation image is an image that represents the situation of the confirmation target C that is captured in the confirmation target image area T.

For example, consider a case in which the conversion rule includes a threshold value used to binarize the average pixel values that make up the target image area T, and status information indicating that the binarized value "0" represents the status of the target C reflected in the target image area T as "off," and that the binarized value "1" represents the status of the target C reflected in the target image area T as "on."

In this case, if the numerical value represented by the quantified data 42 is "1", the conversion unit 28C determines that the status of the confirmation target C reflected in the confirmation target image area T represented by the quantified data 42 is "lit". Then, the conversion unit 28C generates a status image representing the status of the confirmation target C reflected in the confirmation target image area T as "lit".

In addition, when the numerical value represented by the quantified data 42 is "0", the conversion unit 28C determines that the status of the confirmation target C reflected in the confirmation target image area T represented by the quantified data 42 is "lights off". Then, the conversion unit 28C generates a status image representing the status of the confirmation target C reflected in the confirmation target image area T as "lights off".

The status image may be any image that allows confirmation of the status of the target C reflected in the target image area T. For example, a status image that represents the "lit" status is represented by at least one combination of highlighted images such as letters, colors, frames, etc., that indicate that the status of the target C reflected in the target image area T is "lit". For example, a status image that represents the "lights off" status is represented by at least one combination of highlighted images such as letters, colors, frames, etc., that indicate that the status of the target C reflected in the target image area T is "lights off".

The conversion unit 28C may also generate a situation image that further includes identification information for each of the confirmation targets C captured in the confirmation target image area T.

The display control unit 28D generates a superimposed image by superimposing the situation image on the captured image 40, and displays it on the display unit 24B.

Figures 4A and 4B are schematic diagrams of an example of a superimposed image 46.

Fig. 4A is a schematic diagram of an example of a superimposed image 46A. The superimposed image 46A is an example of a superimposed image 46. Fig. 4A shows an example of a superimposed image 46A when the receiving unit 28B receives the captured image 40B shown in Fig. 3 and the digitized data 42A1 and digitized data 42B1, and the converting unit 28C generates a situation image 44 from each of the digitized data 42A1 and digitized data 42B1.

For example, the conversion unit 28C generates a status image 44A representing the status "lit" of the confirmation target C reflected in the confirmation target image area T1 from the numerical value "1" represented by the quantified data 42A1. The conversion unit 28C also generates a status image 44B representing the status "lit" of the confirmation target C reflected in the confirmation target image area T2 from the numerical value "1" represented by the quantified data 42B1. The status images 44A and 44B are examples of status images 44. The conversion unit 28C may generate a status image 44 including identification information of the confirmation target C reflected in each of the confirmation target image areas T1 and T2, as shown in FIG. 4A.

The display control unit 28D generates a superimposed image 46A by superimposing the situation image 44A and the situation image 44B on or near the confirmation target image area T1 and the confirmation target image area T2, respectively, contained in the captured image 40 received simultaneously with or immediately before the digitized data 42A1 and the digitized data 42B1. The display control unit 28D then displays the superimposed image 46A on the display unit 24B.

Figure 4B is a schematic diagram of an example of a superimposed image 46B. Superimposed image 46B is an example of a superimposed image 46. Figure 4B shows an example of a superimposed image 46B when the receiving unit 28B receives the captured image 40B shown in Figure 3, and the digitized data 42A2 and the digitized data 42B2, and the conversion unit 28C generates a situation image 44 from each of the digitized data 42A2 and the digitized data 42B2.

For example, the conversion unit 28C generates a situation image 44A representing the "lights off" status of the confirmation target C reflected in the confirmation target image area T1 from the numerical value "0" represented by the quantified data 42A2. The conversion unit 28C also generates a situation image 44B representing the "lights off" status of the confirmation target C reflected in the confirmation target image area T2 from the numerical value "0" represented by the quantified data 42B2. The situation images 44A and 44B are examples of situation images 44. The conversion unit 28C may generate a situation image 44 including identification information of the confirmation target C reflected in each of the confirmation target image areas T1 and T2, as shown in FIG. 4B.

The display control unit 28D generates a superimposed image 46B by superimposing the situation image 44A and the situation image 44B on or near the confirmation target image area T1 and the confirmation target image area T2, respectively, contained in the captured image 40 received simultaneously with or immediately before the digitized data 42A2 and the digitized data 42B2. The display control unit 28D then displays the superimposed image 46B on the display unit 24B.

Therefore, the display control unit 28D sequentially displays on the display unit 24B superimposed images 46 in which situation images 44 generated from each of the quantified data 42 received at the high frame rate based on the conversion rules are superimposed on the captured images 40 received at the low frame rate.

That is, the display unit 24B displays a situation image 44 that changes sequentially according to the status of the confirmation object C captured in the confirmation object image area T, which is represented by the digitized data 42 received at the high frame rate, superimposed on the captured image 40 received at the low frame rate. For example, if the confirmation object C is a blinking LED, the display unit 24B displays an image of the situation image 44, which indicates that the blinking status changes at a speed according to the high frame rate, on the captured image 40 received at the low frame rate.

As a result, the user can accurately check the status of the object of confirmation C located in a remote location by checking the superimposed image 46 displayed on the display unit 24B.

Next, an example of the flow of information processing executed by the information processing system 1 of this embodiment will be described.

In the following description, it is assumed that the setting unit 18A of the first information processing device 10 and the setting unit 28A of the second information processing device 20 store the image area T to be confirmed and the conversion rule in association with each other in the memory unit 16 of the first information processing device 10 and the memory unit 26 of the second information processing device 20, respectively.

Figure 5 is a flowchart showing an example of the flow of information processing executed by the first information processing device 10.

The acquisition unit 18B of the first information processing device 10 starts acquiring the captured images 40 that are sequentially transmitted from the imaging unit 30 (step S100).

The communication control unit 18D transmits one frame of the Nth captured image 40 received from the imaging unit 30 to the second information processing device 20 (step S102).

The generation unit 18C identifies the confirmation target image area T included in the Nth frame of the captured image 40 received from the image capture unit 30 (step S104). The generation unit 18C reads the confirmation target image area T stored in the storage unit 16, and identifies an area included in the captured image 40 that corresponds to the read confirmation target image area T, thereby identifying the confirmation target image area T included in the captured image 40.

The generating unit 18C generates digitized data 42 by converting the image area T to be confirmed identified in step S104 in accordance with the conversion rules stored in the storage unit 16 (step S106). If the conversion rules include a threshold value for binarization, the generating unit 18C uses the threshold value to binarize the average pixel value of one or more pixels that make up the identified image area T to be confirmed, thereby generating digitized data 42.

The communication control unit 18D transmits the digitized data 42 generated in step S106 to the second information processing device 20 (step S108).

The communication control unit 18D judges whether or not M frames of digitized data 42 have been transmitted to the second information processing device 20 after transmitting the captured image 40 to the second information processing device 20 (step S110). If a negative judgment is made in step S110 (step S110: No), N is counted up by 1 (step S112) and the process returns to step S104. As described above, N is an integer equal to or greater than 1, and M is an integer equal to or greater than 2.

If a positive judgment is made in step S110 (step S110: Yes), the process proceeds to step S114. In step S114, the control unit 18 judges whether or not to end the transmission process of the captured image 40 and the digitized data 42 (step S114). For example, the control unit 18 makes the judgment in step S114 by determining whether or not an end signal has been received by a user's operation instruction on the input unit 14A, etc. Also, for example, the control unit 18 may make the judgment in step S114 by determining whether or not an end signal has been received from the second information processing device 20.

If the answer is negative in step S114 (step S114: No), the process proceeds to step S116. In step S116, N is updated to N+M+1 (step S116), and the process returns to step S102. If the answer is positive in step S114 (step S114: Yes), the process ends this routine.

Next, an example of the flow of information processing executed by the second information processing device 20 of this embodiment will be described.

Figure 6 is a flowchart showing an example of the flow of information processing executed by the second information processing device 20 of this embodiment.

The receiving unit 28B of the second information processing device 20 receives the captured image 40 and the digitized data 42 from the first information processing device 10 (steps S200 and S202).

The conversion unit 28C converts the digitized data 42 received in step S202 into a situation image 44 based on the conversion rules (step S204).

The display control unit 28D generates a superimposed image 46 by superimposing the situation image 44 converted in step S204 on the captured image 40 received immediately before (step S206), and displays the generated superimposed image 46 on the display unit 24B (step S208).

Next, the receiving unit 28B judges whether or not the digitized data 42 has been received (step S210). If the judgment in step S210 is affirmative (step S210: Yes), the process proceeds to step S204. If the judgment in step S210 is negative (step S210: No), the receiving unit 28B judges whether or not the captured image 40 has been received (step S212). If the judgment in step S212 is affirmative (step S212: Yes), the process proceeds to step S202. If the judgment in step S212 is negative (step S212: No), the process proceeds to step S214. In step S214, the control unit 28 judges whether or not the process is to be terminated (step S214). The control unit 28 performs the judgment in step S214 by, for example, judging whether or not an end instruction has been input by the user operating the input unit 24A. If the judgment in step S214 is negative (step S214: No), the process proceeds to step S210. If the answer is yes in step S214 (step S214: Yes), this routine ends.

As described above, the information processing system 1 of this embodiment includes a first information processing device 10 connected to an image capturing unit 30 that captures a captured image 40, and a second information processing device 20 that communicates with the first information processing device 10. The second information processing device 20 includes a receiving unit 28B, a conversion unit 28C, and a display control unit 28D. The receiving unit 28B receives digitized data obtained by converting the captured image 40 and the confirmation target image area T containing the confirmation target C included in the captured image 40 according to a conversion rule. The conversion unit 28C converts the digitized data 42 into a situation image 44 that represents the situation of the confirmation target C captured in the confirmation target image area T based on the conversion rule. The display control unit 28D displays a superimposed image 46 in which the situation image 44 is superimposed on the captured image 40.

Here, there are cases where the second information processing device 20 needs to remotely monitor in detail the status of the confirmation target C located in a remote location from the second information processing device 20. In the conventional technology, in order to confirm the status of the confirmation target C located in a remote location in detail, it was necessary to perform analysis using the captured image 40 received in high image quality and at a high frame rate. However, in the conventional technology, depending on the communication conditions, degradation of image quality and low frame rates due to video delays and dropped frames may occur, making it difficult to communicate high-image-quality video in real time, and therefore making it difficult to provide a highly accurate confirmation of the status of the confirmation target C.

In contrast, in this embodiment, the receiving unit 28B of the second information processing device 20 receives from the first information processing device 10 the photographed image 40 and digitized data obtained by converting the confirmation target image area T, which is included in the photographed image 40 and contains the confirmation target C, in accordance with the conversion rules. The conversion unit 28C converts the digitized data 42 into a situation image 44 that represents the situation of the confirmation target C that is contained in the confirmation target image area T, based on the conversion rules. The display control unit 28D displays a superimposed image 46 in which the situation image 44 is superimposed on the photographed image 40.

Since the digitized data 42 is data obtained by digitizing the confirmation target image area T according to the conversion rules, the amount of data is smaller than that of the photographed image 40, and it is less susceptible to the effects of communication conditions than the photographed image 40. This allows the receiving unit 28B to sequentially receive the digitized data 42 at approximately the same frame rate as when the digitized data 42 was transmitted from the sender, and with the same values as when it was transmitted. The display control unit 28D then converts the digitized data 42 into a situation image 44 that represents the situation of the confirmation target C captured in the confirmation target image area T based on the conversion rules, and displays a superimposed image 46 in which the situation image 44 is superimposed on the photographed image 40.

Therefore, even if the second information processing device 20 receives a captured image 40 with a low image quality or a low frame rate due to communication conditions or the like, it can display a superimposed image 46 in which a situation image 44 showing the situation of the confirmation target C captured in the confirmation target image area T is superimposed with high accuracy.

Therefore, by checking the situation image 44 of the superimposed image 46, the user can accurately and with a high degree of precision check the situation of the confirmation target C represented by the situation image 44.

In addition, in the conventional technology, for example, when checking the blinking state of an LED, which is an example of a confirmation object C, at a remote location, a flicker phenomenon occurs in which the frame rate of the captured image 40 transmitted to a remote monitoring terminal is synchronized with the blinking speed of the LED, making it difficult for the monitoring terminal to provide an image that allows the status of the confirmation object C, such as the blinking pattern and blinking speed of the LED, to be understood. Also, when a captured image 40 captured at a high frame rate to avoid the flicker phenomenon is transmitted to a remote monitoring terminal at the same high frame rate, the amount of data is large and delays in the data transmission speed may occur.

Meanwhile, the receiving unit 28B of the second information processing device 20 of this embodiment receives from the first information processing device 10 the photographed image 40 and the digitized data obtained by converting the confirmation target image area T, which is included in the photographed image 40 and contains the confirmation target C, in accordance with the conversion rules. The conversion unit 28C converts the digitized data 42 into a situation image 44 that represents the situation of the confirmation target C that is contained in the confirmation target image area T, based on the conversion rules. The display control unit 28D displays a superimposed image 46 in which the situation image 44 is superimposed on the photographed image 40.

As a result, the information processing system 1 of this embodiment can suppress the effects of the flicker phenomenon and provide a confirmation target C that is captured in the confirmation target image area T in a manner that allows it to be confirmed accurately and with low transmission volume.

Therefore, the information processing system 1 of this embodiment can provide a highly accurate confirmation of the status of the confirmation target C.

The receiving unit 28B of the second information processing device 20 of this embodiment receives the captured image 40 at a frame rate lower than that at which the captured image 40 was captured, and receives the digitized data 42 at a frame rate higher than that of the captured image 40. The second information processing device 20 of this embodiment then displays a superimposed image 46 in which a situation image 44 generated based on each of the multiple pieces of digitized data 42 received sequentially at a high frame rate is superimposed on the captured image 40 received at the low frame rate.

Therefore, in addition to the above effects, the information processing system 1 of this embodiment can provide accurate, real-time confirmation of the status of the confirmation target C captured in the confirmation target image area T. Specifically, the information processing system 1 can provide, for example, confirmation of how the confirmation target C is moving and how the status is changing in real time.

Next, an example of the hardware configuration of the first information processing device 10 and the second information processing device 20 in the above embodiment will be described.

Figure 7 is a hardware configuration diagram of an example of the first information processing device 10 and the second information processing device 20 of the above embodiment.

The first information processing device 10 and the second information processing device 20 in the above embodiment have a hardware configuration that utilizes a normal computer, with a CPU (Central Processing Unit) 11A, a ROM (Read Only Memory) 11B, a RAM (Random Access Memory) 11C, an I/F 11D, etc., interconnected via a bus 11E.

The CPU 11A is a calculation device that controls the first information processing device 10 and the second information processing device 20 of the above embodiment. The ROM 11B stores programs and the like that realize various processes by the CPU 11A. Although a CPU is used for the explanation here, a GPU (Graphics Processing Unit) may also be used as the calculation device that controls the first information processing device 10 and the second information processing device 20. The RAM 11C stores data necessary for various processes by the CPU 11A. The I/F 11D is an interface for sending and receiving data.

In the first information processing device 10 and the second information processing device 20 of the above embodiment, the CPU 111A reads a program from the ROM 11B onto the RAM 11C and executes it, thereby realizing each of the above functions on the computer.

The programs for executing the above processes executed by the first information processing device 10 and the second information processing device 20 in the above embodiment may be stored in a HDD (hard disk drive). Also, the programs for executing the above processes executed by the first information processing device 10 and the second information processing device 20 in the above embodiment may be provided in advance in the ROM 11B.

The programs for executing the above processes executed by the first information processing device 10 and the second information processing device 20 of the above embodiment may be stored in a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disc), or flexible disk (FD) in an installable or executable format file and provided as a computer program product. The programs for executing the above processes executed by the first information processing device 10 and the second information processing device 20 of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading the programs via the network. The programs for executing the above processes executed by the first information processing device 10 and the second information processing device 20 of the above embodiment may be provided or distributed via a network such as the Internet.

Although an embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

REFERENCE SIGNS LIST 1 Information processing system 10 First information processing device 18B Acquisition unit 18C Generation unit 18D Communication control unit 28B Reception unit 28C Conversion unit 28D Display control unit

Claims (7)

An information processing system including a first information processing device connected to an image capturing unit that captures an image, and a second information processing device that communicates with the first information processing device,
The second information processing device is
a receiving unit for receiving the photographed image and digitalized data obtained by converting average values of pixel values constituting a confirmation target image area, which is included in the photographed image, in accordance with a conversion rule;
a conversion unit that converts the digitized data converted for each captured image into a situation image that represents a situation of the confirmation target captured in the confirmation target image area based on the conversion rule;
a display control unit that displays a superimposed image in which the situation image is superimposed on the captured image;
Equipped with
The conversion rule is:
a threshold value for binarization, and status information of the confirmation target image region represented by each of the binarized values of the average values of the pixel values constituting the confirmation target image region;
The digitized data is
the average value of the pixel values constituting the confirmation target image region is binarized using the threshold value represented by the conversion rule;
The object to be checked is a flashing light,
The status information indicates whether the light is off or on.
Information processing system. the receiving unit receives the captured image at a frame rate lower than that at which the captured image was captured,
receiving the digitized data at a frame rate higher than that of the captured image;
The information processing system according to claim 1 . The confirmation target image area is
The captured image includes an area in which the confirmation target is captured and which is an observation target for at least two types of situation changes.
The information processing system according to claim 1 . an acquisition unit that acquires a photographed image from the photographing unit;
a generating unit that generates digitized data by converting an average value of pixel values constituting a confirmation target image area in which a confirmation target is included in the photographed image in accordance with a conversion rule;
a communication control unit for transmitting the captured image and the digitalized data generated for each frame of the captured image;
Equipped with
The conversion rule is:
a threshold value for binarization, and status information of the confirmation target image region represented by each of the binarized values of the average values of the pixel values constituting the confirmation target image region;
The digitized data is
the average value of the pixel values constituting the confirmation target image region is binarized using the threshold value represented by the conversion rule;
The object to be checked is a flashing light,
The status information indicates whether the light is off or on.
Information processing device. a receiving unit that receives a photographed image and digitalized data obtained by converting average values of pixel values constituting a confirmation target image area, which is included in the photographed image, in accordance with a conversion rule;
a conversion unit that converts the digitized data converted for each captured image into a situation image that represents a situation of the confirmation target captured in the confirmation target image area based on the conversion rule;
a display control unit that displays a superimposed image in which the situation image is superimposed on the captured image;
Equipped with
The conversion rule is:
a threshold value for binarization, and status information of the confirmation target image region represented by each of the binarized values of the average values of the pixel values constituting the confirmation target image region;
The digitized data is
the average value of the pixel values constituting the confirmation target image region is binarized using the threshold value represented by the conversion rule;
The object to be checked is a flashing light,
The status information indicates whether the light is off or on.
Information processing device. An information processing program for causing a computer to execute the program,
acquiring a photographed image from an imaging unit;
generating digitalized data by converting an average value of pixel values constituting a confirmation target image area, which is included in the photographed image and in which the confirmation target is captured, according to a conversion rule;
transmitting the captured image and the digitalized data generated for each frame of the captured image;
Including,
The conversion rule is:
a threshold value for binarization, and status information of the confirmation target image region represented by each of the binarized values of the average values of the pixel values constituting the confirmation target image region;
The digitized data is
the average value of the pixel values constituting the confirmation target image region is binarized using the threshold value represented by the conversion rule;
The object to be checked is a flashing light,
The status information indicates whether the light is off or on.
Information processing program. An information processing program for causing a computer to execute the program,
receiving a photographed image and digitalized data obtained by converting average values of pixel values constituting a confirmation target image area, which is included in the photographed image and includes a confirmation target, according to a conversion rule;
converting the digitized data converted for each captured image into a situation image representing a situation of the confirmation target captured in the confirmation target image area based on the conversion rule;
displaying a superimposed image in which the situation image is superimposed on the captured image;
Including,
The conversion rule is:
a threshold value for binarization, and status information of the confirmation target image region represented by each of the binarized values of the average values of the pixel values constituting the confirmation target image region;
The digitized data is
the average value of the pixel values constituting the confirmation target image region is binarized using the threshold value represented by the conversion rule;
The object to be checked is a flashing light,
The status information indicates whether the light is off or on.
Information processing program.

Images