JP7618358B2 - Information processing system and three-dimensional data generation method - Google Patents

Description

The present invention relates to a technology for generating three-dimensional data showing the situation when an abnormality such as a fall occurs in a subject, such as a worker performing work at a construction site or factory, or a person requiring care living in a care facility.

When an abnormality occurs, such as an injury or accident, the cause is usually determined by a human being directly observing the situation at the site where the abnormality occurred.

Even if surveillance cameras are installed, people are forced to make judgments based on only the partial footage captured by those cameras.

However, the cause of the abnormality that occurs in the subject is not simple, and is often caused by a combination of various factors. Therefore, it is often difficult to guess or determine the cause of the abnormality just by looking at the situation at the scene.

In addition, the circumstances at the scene where the abnormality occurred in the subject are often not preserved, making detailed analysis difficult in some cases.

In addition, depending on where the surveillance cameras are installed, they may not always be able to record the exact moment an accident occurs, and often only a partial record is recorded.

Related technologies include one that generates video of an abnormality occurring as seen from any viewpoint, such as that described in Patent Document 1.

JP 2007-226515 A

The objective of this invention is to address the difficulty of investigating the cause of abnormalities occurring in subjects.

One aspect of the present invention is an information processing system that generates abnormal situation three-dimensional data that reproduces a situation when an abnormality occurs in a subject. The information processing system includes an imaging device, a state estimation device, and an information processing device, and the information processing device includes a first generation unit, a second generation unit, and an integration unit. When an abnormality occurs in the subject is detected by the state estimation device that estimates the state of the subject, the first generation unit generates environmental three-dimensional data that shows the environment around the subject based on an image captured by the imaging device, and the second generation unit generates motion three-dimensional data that shows the motion of the subject based on the image captured by the imaging device. The integration unit integrates the environmental three-dimensional data and the motion three-dimensional data to generate the abnormal situation three-dimensional data.

The present invention makes it possible to generate and reproduce three-dimensional data suitable for subject-centered analysis of the cause of an abnormality that occurs in a subject.

FIG. 1 is a configuration diagram of an information processing system according to an embodiment. 3A to 3D are diagrams illustrating an example of a state estimation device, an imaging device, three-dimensional environment data, three-dimensional operation data, and three-dimensional abnormal situation data. 4 is a flowchart showing an operation of the state estimation device. 4 is a flowchart showing the operation of the imaging apparatus. 4 is a flowchart showing an operation of the information processing device. FIG. 1 is a diagram illustrating an example of augmented reality data and virtual reality data. FIG. 1 illustrates an example of a computer for implementing an information processing device.

The following describes in detail an embodiment of the present invention with reference to the drawings.

Figure 1 is a configuration diagram of an information processing system according to an embodiment. The information processing system shown in Figure 1 includes a state estimation device 1, an imaging device 2, an information processing device 3, and a display device 4.

The state estimation device 1 is attached to the body of a subject, such as a worker performing work at a construction site or a factory, or a person requiring care living in a care facility. For example, as shown in FIG. 2(a), the state estimation device 1 is mounted on a helmet worn by the subject. Note that the attachment position of the state estimation device 1 is not particularly limited as long as it is possible to properly acquire the sensor values described below.

The state estimation device 1 shown in FIG. 1 also includes a sensor 11, such as a gyro sensor or an acceleration sensor, that is attached to a subject and outputs a sensor value that changes according to the subject's movement, and a memory 12 (first memory). The sensor value output from the sensor 11 can be periodically stored in the memory 12.

The state estimation device 1 also estimates the state of the subject based on the sensor value output from the sensor 11. For example, when the sensor value output from the sensor 11 is equal to or less than a threshold value, the state estimation device 1 determines that the subject is in a normal state (e.g., the subject is walking). When the sensor value output from the sensor 11 is greater than the threshold value, the state estimation device 1 determines that the subject is in an abnormal state (e.g., the subject has fallen). The sensor value output from the sensor 11 is not only used when detecting that an abnormality has occurred in the subject, but may also be used when estimating the subject's movement (posture), as described below.

Furthermore, when the state estimation device 1 detects that an abnormality has occurred in the subject, it transmits an abnormality detection time (detection time) indicating the time when the abnormality occurred in the subject to the imaging device 2 via a wired or wireless connection. For example, when the state estimation device 1 detects that an abnormality has occurred in the subject at 12:00:00 on March 31, 2021, it transmits "12:00:00 on March 31, 2021" as the abnormality detection time to the imaging device 2. Note that when the state estimation device 1 detects that an abnormality has occurred in the subject, it may be configured to read from the memory 12 the sensor value output from the sensor 11 in a predetermined period T based on the abnormality detection time, and transmit the read sensor value to the information processing device 3. For example, when the state estimation device 1 detects that an abnormality has occurred in the subject at 12:00:00 on March 31, 2021, the state estimation device 1 reads from the memory 12 a plurality of sensor values output from the sensor 11 during one minute from 11:59:30 on March 31, 2021 to 12:00:30 on March 31, 2021, and transmits each of the read sensor values to the information processing device 3. In this way, the predetermined period T may be any period before or after the abnormality detection time, any period after the abnormality detection time, or any period before the abnormality detection time.

Here, FIG. 3 is a flowchart showing an example of the operation of the state estimation device 1. Note that the series of operations shown in the flowchart of FIG. 3 is executed periodically by the state estimation device 1.

First, the state estimation device 1 acquires the sensor value output from the sensor 11 (step S11).

Next, when the state estimation device 1 determines that the subject is in a normal state based on the sensor value acquired in step S11 (step S12: No), it ends the current state estimation process.

On the other hand, when the state estimation device 1 detects that an abnormality has occurred in the subject based on the sensor value acquired in step S11 (step S12: Yes), it sends the abnormality detection time to the imaging device 2 (step S13).

The imaging device 2 shown in FIG. 1 is attached to the subject's body. For example, the imaging device 2 is attached to a helmet worn by the subject, as shown in FIG. 2(a). The mounting position of the imaging device 2 is not particularly limited as long as it is possible to properly capture an image, which will be described later. In addition, at least one of the functions possessed by the state estimation device 1, the imaging device 2, and the display device 4 may be configured to be included in the information processing device 3. For example, the state estimation device 1, the imaging device 2, and the information processing device 3 may be integrated to form an information processing device of the embodiment. In this case, the integrated information processing device is attached to the subject's body. In addition, by including the imaging device 2 in the state estimation device 1, the subject does not need to wear multiple devices, and the burden on the subject can be reduced.

The imaging device 2 shown in FIG. 1 also includes a camera 21 and a memory 22 (second memory). The camera 21 may be a monocular camera or a 360-degree camera, and periodically captures images (still images or moving images) in the field of view of the subject or images (still images or moving images) of the subject's surroundings, and stores the captured images in the memory 22.

In addition, when the imaging device 2 receives the abnormality detection time from the state estimation device 1, it reads out from the memory 22 images captured by the camera 21 during a predetermined period T based on the abnormality detection time, and transmits the read images to the information processing device 3. For example, when the imaging device 2 receives "12:00:00 on March 31, 2021" as the abnormality detection time, it reads out from the memory 22 images captured by the camera 21 during one minute from 11:59:30 on March 31, 2021 to 12:00:30 on March 31, 2021 (multiple still images during one minute or moving images during one minute), and transmits the read images to the information processing device 3. Note that the images captured by the camera 21 may be transferred from the imaging device 2 to the information processing device 3 via a recording medium such as a memory card.

Here, FIG. 4 is a flowchart showing an example of the operation of the imaging device 2. Note that the imaging device 2 is assumed to repeatedly capture images using the camera 21 and store them in the memory 22.

First, when the imaging device 2 determines that it has received the abnormality detection time (step S21: Yes) and that at least a predetermined period T of images (multiple still images or video images) are stored in the memory 22 (step S22: Yes), it reads out the images for the predetermined period T from the memory 22 (step S23).

Next, the imaging device 2 transmits the image read in step S23 to the information processing device 3 (step S24).

The information processing device 3 shown in FIG. 1 includes a first generation unit 31, a second generation unit 32, an integration unit 33, and a conversion unit 34.

The first generating unit 31 generates environmental three-dimensional data indicating the environment around the subject when an abnormality occurs to the subject, based on images from the imaging device 2. For example, when the subject falls, the first generating unit 31 generates environmental three-dimensional data corresponding to the "floor," "pillar," "stairs," and "pipe" that exist around the subject from a series of images, as shown in FIG. 2(b), in an arbitrary three-dimensional space. The generated environmental three-dimensional data can be generated in chronological order over a predetermined period T.

The second generating unit 32 shown in FIG. 1 generates three-dimensional motion data showing the motion (posture) of the subject when an abnormality occurs in the subject, based on the image from the imaging device 2. For example, when the subject falls, the second generating unit 32 generates three-dimensional motion data showing the skeleton of the subject and the trajectory of the skeleton when the subject falls in any three-dimensional space for each time series in a predetermined period T, as shown in FIG. 2(c). When creating three-dimensional motion data showing the skeleton of the subject and the trajectory of the skeleton, the three-dimensional motion data can be generated more accurately by using the sensor value detected by the state estimation device. For example, by using the sensor value detected by a gyro sensor, it is possible to determine whether the subject simply fell or whether he or she turned over while rotating to avoid something. Furthermore, by adding an image, the movement of the eyes when the subject falls can be detected. If the subject is in a normal state, he or she will try to take a defensive posture, but if the subject is overworked or in poor health, the overall movement will be slow, and the change in the eyes to take a defensive posture will be reduced.

The integration unit 33 shown in FIG. 1 integrates the three-dimensional environment data and the three-dimensional motion data to generate three-dimensional abnormal situation data. For example, the integration unit 33 integrates three-dimensional environment data showing the "floor", "pillar", "stairs", and "pipe" shown in FIG. 2(b) with three-dimensional motion data showing the subject's skeleton and trajectory shown in FIG. 2(c) to generate three-dimensional abnormal situation data that reproduces the situation leading up to the subject's fall in any three-dimensional space in which the "floor", "pillar", "stairs", and "pipe" are present around the subject, as shown in FIG. 2(d).

The conversion unit 34 shown in FIG. 1 converts the abnormal situation three-dimensional data into augmented reality data for display on the display device 4 using AR (Augmented Reality) technology, or into virtual reality data for display on the display device 4 using VR (Virtual Reality) technology.

Figure 5 is a flowchart showing an example of the operation of the information processing device 3.

First, the first generating unit 31 of the information processing device 3 generates three-dimensional environmental data based on images from the imaging device 2 (step S31). For example, the first generating unit 31 performs SfM (Structure from Motion) analysis and MVS (Multi-View Stereo) analysis on each image corresponding to each time series in a predetermined period T to generate three-dimensional environmental data. That is, first, the first generating unit 31 calculates the position of the imaging device 2 based on feature points in the image as SfM analysis, and generates low-density point cloud data in an arbitrary three-dimensional space from the position and feature points of the imaging device 2 using triangulation technology. Next, the first generating unit 31 generates high-density point cloud data from the low-density point cloud data as MVS analysis, and then generates a textured polygon model (three-dimensional environmental data) from the high-density point cloud data. Note that the method of generating three-dimensional environmental data is not limited to the method using SfM analysis and MVS analysis, as long as it is possible to generate three-dimensional environmental data from an image. The first generating unit 31 may also be configured to generate three-dimensional environmental data using images captured by an imaging device other than the imaging device 2, such as a fixed surveillance camera installed at a construction site, factory, nursing home, etc.

Next, the second generation unit 32 of the information processing device 3 generates three-dimensional movement data based on the images from the imaging device 2 (step S32). For example, the second generation unit 32 performs posture estimation using deep learning for each image corresponding to each time series in the predetermined period T, and generates three-dimensional movement data.

That is, the second generating unit 32 estimates the subject's skeleton and skeletal trajectory based on the learning results of the correlation between the movement of a part of the subject's body in the image, changes in the scenery other than the subject in the image, the subject's skeleton, and the subject's skeletal trajectory as posture estimation by deep learning. The second generating unit 32 may further estimate the subject's skeleton and skeletal trajectory using the sensor value output from the sensor 11 of the state estimation device 1. Alternatively, it can be said that the second generating unit 32 estimates the subject's skeleton and skeletal trajectory based on the learning results of the correlation between the movement of a part of the subject's body in the image, changes in the scenery other than the subject in the image, changes in the sensor value, the subject's skeleton, and the subject's skeletal trajectory as posture estimation by deep learning.

By estimating the subject's skeleton and skeleton trajectory in this way, it is possible to reflect not only simple falls, but also "the subject was avoiding obstacles such as reinforcements before falling," "the subject dropped something he was holding before falling," "the subject fell directly backwards," "the subject fell while rotating in an attempt to take a defensive position," etc. in the 3D motion data. Note that the method of generating the 3D motion data is not limited to a method using deep learning, as long as it is possible to generate the 3D motion data from images and sensor values. Also, the order of steps S31 and S32 may be reversed.

Next, the integration unit 33 of the information processing device 3 integrates the 3D environment data generated in step S31 and the 3D action data generated in step S32 to generate 3D abnormal situation data (step S33).

Then, the conversion unit 34 of the information processing device 3 converts the abnormal situation three-dimensional data into augmented reality data or virtual reality data (step S34).

The display device 4 shown in FIG. 1 is composed of a mobile device such as a smartphone or a tablet terminal, a personal computer, or a head-mounted display, and displays augmented reality data or virtual reality data. For example, when augmented reality data is displayed on the display of a tablet terminal as the display device 4, the display device 4 displays an image (image showing the situation at the site after the abnormality occurs) captured by a camera (not shown) mounted on the display device 4 and augmented reality data (image showing the situation at the site when the abnormality occurs) superimposed on the display device as shown in FIG. 6(a). For example, unlike FIG. 6(a), the image showing the situation at the site after the abnormality occurs does not show a "pipe". In addition, the image reproducing the situation at the site when the abnormality occurs shows a "pipe". This makes it possible to reproduce that "the subject did not fall by tripping over the gap between the stairs and the floor, but fell by stepping on a pipe that had fallen on the floor". In addition, it is possible to analyze and verify the cause of the abnormality from multiple angles or in a complex manner, centering on the subject, such as "the fall could have been prevented if there had been a handrail on the stairs when the abnormality occurred".

When the display of a personal computer is used as the display device 4 for displaying virtual reality data, the display device 4 can display BIM (Building Information Modeling) data (an image showing the situation at the site before the abnormality occurs) and virtual reality data (an image showing the situation at the site when the abnormality occurs) superimposed on each other, as shown in FIG. 6(b). In the example shown in FIG. 6(b), the image showing the situation at the site before the abnormality shows "handrails" and does not show "reinforcement materials". Also, the image showing the situation at the site when the abnormality occurs shows "reinforcement materials". This allows the subject to analyze and verify the cause of the abnormality from multiple angles or in a complex manner, such as "the reinforcement materials not included in the BIM data may have been present at the time of the abnormality, making the work flow dangerous" or "the fall may have been prevented if the handrails had been present". The display device 4 is not particularly limited as long as it is capable of displaying augmented reality data or virtual reality data.

According to the information processing device 3 of the embodiment, the environment around the subject and the subject's movements when an abnormality occurs can be reproduced using three-dimensional abnormal situation data. As a result, when analyzing the cause of an abnormality by reviewing three-dimensional abnormal situation data, it is possible to perform a more detailed analysis of what kind of abnormality occurred, what caused it, and so on, compared to analyzing the cause of the abnormality by reviewing the scene after the abnormality occurred or a recorded video at the time of the abnormality. In other words, according to the information processing system of the embodiment, it is possible to generate three-dimensional abnormal situation data suitable for multifaceted analysis of the cause of an abnormality occurring to a subject, with the subject at the center.

In addition, according to the information processing device 3 of the embodiment, the environment around the subject when the abnormality occurs can be reproduced using three-dimensional abnormality data, so that the situation when the abnormality occurs can be analyzed in detail even if the on-site environment is not preserved after the abnormality occurs.

In addition, according to the information processing device 3 of the embodiment, the situation when an abnormality occurs can be reproduced using three-dimensional abnormal situation data generated based on an image captured by the imaging device 2 attached to the subject's body. Therefore, compared to confirming the situation when an abnormality occurs using an image captured by a fixed camera such as a surveillance camera that is prone to blind spots, it is possible to avoid analyzing the cause of the abnormality using a partial recorded image and to analyze the situation when an abnormality occurs in detail.

Figure 7 shows an example of a computer for implementing the information processing device 3.

The computer 70 shown in FIG. 7 includes a processor 71, a memory 72, a storage device 73, a reader 74, a communication interface 75, and an input/output interface 76. The processor 71, the memory 72, the storage device 73, the reader 74, the communication interface 75, and the input/output interface 76 are connected to each other via a bus 77.

The processor 71 is configured as a single processor, a multiprocessor, a multicore, etc. The processor 71 also reads out a program stored in the storage device 73 and uses the memory 72 to execute a program describing the procedure of the flowchart shown in FIG. 5, thereby providing some or all of the functions of the first generation unit 31, the second generation unit 32, the integration unit 33, and the conversion unit 34.

Memory 72 may be a semiconductor memory or the like, and may include a RAM (Random Access Memory) area and a ROM (Read Only Memory) area.

The storage device 73 is a semiconductor memory such as a hard disk or flash memory, or an external storage device.

The reader 74 accesses the removable storage medium 78 according to instructions from the processor 71. The removable storage medium 78 is realized by a semiconductor device, a medium where information is input and output by magnetic action, a medium where information is input and output by optical action, etc. An example of a semiconductor device is a USB (Universal Serial Bus) memory, etc. An example of a medium where information is input and output by magnetic action is a magnetic disk, etc. An example of a medium where information is input and output by optical action is a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc), a Blu-ray Disc (Blu-ray is a registered trademark), etc.

The communication interface 75 communicates with other devices according to instructions from the processor 71. For example, the information processing device 3 collects information from the state estimation device 1 and the imaging device 2 via the communication interface 75.

The input/output interface 76 is, for example, an interface between an input device and an output device. The input device is a device such as a keyboard, mouse, or touch panel that accepts instructions from a subject or an administrator who checks the augmented reality data or virtual reality data. The output device is a display device such as a display, or an audio device such as a speaker.

Each program according to the embodiment is provided to the information processing device 3 in the following form, for example.
(1) It is pre-installed in the storage device 73.
(2) Provided by a removable storage medium 78.
(3) Provided from a server such as a program server.

The hardware configuration of the computer 70 for realizing the information processing device 3 described with reference to FIG. 7 is an example, and the embodiment is not limited to this. For example, part of the above-mentioned configuration may be deleted, or new configuration may be added. In another embodiment, for example, part or all of the functions of the above-mentioned information processing device 3 may be implemented as hardware such as an FPGA, a System-on-a-Chip (SoC), an Application Specific Integrated Circuit (ASIC), or a PLD.

The above describes the embodiments. However, the embodiments are not limited to the above embodiments, and should be understood to include various modified and alternative forms of the above embodiments. For example, it will be understood that the various embodiments can be embodied by modifying the components without departing from the spirit and scope of the embodiments. It will also be understood that various embodiments can be implemented by appropriately combining multiple components disclosed in the above-mentioned embodiments. Furthermore, it will be understood by those skilled in the art that various embodiments can be implemented by deleting some components from all the components shown in the embodiments, or by adding some components to the components shown in the embodiments.

REFERENCE SIGNS LIST 1 State estimation device 2 Imaging device 3 Information processing device 4 Display device 11 Sensor 12 Memory 21 Camera 22 Memory 31 First generation unit 32 Second generation unit 33 Integration unit 34 Conversion unit

Claims (6)

The present invention includes an imaging device, a state estimation device, and an information processing device,
The information processing device includes a first generation unit, a second generation unit, and an integration unit,
When the state estimation device that estimates the state of a subject detects that an abnormality has occurred in the subject,
The first generation unit generates three-dimensional environmental data indicating an environment around the subject based on an image captured by the imaging device,
The second generation unit generates three-dimensional motion data indicating a motion of the subject based on an image captured by the imaging device,
The integrating unit integrates the three-dimensional environment data and the three-dimensional action data to generate three-dimensional abnormal situation data. In claim 1, the state estimation device and the imaging device are an information processing system worn by the subject. In claim 1 or 2,
the state estimation device includes a sensor and a first memory, and periodically stores a sensor value detected by the sensor in the first memory;
When the state estimation device that estimates the state of the subject detects that an abnormality has occurred in the subject,
The state estimation device notifies the imaging device of a detection time indicating the time when an abnormality occurred in the subject , and transmits the sensor value for a set period from the detection time to the information processing device. In any one of claims 1 to 3,
the imaging device includes a camera and a second memory, and periodically stores images captured by the camera in the second memory;
When the state estimation device that estimates the state of the subject detects that an abnormality has occurred in the subject,
The imaging device transmits to the information processing device the images for a set period from a detection time indicating the time when an abnormality occurred in the subject , which was notified by the state estimation device. In any one of claims 1 to 4,
The information processing system includes a display device,
The information processing device further includes a conversion unit,
The conversion unit converts the abnormal situation three-dimensional data into augmented reality data or virtual reality data,
The display device displays the converted augmented reality data or virtual reality data. A three-dimensional data generation method for generating abnormal situation three-dimensional data that reproduces a situation when an abnormality occurs in a subject in an information processing device, comprising:
The information processing device includes:
When a state estimation device that estimates a state of the subject detects that an abnormality has occurred in the subject,
generating three-dimensional environmental data indicating an environment around the subject based on an image captured by an imaging device;
generating three-dimensional motion data indicating a motion of the subject based on an image captured by the imaging device;
generating the abnormal situation three-dimensional data by integrating the environmental three-dimensional data and the action three-dimensional data;
A three-dimensional data generating method comprising the steps of:

Images