JP7801976B2 - Subject imaging device, subject imaging method, and program - Google Patents

Description

This disclosure relates to a subject imaging device, a subject imaging method, and a program.

When photographing subjects such as animals, there is a need to attract the animal's attention and capture the animal in the best possible condition. However, because animals move freely regardless of human will, it has been difficult to keep the animal in the right condition for photography. Even when the subject is a human, it has been difficult to keep an animal in the right condition for photography, for example, with infants.

For example, Patent Document 1 below discloses a photography support device that supports photographers in photographing living organisms, and includes: a living organism-related information database that stores living organism-related information about living organisms, including their habitat locations; a location information acquisition means that acquires the location information of the photographer photographing the living organisms; a living organism-related information extraction unit that extracts living organism-related information about living organisms living around the photographer's current location, as indicated in the acquired location information; and a living organism-related information transmission means that transmits the extracted living organism-related information.

JP 2012-199885 A

However, while the photography support device described in Patent Document 1 above can determine the habitat of living creatures, it cannot put the subject in a state suitable for photography.

In consideration of the above-mentioned problems, the present disclosure aims to provide a subject imaging device, subject imaging method, and program that can capture images of subjects such as people and animals in a state suitable for photography.

In order to solve the above-mentioned problems and achieve the objectives, the subject imaging device according to the present disclosure comprises an imaging unit that captures video of a subject, including people and animals; a discrimination unit that determines the type of subject based on the video of the subject captured by the imaging unit; a generation unit that generates sound information that prompts a reaction from the subject by changing the variation of the sound according to the type of subject determined by the discrimination unit; an output unit that outputs the sound information generated by the generation unit; an acquisition unit that acquires the video of the subject when the sound information is output; a discrimination unit that determines the reaction of the subject to the output sound information based on the video of the subject when the sound information is output; and a learning unit that learns the relationship between the characteristics of the output sound information and the reaction of the subject, and the generation unit generates the sound information by changing the variation of the sound based on the relationship between the characteristics of the output sound information and the reaction of the subject.

According to one aspect of the embodiment, it is possible to provide a subject imaging device, subject imaging method, and program that can capture images of subjects such as people and animals in a state suitable for photography.

FIG. 1 is a diagram illustrating an example of information processing according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of a subject imaging apparatus according to the embodiment. FIG. 3 is a flowchart showing the flow of information processing according to the embodiment. FIG. 4 is a hardware configuration diagram showing an example of a computer that realizes the functions of the subject imaging device.

The following describes in detail, with reference to the drawings, aspects (hereinafter referred to as "embodiments") of the subject imaging device, subject imaging method, and program according to the present application. Note that the subject imaging device, subject imaging method, and program according to the present application are not limited to these embodiments.

(Embodiment)
1. Information Processing According to the Embodiment
1-1. Example of information processing according to the embodiment
First, an example of information processing according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of information processing according to an embodiment. First, an overview of the information processing according to the embodiment will be described, and then each process will be described in detail.

Figure 1 shows a process in which the subject imaging device 10 determines the type of animal A1 serving as the subject, generates sound information according to the determined type of subject, emits the generated sound information toward the animal A1, and captures an image of the animal A1 that reacts to the sound. Below, an example of information processing according to this embodiment will be described in detail step by step using Figure 1.

First, the subject imaging device 10 acquires imaging data including video of the subject (step S1). For example, the subject imaging device 10 acquires imaging data including video of the subject, such as a person or animal, captured by the imaging unit 13 (described below) by reading it from the storage unit 12 (described below).

Next, the subject imaging device 10 determines the type of subject based on the imaging data (step S2). For example, the subject imaging device 10 performs an object detection process, which detects objects from the imaging data and classifies the detected objects, based on the imaging data, to determine the type of subject from the imaging data.

Next, the subject imaging device 10 generates sound information that prompts a response from the subject by varying the sound variations depending on the type of subject (step S3). For example, if the subject is a child, the subject imaging device 10 generates sound information that can be expressed using onomatopoeia such as the squeals and squeals of toys, as well as sound information including the voices of trains, cars, and anime characters. If the subject is a dog or cat, the subject imaging device 10 generates sound information that can be expressed using onomatopoeia such as the rustling of toys and plastic bags, as well as the voice of the owner, birds, insects, animals, the sound of opening a can, the sound of opening a bag, and the sound of objects rustling.

Next, the subject imaging device 10 outputs the generated sound information (step S4). For example, the subject imaging device 10 outputs the generated sound information as audio from a speaker toward the subject.

Next, the subject imaging device 10 captures an image of the subject when the sound information is output (step S5). For example, the subject imaging device 10 outputs sound information toward the subject, captures an image of the subject reacting to the sound information, and acquires the captured image.

This makes it possible to put the subject in a state suitable for photography by emitting sound, allowing the subject to be photographed in a state suitable for photography. Therefore, it is possible to provide a subject imaging device 10 that can photograph subjects such as people and animals in a state suitable for photography.

1-2. Other Examples of Information Processing According to the Embodiment
The subject imaging device 10 generates sound information by changing the variation of sound according to the type of subject as well as the context including the subject's mood, time, place, surrounding conditions, and whether or not there is noise.

This information processing will be explained step by step. First, the subject imaging device 10 executes the same processing as steps S1 to S2 shown in Figure 1. The processing from steps S1 to S2 is the same as the processing from steps S1 to S2 described above, so its explanation will be omitted.

Next, the subject imaging device 10 generates sound information that changes the sound variation depending on the type of subject and prompts a response from the subject (step S3). In this case, the subject imaging device 10 generates sound information by changing the sound variation depending on the context, including the subject's mood, time, location, surrounding circumstances, and whether or not there is noise, in addition to the type of subject. Note that here, sound variation refers to the content of the sound, including the volume, tone, length, and type of sound. For example, if the location is a "park" and the surrounding circumstances are "children playing," the subject imaging device 10 generates sound information with a medium volume and a bright tone so as not to disrupt the cheerful atmosphere.

Next, the subject imaging device 10 executes the same processing as steps S4 to S5 shown in FIG. 1. Since the processing from steps S4 to S5 is the same as the processing from steps S4 to S5 described above, a description thereof will be omitted.

This allows the sound output to be changed depending on the context, such as the surrounding situation. Therefore, by emitting sound while taking into consideration the surrounding situation, it is possible to put the subject in an optimal state for shooting, and capture the subject in an optimal state for shooting.

2. Configuration of the object imaging device
Next, the configuration of the subject imaging device according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the subject imaging device according to the embodiment. As shown in Fig. 2, the subject imaging device 10 includes a communication unit 11, a storage unit 12, an imaging unit 13, a sound output unit 14, and a control unit 15.

(Regarding the communication unit 11)
The communication unit 11 connects the inside and outside of the subject imaging device 10 so that they can communicate with each other, and transmits and receives information between the inside and outside of the subject imaging device 10. The communication unit 11 may be realized by, for example, a NIC (Network Interface Card), a wireless LAN (Local Area Network) card, a Bluetooth (registered trademark) module, a Wi-Fi (registered trademark) module, an antenna, or the like.

(Regarding the storage unit 12)
The storage unit 12 is a storage device that stores various types of information. The storage unit 12 includes a main storage device and an auxiliary storage device. The main storage device may be realized by a semiconductor memory element such as a random access memory (RAM), a read-only memory (ROM), or a flash memory. The auxiliary storage device may be realized by a hard disk, a solid state drive (SSD), an optical disk, or the like.

(Regarding the imaging unit 13)
The imaging unit 13 captures video of a subject such as a person or an animal. The imaging unit 13 is, for example, a camera, and may be configured as a combination of multiple cameras or an all-around camera that captures video in all directions 360 degrees. After capturing video of a subject such as a person or an animal, the imaging unit 13 stores the captured video data in the storage unit 12.

Now, to explain cameras, cameras include optical elements and image sensors. Optical elements are elements that make up an optical system, such as lenses, mirrors, prisms, and filters. Image sensors are elements that convert light that enters through optical elements into electrical image signals. Examples of image sensors include CCD (Charge Coupled Device) sensors and CMOS (Complementary Metal Oxide Semiconductor) sensors.

(Regarding the sound output unit 14)
The sound output unit 14 outputs sound information toward a subject, such as a person or an animal. The sound output unit 14 may be realized by, for example, a speaker. The speaker converts an electrical signal into sound using a diaphragm. That is, based on a control command given by the electrical signal, the sound output unit 14 vibrates the diaphragm at a predetermined amplitude and frequency, thereby vibrating the air in contact with the diaphragm and outputting sound.

(Regarding the control unit 15)
The control unit 15 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs stored in the storage device of the subject imaging device 10 using RAM as a work area. The control unit 15 may also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 2, the control unit 15 has an acquisition unit 151, a discrimination unit 152, a determination unit 153, a generation unit 154, a learning unit 155, and an output unit 156. The control unit 15 realizes these functions and performs these processes by reading and executing programs from the storage unit 12. Note that the control unit 15 may perform these processes using a single CPU, or may be equipped with multiple CPUs that perform these processes in parallel. Each of these components will be described below in order.

(Regarding the Acquisition Unit 151)
The acquisition unit 151 acquires imaging data including video of subjects such as people, animals, etc. That is, the acquisition unit 151 reads and acquires imaging data including video of subjects such as people, animals, etc. captured by the imaging unit 13 from the storage unit 12.

The acquisition unit 151 acquires the video of the subject when the sound information is output. For example, the acquisition unit 151 reads and acquires from the storage unit 12 the video of the subject captured by the imaging unit 13 when the sound information is output.

(Regarding the determination unit 152)
The discrimination unit 152 discriminates the type of animal based on the imaging data of the imaging unit 13. The discrimination unit 163 may detect objects from the imaging data based on the imaging data and classify the detected objects, i.e., perform object detection processing, and execute processing to discriminate the type of animal from the imaging data. The discrimination unit 163 may be realized, for example, by using semantic segmentation, which is a method of classifying pixels of imaging data into object classes.

(Regarding the determination unit 153)
The determination unit 153 determines the subject's reaction to the output sound information based on the video of the subject when the sound information is output. For example, the determination unit 153 may determine the subject's reaction to the sound information by inputting the video of the subject into a trained model that has learned the relationship between the video of the subject and a label indicating the subject's reaction. In this case, for example, a convolutional neural network (CNN) or the like may be used as the training model. In addition, the label indicating the subject's reaction in this case may be set arbitrarily, and examples include "good,""normal," and "no reaction."

(Regarding the generation unit 154)
The generation unit 154 generates sound information that prompts a response from the subject by changing the variation of the sound depending on the type of subject determined by the determination unit 152. For example, if the subject is a child, the generation unit 154 generates sound information that can be expressed using onomatopoeia such as the squealing and squealing of toys, as well as sound information including the voices of trains, cars, anime characters, etc. Furthermore, if the subject is a dog or cat, the generation unit 154 generates sound information that can be expressed using onomatopoeia such as the rustling of toys and plastic bags, as well as the voice of the owner, the voice of birds, the voice of insects, the voice of animals, the sound of opening a can, the sound of opening a bag, the sound of objects rustling, etc.

The generation unit 154 generates sound information by changing the sound variation in accordance with the context, including the mood of the subject, time, location, surrounding circumstances, and whether or not there is noise, in addition to the type of subject. Note that here, sound variation represents the content of the sound, including the volume, tone, length, and type of sound. For example, if the location is a "park" and the surrounding circumstances are "children playing," the generation unit 154 generates sound information with a medium volume and a bright tone so as not to ruin the cheerful atmosphere. Note that the generation unit 154 may generate sound information by changing the sound variation, using a trained model trained by the learning unit 155 (described later), so as to elicit a "good" response from the subject.

(Regarding the learning unit 155)
The learning unit 155 causes a model to learn the relationship between the features of the output sound information and the subject's response. That is, the learning unit 155 causes the model to learn the output sound information data and the subject's response labels as learning data. In this case, for example, a deep neural network or the like may be used as the model. After the learning unit 155 has caused the model to learn the relationship between the sound information data and the subject's response, the learning unit 155 stores the learned model in the storage unit 12.

(Regarding the output unit 156)
The output unit 156 outputs the sound information generated by the generation unit 154. That is, the output unit 156 outputs the sound information by giving a control command to the sound output unit 14 to output the sound information generated by the generation unit 154. Note that the output unit 156 is not limited to giving a control command to the sound output unit 14, and may give a control command via the communication unit 11 to an external device capable of outputting sound information to output the sound information.

[3. Information processing flow]
Next, the procedure of information processing by the subject imaging device 10 according to the embodiment will be described with reference to FIG. 3 . FIG. 3 is a flowchart illustrating an example of information processing according to the embodiment. For example, the subject imaging device 10 captures video of a subject, including a person or an animal (step S101). Then, the subject imaging device 10 determines the type of subject based on the captured video of the subject (step S102). Then, the subject imaging device 10 generates sound information by changing the variation of a sound that prompts a response from the subject according to the determined type of subject (step S103). Then, the subject imaging device 10 outputs the generated sound information (step S104). Then, the subject imaging device 10 acquires a video of the subject when the sound information is output (step S105). Then, the subject imaging device 10 determines the subject's response to the output sound information based on the video of the subject when the sound information is output (step S106). Then, the subject imaging device 10 learns the relationship between the characteristics of the output sound information and the subject's response (step S107).

4. Hardware Configuration
The subject imaging device 10 according to the embodiment described above is realized by a computer 1000 having a configuration such as that shown in Fig. 4. Fig. 4 is a hardware configuration diagram showing an example of a computer that realizes the functions of the subject imaging device. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected via a bus 1090.

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and secondary storage device 1050, or programs read from the input device 1020, and executes various processes. The primary storage device 1040 is a memory device, such as RAM, that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device that stores data used by the arithmetic device 1030 for various calculations and various databases, and is realized by a ROM (Read Only Memory), HDD (Hard Disk Drive), flash memory, etc.

The output IF 1060 is an interface for transmitting information to be output to an output device 1010, such as a monitor or printer, which outputs various types of information. It is implemented, for example, by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (High Definition Multimedia Interface). The input IF 1070 is an interface for receiving information from various input devices 1020, such as a mouse, keyboard, and scanner, and is implemented, for example, by USB.

The input device 1020 may be a device that reads information from optical recording media such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording media such as an MO (Magneto-Optical Disk), a tape medium, a magnetic recording medium, or a semiconductor memory. The input device 1020 may also be an external storage medium such as a USB memory.

Network IF 1080 receives data from other devices via network N and sends it to the computing device 1030, and also transmits data generated by the computing device 1030 to other devices via network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the subject imaging device 10, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040, thereby realizing the functions of the control unit 15 of the subject imaging device 10.

[5. Composition and Effects]
The subject imaging device according to the present disclosure includes an imaging unit that captures video of a subject including people and animals, a discrimination unit that discriminates the type of subject based on the video of the subject captured by the imaging unit, a generation unit that generates sound information that prompts a reaction from the subject by changing the variation of the sound according to the type of subject discriminated by the discrimination unit, an output unit that outputs the sound information generated by the generation unit, an acquisition unit that acquires the video of the subject when the sound information is output, a discrimination unit that determines the reaction of the subject to the output sound information based on the video of the subject when the sound information is output, and a learning unit that learns the relationship between the characteristics of the output sound information and the reaction of the subject, and the generation unit generates sound information by changing the variation of the sound based on the relationship between the characteristics of the output sound information and the reaction of the subject.

With this configuration, it is possible to put the subject in a state suitable for photography by emitting sound, and then photograph the subject in a state suitable for photography. Therefore, it is possible to provide a subject imaging device 10 that can photograph subjects such as people and animals in a state suitable for photography.

The generation unit of the subject imaging device according to the present disclosure generates the sound information by varying the sound variations according to the context, including the subject's mood, time, location, surrounding circumstances, and whether or not there is noise, in addition to the type of subject.

With this configuration, the sound output can be changed depending on the context, such as the surrounding situation. Therefore, by emitting sound while taking the surrounding situation into consideration, it is possible to put the subject in a state suitable for shooting, and capture the subject in a state suitable for shooting.

The subject imaging method according to the present disclosure comprises the steps of capturing video of a subject, including a person or an animal; determining the type of subject based on the captured video of the subject; generating sound information that prompts a reaction from the subject by changing the sound variation according to the determined type of subject; outputting the generated sound information; acquiring video of the subject when the sound information is output; determining the subject's reaction to the output sound information based on the video of the subject when the sound information is output; and learning the relationship between the characteristics of the output sound information and the subject's reaction. In the generating step, the sound information is generated by changing the sound variation based on the relationship between the characteristics of the output sound information and the subject's reaction.

With this configuration, it is possible to put the subject in a state suitable for photography by emitting sound, and then photograph the subject in a state suitable for photography. This provides a subject imaging method that can photograph subjects such as people and animals in a state suitable for photography.

The program disclosed herein includes the steps of capturing video of a subject, including people and animals; determining the type of subject based on the captured video of the subject; generating sound information that prompts a response from the subject by changing the sound variation according to the determined type of subject; outputting the generated sound information; acquiring a video of the subject when the sound information is output; determining the subject's response to the output sound information based on the video of the subject when the sound information is output; and learning the relationship between the characteristics of the output sound information and the subject's response. In the generating step, the program causes a computer to generate sound information by changing the sound variation based on the relationship between the characteristics of the output sound information and the subject's response.

With this configuration, it is possible to put the subject in a state suitable for photography by emitting sound, allowing the subject to be photographed in a state suitable for photography. Therefore, it is possible to provide a program that allows subjects such as people and animals to be photographed in a state suitable for photography.

The above describes in detail the embodiments of the present application based on the drawings, but this is merely an example, and the present invention can be implemented in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the Disclosure of the Invention section.

Furthermore, the above-mentioned "section, module, unit" can be interpreted as "means" or "circuit." For example, the acquisition unit 151 can be interpreted as acquisition means or acquisition circuit.

REFERENCE SIGNS LIST 10 Subject imaging device 11 Communication unit 12 Storage unit 13 Imaging unit 14 Sound output unit 15 Control unit 151 Acquisition unit 152 Discrimination unit 153 Determination unit 154 Generation unit 155 Learning unit 156 Output unit

Claims (4)

an imaging unit that captures video of subjects including people and animals;
a discrimination unit that discriminates the type of subject based on the image of the subject captured by the imaging unit;
a generator that generates sound information that changes the variation of a sound depending on the type of subject determined by the determiner and prompts the subject to respond;
an output unit that outputs the sound information generated by the generation unit;
an acquisition unit that acquires an image of a subject when the sound information is output;
a determination unit that determines a response of the subject to the output sound information based on an image of the subject when the sound information is output;
a learning unit that learns the relationship between the characteristics of the output sound information and the reaction of the subject,
the generation unit generates the sound information by changing a variation of the sound based on a relationship between a feature of the output sound information and a reaction of the subject.
Subject imaging device. the generation unit generates the sound information by changing the sound variation in accordance with the type of subject as well as context including the subject's mood, time, place, surrounding circumstances, and presence or absence of noise.
The subject imaging device according to claim 1 . capturing an image of a subject including a person or an animal;
determining the type of the subject based on the captured image of the subject;
generating sound information that changes the sound variation depending on the type of the identified subject, and prompts the subject to respond;
outputting the generated sound information;
acquiring an image of the subject when the sound information is output;
determining a response of the subject to the output sound information based on an image of the subject when the sound information is output;
and learning a relationship between the characteristics of the output sound information and a response of the subject,
In the generating step, the sound information is generated by changing a variation of the sound based on a relationship between a feature of the output sound information and a response of the subject.
Subject imaging method. capturing an image of a subject including a person or an animal;
determining the type of the subject based on the captured image of the subject;
generating sound information that changes the sound variation depending on the type of the identified subject, and prompts the subject to respond;
outputting the generated sound information;
acquiring an image of the subject when the sound information is output;
determining a response of the subject to the output sound information based on an image of the subject when the sound information is output;
and learning a relationship between the characteristics of the output sound information and a response of the subject,
In the generating step, the sound information is generated by changing a variation of the sound based on a relationship between a feature of the output sound information and a reaction of the subject;
A program that causes a computer to execute the following.