JP7757975B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

This technology relates to an information processing device, an information processing method, and an information processing program.

When taking photographs with a camera, there is a demand for changing the shooting method and camera settings appropriately depending on the subject's movements and condition. Therefore, technology has been proposed that can achieve continuous shooting linked to the movement of the target person (Patent Document 1).

JP 2009-060355 A

However, the technology described in Patent Document 1 requires that the subject to be photographed be identified in advance using facial recognition, and the only camera setting that can be changed is whether or not to perform continuous shooting, so it cannot be applied to a variety of photography purposes.

This technology has been developed in consideration of these points, and aims to provide an information processing device, information processing method, and information processing program that enable the settings of an imaging device to be changed in accordance with the state of the subject by generating a trigger in accordance with the state of the subject.

In order to solve the above-mentioned problems, the first technology is an information processing device that includes a state determination unit that determines whether the state of the subject has reached a predetermined pre-state, a trigger processing unit that generates a trigger for changing the settings of the imaging device when the state determination unit determines that the state of the subject has reached the pre-state, and a setting change processing unit that performs processing to change the settings of the imaging device after a predetermined time has elapsed since the generation of the trigger , wherein the pre-state is the state of the subject before it reaches a shooting target state, which is a state that is the target of shooting using the imaging device, and the predetermined time is an adjustment time for adjusting the length of time until it reaches the shooting target state after the setting change processing of the imaging device is completed .

The second technology is an information processing method that determines whether the state of the subject has reached a predetermined pre-state, and if it is determined that the state of the subject has reached the pre-state, generates a trigger for changing the settings of the imaging device, and performs processing to change the settings of the imaging device after a predetermined time has elapsed since the trigger was generated, where the pre-state is the state of the subject before it reaches a state to be photographed, which is a state to be photographed using the imaging device, and the predetermined time is an adjustment time for adjusting the length of time until it reaches the state to be photographed after the setting change processing of the imaging device is completed .

Furthermore, the third technology is an information processing program that causes a computer to execute an information processing method that determines whether the state of the subject has reached a predetermined pre-state, and if it is determined that the state of the subject has reached the pre-state, generates a trigger for changing the settings of the imaging device, and performs processing to change the settings of the imaging device after a predetermined time has elapsed since the trigger was generated, where the pre-state is the state of the subject before it reaches a state to be photographed, which is a state to be photographed using the imaging device, and the predetermined time is an adjustment time for adjusting the length of time until it reaches the state to be photographed after the setting change processing of the imaging device is completed.

FIG. 1 is a block diagram showing a configuration of an imaging device 100. FIG. 2 is a block diagram showing the configuration of an information processing device 200. FIG. 10 is a diagram illustrating an outline of a process performed by the information processing device 200 when a trigger is generated once. 10 is a flowchart showing a process performed by the information processing device 200 when a trigger is generated once. FIG. 10 is an explanatory diagram of state determination when a trigger is generated once. FIG. 10 is an explanatory diagram of state determination when a trigger is generated once. FIG. 10 is an explanatory diagram of the length of an adjustment time. FIG. 10 is a diagram illustrating an outline of a process performed by the information processing device 200 when a trigger is generated multiple times. FIG. 10 is an explanatory diagram of state determination when a trigger is generated multiple times. 10 is a flowchart showing a process performed by the information processing device 200 when a trigger is generated multiple times. 10 is a flowchart showing a process performed by the information processing device 200 when a trigger is generated multiple times. FIG. 10 is an explanatory diagram of an example in which the settings of the imaging device 100 are not changed when the assist mode is turned on. FIG. 10 illustrates a first example of a notification in a user interface. FIG. 10 illustrates a second example of a notification in a user interface. FIG. 10 illustrates a third example of a notification in a user interface. FIG. 10 is a diagram illustrating an example of a user interface for determining a pre-existing state and setting changes. FIG. 10 is a diagram illustrating an example of a user interface for determining a pre-existing state and setting changes.

Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order.
1. Embodiment
[1-1. Configuration of imaging device 100]
[1-2. Configuration of information processing device 200]
[1-3. Processing by Information Processing Device 200]
[1-3-1. Example of generating a trigger once]
[1-3-2. Example of generating a trigger multiple times]
[1-3-3. User Interface]
2. Modified Examples

1. Embodiment
[1-1. Configuration of imaging device 100]
The configuration of the imaging device 100 will be described with reference to Fig. 1. The imaging device 100 is configured to include a lens block 110, an image sensor block 120, an image processing circuit 130, an image compression/decompression circuit 140, an image data bus 150, an image RAM (Random Access Memory) 160, a storage block 170, a display block 180, a control unit 190, a lens information determination circuit 191, an input unit 192, an interface 193, a speaker 194, a microphone 195, and an information processing device 200.

The lens block 110 is composed of a lens 111, which includes an optical lens for focusing light from the subject onto the image sensor 121 and a focus lens for adjusting the focus, a lens driver 112 for moving the lens 111, a focus lens control circuit 113 for controlling the focus lens, an optical lens control circuit 114 for controlling the optical lens, and an optical lens memory device 115. The optical lens memory device 115 stores individual lens information, such as parameters such as the relationship between zoom position and focal length, and the relationship between focus position and shooting distance. There may be multiple lens blocks 110 (compound eyes). An optical image of the subject obtained through the lens block 110 is formed on the image sensor 121 of the image sensor block 120.

The image sensor 121 included in the image sensor block 120 photoelectrically converts incident light from the subject obtained through the lens 111 into an electric charge and outputs an image signal. The image sensor 121 then outputs the pixel signal to the image processing circuit 130. The image sensor 121 may be a CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor), or other device. There may be multiple image sensors 121 (such as a three-plate prism or a compound eye). The image sensor block 120 may also include a photodiode for image plane phase difference AF (Auto Focus). The image sensor block 120 may also include an image sensor driver 122 that physically drives the image sensor 121 and doubles as an image stabilization unit, and an image sensor control circuit 123 that controls the image sensor 121.

The image processing circuit 130 converts the image signal output from the image sensor 121 into digital data (RGB creation/bit conversion, etc.), performs various detection processes on the digitized image data, image quality correction processes on the image data (gain correction, white balance, etc.), aspect conversion processes, etc. The imaging device 100 may be equipped with multiple image processing circuits 130 to improve processing speed.

The image compression/expansion circuit 140 receives image data output from the image processing circuit 130 via the image RAM 160, compresses and encodes it, and then outputs it to the storage unit 171 as a moving image or still image data file. MPEG (Moving Picture Experts Group), JPEG (Joint Photographic Experts Group), etc. are used as encoding methods. This does not need to be used if the data is to be saved as uncompressed RAW. The imaging device 100 may be equipped with multiple image compression/expansion circuits 140 to improve processing speed.

The image data bus 150 is a bus for sending and receiving image data. The image RAM 160 is used to supply image data to the image processing circuit 130, image compression/expansion circuit 140, and information processing device 200 connected via the image data bus 150. Multiple image RAMs 160 may be provided to improve processing speed.

The memory block 170 is composed of a memory unit 171 and a memory unit control circuit 172. The memory unit 171 stores the image data finally generated after passing through each processing circuit, and the memory unit control circuit controls the saving and reading of image data in the memory unit 171. The memory unit 171 is composed of a drive device for portable recording media such as magnetic tape or optical disks, a hard disk drive (HDD), a solid state drive (SSD), etc. The memory unit 171 may be built into the imaging device 100 or may be externally attached via a wired or wireless connection. The imaging device 100 may be equipped with multiple memory units 171 to store different formats for inline/offline processing or to improve processing speed and reliability.

The display block 180 is composed of a display unit 181 and a display control circuit 182. The display unit 181 displays the image data, through image, and GUI (Graphical User Interface) finally generated after passing through each processing circuit, while the display control circuit 182 controls the display. The display unit 181 is realized, for example, by a monitor display configured with an LCD (Liquid Crystal Display), PDP (Plasma Display Panel), or organic EL (Electro Luminescence) panel, or an EVF (Electronic Viewfinder). Multiple different settings may be provided so that a through image without aspect conversion and a completed image after aspect conversion processing can be simultaneously viewed. The display unit 181 may also be an external device attached to the imaging device 100 via a wired or wireless connection.

The control unit 190 is composed of a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc. The CPU executes various processes and issues commands in accordance with programs stored in the ROM, thereby controlling the entire imaging device 100 and each component. It also performs autofocus and autoexposure processing correction control, calculates correction processing control values, etc. based on detection information.

The lens information determination circuit 191 is a circuit designed for interchangeable lens cameras that identifies and determines the attached lens. The lens information determination circuit 191 is not a required component.

The input unit 192 allows the photographer (user) to input various instructions and the like to the imaging device 100. When the photographer inputs to the input unit 192, a control signal corresponding to the input is generated and supplied to the control unit 190. The control unit 190 then performs various processes corresponding to the control signal. The input unit 192 may include a release button for issuing a release command, physical buttons for various operations, a touch panel, a touch screen integrated with the display unit 181, and the like.

The interface 193 is an interface between other devices, the Internet, etc. The interface 193 may include a wired or wireless communication interface. More specifically, the wired or wireless communication interface may include cellular communications such as 3TTE, 4G (fourth generation mobile communications system), 5G (fifth generation mobile communications system), Wi-Fi, Bluetooth (registered trademark), NFC (Near Field Communication), Ethernet (registered trademark), HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), etc. Furthermore, if the imaging device 100 and the information processing device 200 are connected via hardware, the interface 193 may include a connection terminal between the devices or a bus within the device. Furthermore, if the imaging device 100 and the information processing device 200 are implemented as separate devices, the interface 193 may include different types of interfaces for each device. For example, the interface 193 may include both a communication interface and an interface within the device.

Speaker 194 is an audio output device used to provide audio notifications to the photographer, etc.

Microphone 195 is a sound collection device for recording audio during shooting.

The imaging device 100 is configured as described above. Examples of imaging devices 100 include digital cameras, single-lens reflex cameras, camcorders, commercial cameras, and professional-grade imaging equipment.

[1-2. Configuration of information processing device 200]
Next, the configuration of the information processing device 200 will be described with reference to Fig. 2. The information processing device 200 is configured to include a state determination unit 201, a trigger processing unit 202, a setting change processing unit 203, and a notification processing unit 204.

The state determination unit 201 recognizes the current state of the subject being photographed from the image generated by the imaging device 100 and determines whether the subject's state has reached the pre-state. The state determination unit 201 determines whether the subject's current state has reached the pre-state based on whether a determination value indicating the degree of match between the subject's current state and the pre-state exceeds a threshold. Details of the pre-state and state determination will be described later. Subject recognition techniques may include methods using machine learning or deep learning, template matching, matching methods based on the subject's luminance distribution information, methods based on the colors contained in the image or human facial features, and methods using artificial intelligence. These methods may also be combined to improve recognition accuracy. The state determination unit 201 may be composed of a dedicated AI circuit for advanced image determination.

The trigger processing unit 202 generates a trigger that triggers the setting change processing unit 203 to change the settings of the imaging device 100 based on the judgment result of the state judgment unit 201.

The setting change processing unit 203 performs processing to change the settings of the imaging device 100. The setting change processing unit 203 performs processing to change the settings of the imaging device 100 after a predetermined time (adjustment time) has elapsed from the occurrence of a trigger until the processing to change the settings of the imaging device 100 is performed. Details of the adjustment time will be described later.

The setting change processing unit 203 performs setting change processing, for example, by sending to the imaging device 100 a control signal instructing the setting change along with information indicating the type of parameter to be changed and the changed value of the parameter. Settings to be changed include, for example, F-number, shutter speed, ISO, white balance, activation of continuous shooting mode, continuous shooting speed, AF frame position, and focus position, but any other parameter that can be changed in the imaging device 100 may also be used. The photographer must set in advance the settings of the imaging device 100 that will be changed by the processing of the setting change processing unit 203 (which parameters will be changed to what values). The settings may be set by the photographer by inputting specific parameters and their values, or the information processing device 200 may set them automatically based on predetermined templates or algorithms corresponding to the subject or type of subject being photographed.

The notification processing unit 204 performs various notification processes to the photographer regarding changes to the settings of the imaging device 100, using the display unit 181 provided in the imaging device 100, etc.

The information processing device 200 is configured as described above. The information processing device 200 may be configured as a dedicated device with hardware that has that function. Furthermore, the information processing device 200 may not only be configured as a dedicated device, but may also be configured with a program, and execution of that program may cause the imaging device 100 to have the functions of the information processing device 200. The program may be pre-installed on the imaging device 100, or may be distributed by download or on a storage medium, and installed in the imaging device 100 by the photographer himself.

[1-3. Control by Information Processing Device 200]
[1-3-1. Example of generating a trigger once]
Next, a description will be given of processing by the information processing device 200. First, an overview of processing realized by the present technology will be described. In this embodiment, a person is the subject, and a description will be given of an example in which the person is photographed running a sprint in a track and field event.

As shown in Figure 3, for ease of explanation, the period during which the subject is sprinting is divided into time 0 to time 6. In a sprint, the subject, who is a runner, first assumes a starting position with his or her hands on the ground at the starting position, as shown at time 1. Then, as shown at time 2, the subject starts running and gradually stands up, as shown at time 3. Then, as shown at time 4, the subject continues running in an upright position, reaching the finish line, as shown at time 5, and the sprint ends at time 6. This period from time 0 to time 6 is the shooting period.

In the example of Figure 3, the imaging device 100 is set to setting A from time 0 to time 3, and the state of the subject (photographed object state) at time 4 is photographed using setting B. To do this, the information processing device 200 changes the imaging device 100 to setting B before the subject enters the photographed object state at time 4. In this way, Figure 3 is an example of generating a trigger once within the shooting period and changing the settings of the imaging device 100 once.

In order to capture the subject state at time 4 using setting B, it is necessary to change the imaging device 100 from setting A to setting B before the subject reaches the subject state at time 4. If the imaging device 100 were to be changed from setting A to setting B at the time the subject reaches the subject state at time 4, it would be too late to capture the image, or even if it were, it would be difficult to capture the image.

The state determination unit 201 determines whether the state of the subject has reached the state (pre-state) before the subject reached the shooting target state at time 4, and if the subject's current state reaches the pre-state, the trigger processing unit 202 generates a trigger that triggers a setting change. Then, after the adjustment time has elapsed since the trigger was generated and before the subject reaches the shooting target state at time 4, the setting change processing unit 203 changes the imaging device 100 from setting A to setting B. This allows the photographer to capture the shooting target state of the subject with setting B. To do this, it is necessary to identify in advance which state of the subject is the pre-state.

The pre-state is the state of the subject before it reaches the capture target state, which is the state that is the target of capture using the imaging device 100. The pre-state can be identified using specific images of the subject captured in the past. In the case of a sprint, for example, it is a good idea to capture images of a pre-run practice session or a sprint run by another runner other than the subject to obtain images to serve as the pre-state. Alternatively, the information processing device 200 may store multiple images or information representing pre-states as templates in advance, and the photographer may specify the type of subject or type of scene to identify the pre-state from among those templates. Furthermore, if the pre-state is a numerical value such as the subject's movement speed, the photographer can identify the pre-state by inputting a specific numerical value.

The adjustment time is the time from when a trigger occurs until the process of changing the settings of the imaging device 100 is performed, and is set in advance. The adjustment time can be said to be the time for adjusting the length of time from when the settings of the imaging device 100 are changed until the shooting timing (when the subject becomes a subject to be photographed). The adjustment time can be set to any length by the photographer, from 0 seconds onwards.

In this embodiment, the mode in which the information processing device 200 automatically changes the settings of the imaging device 100 to capture the state of the subject to be photographed is referred to as the assist mode. In the example of Figure 3, photography is performed using setting A from time 0 until time 4, but this can be achieved by changing the setting of the imaging device 100 from setting X to setting A when the assist mode is turned on.

The processing by the information processing device 200 will be described with reference to the flowchart in Figure 4. First, in step S101, the assist mode in which the information processing device 200 operates in response to input from the photographer is turned on. At the same time as turning on this assist mode, the imaging device 100 changes the setting to setting A.

Next, in step S102, the state determination unit 201 starts determining the state of the subject. The state determination unit 201 continues to perform state determination while the assist mode is on.

Now, with reference to Figures 5 and 6, the state determination by the state determination unit 201 will be described. In this description, the movement of the subject is considered to be the state of the subject. The setting change processing unit 203 needs to change the setting of the imaging device 100 to setting B before the subject enters the shooting target state at time 4. To do this, the state before the subject enters the shooting target state at time 4 (the state at time 3 in this embodiment) is considered to be the pre-state, and it is necessary to determine whether the subject has entered the pre-state.

The state determination unit 201 therefore recognizes the current state of the subject being photographed from the image generated by the imaging device 100 and compares the current state of the subject with the prior state. If a determination value indicating the degree of agreement between the current state of the subject and the prior state exceeds a threshold, it determines that the current state of the subject has become the prior state. If it determines that the current state of the subject has become the prior state, the state determination unit 201 supplies the determination result to the trigger processing unit 202.

When the subject's movement is considered to be the subject's state, the comparison of the subject's current movement with the movement in the previous state can be made based on the subject's posture, movement speed, movement acceleration, movement trajectory, sound, vibration, etc.

When determining the state of movement based on the posture of the subject, the determination value in the state determination indicates the degree of match between the subject's current posture and the posture in the previous state. If the determination value exceeds a threshold, the current movement of the subject is deemed to be the movement of the previous state, and the state determination unit 201 supplies the determination result to the trigger processing unit 202. The degree of match between the subject's current posture and the posture in the previous state can be obtained, for example, by a known image comparison method using machine learning, deep learning, etc., between an image capturing the subject's current posture and an image showing the posture in the previous state.

Furthermore, when determining the state of movement based on the speed of the subject, the determination value in the state determination indicates the degree of match between the current speed of the subject and the speed in the previous state. If the determination value exceeds a threshold, the current movement of the subject is deemed to be the movement of the previous state, and the state determination unit 201 supplies the determination result to the trigger processing unit 202. The degree of match between the current speed of the subject and the speed in the previous state can be obtained, for example, by calculating the movement speed based on the shooting frame rate and the movement distance of the subject between multiple frames, and comparing them.

Furthermore, when determining the state of movement based on the trajectory of the subject's movement, the determination value in the state determination indicates the degree of match between the subject's current movement trajectory and the movement trajectory in the previous state. If the determination value exceeds a threshold, the state determination unit 201 determines that the subject's current movement trajectory matches the movement trajectory in the previous state, and supplies the determination result to the trigger processing unit 202. The degree of match of the movement trajectories can be obtained, for example, by comparing or tracing line segments that indicate the movement trajectories of specific parts of the subject's body.

Furthermore, when determining the state of an action based on the sound emitted by the subject, the determination value in the state determination indicates the degree of similarity between the current sound of the subject and the sound in the previous state. The degree of similarity of the sound can be determined, for example, by comparing the frequency, sound volume (dB), or sound waveform. The sound during shooting can be acquired by microphone 195.

Furthermore, when determining the state of movement based on the vibrations emitted by the subject, the determination value indicates the degree of similarity between the subject's current vibrations and the vibrations in the previous state. The degree of similarity of the vibrations can be determined, for example, by comparing the frequency, amplitude, and period of the vibrations.

In addition, the state may be determined by combining multiple of the above-mentioned posture, speed, acceleration, trajectory, sound, and vibration and comparing the subject's movement with the previous state.

The threshold can be set arbitrarily. For example, the photographer may set the threshold by inputting a specific value, or the photographer may input criteria such as top, middle, or bottom, and the information processing device 200 may automatically set the threshold based on the input criteria. The information processing device 200 may also set the threshold based on an algorithm, etc. Furthermore, in addition to inputting a value or criteria, the threshold may also be set by inputting an image or video that corresponds to the threshold.

In the example of Figure 5, the subject takes the starting position for a sprint, begins to run, and stands up. As the subject's movements gradually approach those of the pre-state at time 3, the judgment value increases. When the judgment value exceeds the threshold, the subject's movements are deemed to have become those of the pre-state at time 3, and the state judgment unit 201 supplies the judgment result to the trigger processing unit 202.

Note that the pre-state is not limited to the state immediately before the state to be captured. As shown in Figure 6, when time 4 is the state to be captured, the state of time 1, which occurs before time 3, can also be the pre-state. By setting the state of time 1, which occurs before time 3, as the pre-state, the time from trigger occurrence to the state to be captured is longer, so that during that time, the photographer can be notified that "settings will be changed," for example, and the photographer can calmly proceed with the capture, knowing that settings will be changed. Note that any state that occurs before the state to be captured can be set as the pre-state, so the state of time 2 can also be set as the pre-state.

Note that "the subject's state has become the pre-existing state" does not necessarily mean a perfect match, but can also mean a similar state with some deviation. The amount of deviation allowed in state determination can be adjusted by setting a threshold. If the threshold is set high, the subject's current state will be determined to have become the pre-existing state with less deviation than when the threshold is low. On the other hand, if the threshold is set low, the subject's current state will be determined to have become the pre-existing state with more deviation than when the threshold is high.

Returning to the explanation of the flowchart, if a certain period of time has not elapsed since the assist mode was turned on in step S103, processing proceeds to step S104 (No in step S103).

Next, in step S104, if the state determination unit 201 determines that the determination value is equal to or greater than the threshold value, the process proceeds to step S105 (Yes in step S104). Then, in step S105, the trigger processing unit 202 generates a trigger.

Next, in step S106, if the adjustment time has elapsed since the trigger occurred, the process proceeds to step S107 (Yes in step S106). Then, in step S107, the setting change processing unit 203 changes the setting of the imaging device 100 from setting A to setting B.

The adjustment time can be set to any length from 0 seconds onwards. It is generally advisable to adjust the adjustment time according to the photographer's photography skills.

For example, a professional photographer with advanced photography skills can shorten the leeway time, which is the time from when the settings of the imaging device 100 are changed until the subject enters the subject state, by lengthening the adjustment time as shown in Figure 7. This means that the settings will not be changed until just before the subject enters the subject state, allowing photography to be carried out with setting A until just before the subject enters the subject state. It is believed that because the photographer has advanced photography skills, they will be able to take photographs even if the settings of the imaging device 100 are changed just before the subject enters the subject state.

On the other hand, amateur photographers with less advanced photography skills can increase the time between changing the settings of the imaging device 100 and the subject being photographed by shortening the adjustment time as shown in Figure 7. In other words, the settings of the imaging device 100 will be changed sooner than if the adjustment time were longer. This increases the time between changing the settings of the imaging device 100 and the subject being photographed, allowing the photographer to take photographs calmly and with more time to spare.

The length of the adjustment time may also be determined depending on the type of setting before the change and the type of setting after the change. For example, suppose setting B is a mode that consumes more power than setting A because it continuously drives the image sensor 121, lens 111, etc. for autofocus. In this case, it is possible to reduce power consumption by lengthening the adjustment time and lengthening the time the imaging device 100 is in setting A (shortening the time it is in setting B).

The adjustment time can also be set to 0 seconds. If the adjustment time is set to 0 seconds, the setting change processing unit 203 changes the setting of the imaging device 100 to setting B at the same time as the trigger processing unit 202 generates a trigger.

In this way, it is preferable to set the length of the adjustment time according to the photographer's photography technique and the performance of the imaging device. Therefore, the photographer needs to confirm and decide in advance what the optimal length of adjustment time should be. If filming a sprint, the length of the adjustment time should be determined in advance based on prior practice and past sprints, and set in the information processing device 200.

Alternatively, a table that associates multiple standards, such as large, medium, and small, with multiple lengths of adjustment time can be stored in advance in the information processing device 200, and the standards can be presented to the photographer to select one, with the length of adjustment time being determined based on the selected standard. This is mainly effective for amateur photographers who do not have high photography skills.

Returning to the explanation of the flowchart, next, in step S108, if a certain period of time has elapsed since the settings of the imaging device 100 were changed, processing proceeds to step S109 (Yes in step S108).

Next, in step S109, the settings of the imaging device 100 are returned to setting X, which is the setting before the change in assist mode. Note that the process of returning to the setting before the change in assist mode in step S109 is not required, and the setting may or may not be returned. Then, in step S110, assist mode is turned off and the process ends.

If a certain amount of time has passed in step S103, the process proceeds to step S108, where the assist mode is turned off and the process ends. This is because if a certain amount of time has passed after the start of state determination but before the determination value reaches or exceeds the threshold, the assist mode is ended as it is determined that the desired state for the photographer has not been achieved.

In the above explanation, the subject's movement is considered to be the subject's state, but other factors such as the subject's posture, speed, position, and the position of specific body parts can also be processed as the subject's state. Furthermore, a combination of these can also be processed as the subject's state.

For example, by setting the subject's posture to the subject state, a trigger can be generated at the timing when the subject assumes a posture in the pre-state before assuming the posture in the photographing target state, thereby changing the settings of the imaging device 100. This allows the photographer to photograph the subject in the photographing target state posture with specific imaging device 100 settings.

When changing the subject's posture to the subject's state, the judgment value in the state judgment indicates the degree of match between the subject's current posture and the posture in the previous state. If the judgment value exceeds a threshold, the current posture of the subject is deemed to have become the posture of the previous state, and the state judgment unit 201 supplies the judgment result to the trigger processing unit 202. The degree of match between the subject's current posture and the posture in the previous state can be obtained using a known image comparison method using machine learning, deep learning, etc.

Furthermore, by setting the speed of the subject to the subject's state, a trigger can be generated at the timing when the subject reaches the speed of the pre-state before reaching the speed of the subject state, thereby changing the settings of the imaging device 100. This allows the photographer to capture the state in which the subject has reached the speed of the subject state with specific imaging device 100 settings. The same applies to the acceleration of the subject.

When the subject's speed is set to the subject's state, the judgment value in the state judgment indicates the degree of match between the subject's current speed and the speed in the previous state. If the judgment value exceeds a threshold, the current speed of the subject is deemed to have become the speed in the previous state, and the state judgment unit 201 supplies the judgment result to the trigger processing unit 202. The degree of match between the subject's current speed and the speed in the previous state can be obtained, for example, by calculating the speeds based on the shooting frame rate and the distance traveled by the subject between multiple frames, and comparing them.

Furthermore, by setting the subject's position to the subject state, the settings of the imaging device 100 can be changed at the timing when the subject reaches the position of the previous state before reaching the position of the subject state. This allows the photographer to capture the state when the subject reaches the position of the subject state with the settings of the specific imaging device 100.

When the subject's position is set to the subject's state, the judgment value in the state judgment indicates, for example, the degree of match between the subject's current position within the angle of view and the position in the previous state. If the judgment value exceeds a threshold, the current position of the subject is deemed to have reached the position in the previous state, and the state judgment unit 201 supplies the judgment result to the trigger processing unit 202. The degree of match between the subject's current position and the position in the previous state can be obtained, for example, by comparing the subject's current coordinates within the angle of view with the coordinates in the previous state.

Furthermore, by setting the position of a specific part of the subject's body to the subject's state, the settings of the imaging device 100 can be changed at the time when the subject's body part reaches a position in a previous state before reaching a position in the state to be photographed. This allows the photographer to photograph the state in which the subject's body part has reached the position in the state to be photographed with the specific settings of the imaging device 100.

For example, when the position of a specific part, such as the height of the subject's head from the ground, is set as the state of the subject, the judgment value in the state judgment indicates the degree of match between the current head height and the head height in the previous state. If the judgment value exceeds a threshold, the current head height of the subject has become the head height in the previous state, and the state judgment unit 201 supplies the judgment result to the trigger processing unit 202. The degree of match between the subject's head height and the head height in the previous state can be obtained, for example, by comparing the coordinates of the current head position within the angle of view with the coordinates of the head position in the previous state.

[1-3-2. Example of generating a trigger multiple times]
Next, an example will be described in which a trigger is generated multiple times within a shooting period to change multiple times the settings of the image capture device 100. In the example of Fig. 3 described above, a trigger is generated once within a shooting period to change the settings of the image capture device 100 once, but it is also possible to generate a trigger multiple times within a shooting period to change the settings of the image capture device 100 multiple times.

As shown in Figure 8, when photographing a sprint race in track and field, the settings of the imaging device 100 are changed so that the subject's actions immediately after the start at time 2 are photographed with setting B as subject state B, the subject's actions as they get up and run at time 4 are photographed with setting C as subject state C, and the subject's actions after crossing the finish line at time 6 are photographed with setting D as subject state D.

In order to change the imaging device 100 to setting B for capturing subject state B at time 2, the trigger processing unit 202 generates trigger B when the subject state reaches pre-state B. Thereafter, in order to change the imaging device 100 to setting C for capturing subject state C at time 4, the trigger processing unit 202 generates trigger C when the subject state reaches pre-state C. Further thereafter, in order to change the imaging device 100 to setting D for capturing subject state D at time 6, the trigger processing unit 202 generates trigger D when the subject state reaches pre-state D. In Figure 8, the trigger is generated three times and then the settings of the imaging device 100 are changed three times, but this number is merely an example, and the number of triggers generated and settings changed may be any number of times as long as they are one or more.

To generate each of these triggers, the state determination unit 201 determines whether the current state of the subject has reached each of the pre-states, using a determination value for each pre-state and a threshold value for each pre-state, as shown in Figure 9. Note that in order to change the settings of the imaging device 100 multiple times within a shooting period, it is necessary to identify multiple pre-states, and it is also necessary to identify the order of these multiple pre-states. The determination method by the state determination unit 201 is the same as when changing the settings once, as described above.

Let judgment value B be the judgment value indicating the degree of match between the subject's current state and pre-state B, and let threshold B be the threshold used to determine that the subject's current state has become pre-state B. Let judgment value C be the judgment value indicating the degree of match between the subject's current state and pre-state C, and let threshold C be the threshold used to determine that the subject's current state has become pre-state C. Let judgment value D be the judgment value indicating the degree of match between the subject's current state and pre-state D, and let threshold D be the threshold used to determine that the subject's current state has become pre-state D. The method for identifying the pre-state is the same as the method described in the example of generating a trigger once above.

Referring to the flowcharts of Figures 10 and 11, we will explain the process of generating a trigger multiple times by the information processing device 200 to change the settings of the imaging device 100 multiple times.

First, in step S201, the information processing device 200 turns on the assist mode in response to input from the photographer. At the same time as turning on this assist mode, the imaging device 100 changes the setting to setting A. Next, in step S202, the state determination unit 201 begins determining the state of the subject.

Next, in step S203, it is checked whether the most recent prior state is prior state B. If the most recent prior state is prior state B, processing proceeds to step S204 (Yes in step S203).

Next, in step S204, if a certain amount of time has not elapsed since the assist mode was turned on, processing proceeds to step S205 (No in step S204).

If a certain period of time has elapsed in step S204, trigger B is not generated, and in step S210 the setting of the imaging device 100 is changed to setting X, which is the previous setting, and in step S211 the assist mode is turned off and the processing ends.

Next, in step S205, if the state determination unit 201 determines that the determination value is equal to or greater than threshold value B, the process proceeds to step S206 (Yes in step S205). Then, in step S206, the trigger processing unit 202 generates trigger B.

Next, in step S207, if the adjustment time has elapsed after trigger B was generated, the process proceeds to step S208 (Yes in step S207). Then, in step S208, the setting change processing unit 203 changes the setting of the imaging device 100 to setting B.

Next, in step S209, if all setting changes have not yet been made, processing proceeds to step S203 (No in step S209).

Since the subject's state has reached pre-state B, trigger B has been generated, and the imaging device 100 has changed to setting B, the next pre-state is pre-state C. Therefore, processing proceeds from step S203 to step S213 (No in step S203).

Next, in step S213, it is checked whether the most recent prior state is prior state C. If the most recent prior state is prior state C, processing proceeds to step S214 (Yes in step S213).

Next, in step S214, if a certain amount of time has not elapsed since the assist mode was turned on, processing proceeds to step S215 (No in step S214).

If a certain period of time has elapsed in step S214, trigger C is not generated, and in step S219 the setting of the imaging device 100 is changed to setting X, which is the previous setting, and in step S220 the assist mode is turned off and the processing ends.

Next, in step S215, if the state determination unit 201 determines that the determination value is equal to or greater than threshold C, the process proceeds to step S216 (Yes in step S215). Then, in step S216, the trigger processing unit 202 generates trigger C.

Next, in step S217, if the adjustment time has elapsed since the occurrence of trigger B, the process proceeds to step S218 (Yes in step S207). Then, in step S218, the setting change processing unit 203 changes the setting of the imaging device 100 to setting C.

Next, in step S209, if all setting changes have not yet been made, processing proceeds to step S203 (No in step S209).

Since trigger C was generated and the settings of the imaging device 100 were changed to setting C, the next trigger is trigger D. Therefore, processing proceeds from step S203 to step S213 (No in step S203), and from step S213 to step S221 (No in step S213).

Next, in step S221, if a certain amount of time has not elapsed since the assist mode was turned on, processing proceeds to step S222 (No in step S221).

If a certain period of time has elapsed in step S221, trigger D is not generated, and in step S226 the setting of the imaging device 100 is changed to setting X, which is the previous setting, and in step S227 the assist mode is turned off and the processing ends.

Next, in step S222, if the state determination unit 201 determines that the determination value is greater than or equal to threshold value D, the processing proceeds to step S223 (Yes in step S222). Then, in step S223, the trigger processing unit 202 generates trigger D.

Next, in step S224, if the adjustment time has elapsed since the occurrence of trigger D, the process proceeds to step S225 (Yes in step S224). Then, in step S225, the setting change processing unit 203 changes the setting of the imaging device 100 to setting D.

Next, in step S209, if all setting changes have been made, processing proceeds to step S228 (Yes in step S209).

Then, if a certain amount of time has passed since the last setting change in step S228, processing proceeds to step S229 (Yes in step S228).

Next, in step S229, the setting before the change in assist mode is restored to setting X. Then, in step S230, the assist mode is turned off and the process ends.

Note that the processes of returning to the settings before the changes in assist mode in steps S210, S219, S226 and S229 are not required processes, and the settings may or may not be returned.

In both the examples of Figures 3 and 8, the setting of the imaging device 100 is set to setting A when the assist mode is turned on, and after a trigger is generated, the setting of the imaging device 100 is changed from setting A to setting B. However, it is not necessary to change the setting of the imaging device 100 when the assist mode is turned on; as shown in Figure 12, setting X, which is the setting before the assist mode was turned on, may be maintained as is, and setting X may be changed to setting B after a trigger is generated.

In either case where a trigger is generated once or multiple times during the above-mentioned shooting period, the release, which is the shooting instruction, may be based on input from the photographer to the input unit 192 of the imaging device 100, or may be performed automatically. For example, the release may be performed automatically based on the subject recognition result, or may be performed automatically when a predetermined time has elapsed after the trigger is generated or after a setting change on the imaging device 100. The release may also be remotely input from an external device rather than from the imaging device 100 itself. When shooting automatically, if the imaging device 100 has an automatic shooting function, this may be used, or the information processing device 200 may perform automatic shooting processing in which a control signal instructing the imaging device 100 to shoot is sent from the information processing device 200.

[1-3-3. User Interface]
Next, a description will be given of a user interface for notifications by the notification processing unit 204. Notification contents include the type of setting of the image capture device 100, the current setting status of the image capture device 100, and a notice that the setting of the image capture device 100 will be changed.

One notification method, for example, is to superimpose a frame 301 on the through image displayed on the display unit 181 of the imaging device 100, as shown in Figure 13. If the settings of the imaging device 100 are changed multiple times within a shooting period, it is advisable to use different colors for the frame 301 for each type of setting, as shown in Figures 13A to 13D. In Figure 13, by using different frame colors, Figure 13A indicates setting A, Figure 13A indicates setting B, Figure 13A indicates setting C, and Figure 13D indicates setting D.

Furthermore, by keeping the frame 301 lit, it is notified that the imaging device 100 is currently operating with the settings indicated by the above-mentioned color. This notification should be made from the time the settings of the imaging device 100 are changed, and should continue as long as the imaging device 100 is operating with the changed settings. Furthermore, by flashing the frame 301, a notice is given that the settings of the imaging device 100 will be changed. This notice may be given at any time between the occurrence of a trigger and the setting change process of the imaging device 100.

Another notification method is to superimpose an icon 302 on the through image displayed on the display unit 181 of the imaging device 100, as shown in Figure 14. In this case, the type of setting is notified using characters or symbols included in the icon 302, and by keeping the icon 302 lit, it is notified that the imaging device 100 is currently operating with the settings represented by the characters or symbols in the icon 302, and by flashing the icon 302, it is notified that a change to the settings of the imaging device 100 will be made.

The notification may be not only a display on the display unit 181, but also an audio notification using the audio output from the speaker 194 of the imaging device 100. If the imaging device 100 has a vibration function, the notification may be a notification using vibration. Furthermore, if the imaging device 100 has an illuminant such as an LED, the notification may be a notification using the light emitted by the LED. If the settings are changed multiple times, it is recommended that the type of sound, type of vibration, and type of light emission be different for each setting.

The photographer may also be notified of the timing when the settings of the imaging device 100 should be changed. After confirming this notification, the photographer changes the settings of the imaging device 100 by inputting an instruction to change the settings to the imaging device 100. This notification may be the same as or a different method from the advance notification of the setting change to the imaging device 100 described above.

As shown in FIG. 15, when displaying on a device other than the imaging device 100, such as a personal computer or tablet terminal that has a large display unit 400, the icon 302 may be displayed without being superimposed on the live view display 401 or the operation panel display 402. This prevents the icon 302 from obscuring part of the display.

Figures 16 and 17 are examples of a user interface for identifying a pre-condition, which is displayed on the display unit 181. As shown in Figure 16A, the user interface for identifying a pre-condition has a hierarchical structure consisting of the type of subject to be photographed and the type of object.

For example, in the example of Figure 16, when a sport is selected as the subject of filming, pre-prepared candidate sport types such as soccer, baseball, and volleyball are presented. Therefore, if soccer is selected, pre-prepared soccer scene types such as shooting, passing, throwing, and catching/punching are presented. Therefore, if shooting is selected, pre-prepared scenes related to shooting such as take-back, impact, and follow-through are presented. Since a take-back is usually present before the impact of a soccer shot, when the state to be filmed is impact, the photographer selects take-back as the pre-state. This identifies the take-back motion as the pre-state.

Furthermore, the setting parameters of the imaging device 100 (in Figure 16A, shutter speed, ISO, F-number, focus position, white balance, etc.) that are changed when it is determined that the state of the subject has reached the pre-condition are set based on input from the photographer, a predetermined template, algorithm, etc.

For professional photographers with advanced photography skills, for example, if the subject of photography is sports, the type of sport, scene, and settings of the imaging device 100 can be combined in advance as shown in Figure 16B, and this can be presented in a user interface for identifying pre-conditions, allowing the photographer to identify any pre-condition.

Furthermore, for amateur photographers with low photography skills, for example, if the subject of photography is sports, only the type of sport can be presented in the user interface as shown in Figure 17A, and when the photographer selects one of them, the imaging device 100 and the information processing device 200 can recognize the characteristic state of the subject in that sport and identify the pre-condition.

Furthermore, for amateur photographers with low photography skills, as shown in Figure 17B, only the types of subjects to be photographed can be presented in the user interface, and when the photographer selects one of them, the imaging device 100 and the information processing device 200 can recognize the characteristic subject conditions of that subject and identify the pre-condition.

The processing of this technology is carried out in this manner. This technology allows the imaging device to automatically change settings appropriate for the subject before it becomes a target for photography, making it possible to capture images with optimal settings regardless of the photographer's abilities, such as dynamic visual acuity, athletic ability, or photography technique.

This reduces the variation in the time required to change the settings of the imaging device 100 due to differences in the photographer's individual abilities, providing the photographer with the benefit of being able to take photos with appropriate settings without missing any photo opportunities.

In addition, the settings of the imaging device 100 that can be changed are not limited to specific ones, so the settings of the imaging device 100 can be changed widely to suit various shooting conditions, such as the type of subject.

2. Modified Examples
Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical concept of the present technology are possible.

The information processing device 200 is not limited to the imaging device 100, but can also be used in any device that has a camera function, such as a smartphone, personal computer, tablet terminal, portable game console, or wearable device.

The information processing device 200 may also operate on a server or in the cloud. In this case, the information processing device 200 performs status determination using captured images received from the imaging device 100 via the network, and performs setting change processing by transmitting control information instructing a setting change to the imaging device 100 via the network.

The state determination unit 201 may perform state determination based on the state of the subject in a partial area within the angle of view of the imaging device 100. For example, if the subject changes movement but remains in the same position on the right side of the angle of view, the angle of view can be divided into left and right areas from the center, and state determination can be performed using only the captured image of the right area. This can reduce processing time compared to performing state determination using captured images for the entire angle of view.

In the embodiment, the setting change processing unit 203 performs processing to change the settings of the imaging device 100 after a trigger occurs, but regardless of whether a trigger occurs or not, if the photographer inputs an instruction to change the settings, the setting change processing unit 203 may prioritize the instruction input from the photographer and change the settings of the imaging device 100.

Methods for inputting instructions from the photographer include input operations on the input unit 192 provided on the imaging device 100, such as buttons, and voice input. If settings are changed multiple times during the shooting period, it is advisable to assign each of the multiple settings to a different button on the imaging device 100. In addition, in the case of voice input, it is advisable to assign each of the multiple settings to a different sound. For example, setting A can be assigned to a sound that sounds "Ah," and setting B can be assigned to a sound that sounds "Bee."

The information processing device 200 may also change settings according to the acceleration of the movement of the imaging device 100. For example, if the acceleration of the movement of the imaging device 100 is equal to or greater than a predetermined value, the settings of the imaging device 100 may be changed, or the assist mode may be turned off. Furthermore, when changing to multiple settings within a shooting period, it is advisable to assign each of the multiple settings to a different acceleration. For example, if the acceleration of the imaging device 100 is equal to or greater than a predetermined acceleration A, the setting may be changed to setting A; if the acceleration of the imaging device 100 is equal to or less than a predetermined acceleration C, the setting may be changed to setting C; if the acceleration of the imaging device 100 is less than the predetermined acceleration A and exceeds acceleration C, the setting may be changed to setting B. In order to perform setting change processing according to the acceleration of the movement of the imaging device 100, the imaging device 100 needs to be equipped with an acceleration sensor, and sensor information detected by the acceleration sensor needs to be supplied to the information processing device 200.

Furthermore, the information processing device 200 may change the settings of the imaging device 100 in response to framing or gestures. Examples of framing include changing the settings of the imaging device 100 or turning off assist mode when it is detected that the lens of the imaging device 100 is facing the ground (downward). Whether the lens of the imaging device 100 is facing the ground can be detected by subject recognition in the image or by an attitude detection method using a gyro sensor or acceleration sensor. In order to perform setting change processing in response to framing, the imaging device 100 needs to be equipped with a gyro sensor or acceleration sensor and supply sensor information to the information processing device 200.

Examples of gestures include changing the settings of the imaging device 100 or turning off assist mode when a specific gesture, such as the photographer holding their hand in front of the lens, is detected. The photographer's gestures can be detected by subject recognition in the image, or by providing the imaging device 100 with an infrared sensor that detects non-contact hand movements. In order to perform setting change processing in response to gestures, the imaging device 100 needs to be equipped with a gyro sensor or acceleration sensor and supply sensor information to the information processing device 200.

The present technology can also be configured as follows.
(1)
a state determination unit that determines whether the state of the subject has reached a predetermined pre-state;
and a trigger processing unit that generates a trigger for changing settings of an imaging device when the state determination unit determines that the state of the subject has reached the pre-state.
(2)
The information processing device according to (1), wherein the pre-state is a state of the subject before the subject becomes a state to be photographed using the imaging device.
(3)
The information processing device according to (2), further comprising a setting change processing unit that performs processing to change the setting of the imaging device after a predetermined time has elapsed since the trigger was generated.
(4)
The information processing device according to (3), wherein the predetermined time is an adjustment time for adjusting a length of time until the imaging device reaches the imaging subject state after a setting change process of the imaging device is completed.
(5)
The information processing device according to (4), wherein the adjustment time is set by a user.
(6)
The information processing device according to (4) or (5), wherein the adjustment time can be set to any length equal to or greater than 0 seconds.
(7)
The information processing device according to any one of (1) to (6), wherein the state determination unit determines that the current state of the subject has become the pre-state when a determination value indicating the degree of coincidence between the current state of the subject in the image generated by the imaging device and the pre-state exceeds a threshold value.
(8)
The information processing device according to any one of (1) to (7), wherein the state of the subject is one of or a combination of the movement, posture, position, and speed of the subject.
(9)
The information processing device according to any one of (1) to (8), wherein the trigger processing unit generates a plurality of the triggers in accordance with a plurality of states of the subject.
(10)
The information processing device according to any one of (1) to (9), further comprising a notification processing unit that processes a notification to a photographer after the trigger occurs.
(11)
The information processing device according to (10), wherein the notification processing unit performs a process of notifying the photographer to change a setting of the imaging device after the trigger occurs.
(12)
The information processing device according to (10) or (11), wherein the notification processing unit performs processing for notifying the photographer of the current settings of the imaging device.
(13)
The information processing device according to any one of (10) to (12), wherein the notification processing unit performs a process of notifying the photographer that it is time to change settings of the imaging device after the trigger occurs.
(14)
The information processing device according to (3), wherein automatic photographing processing of the imaging device is performed after the setting change processing unit changes the setting of the imaging device.
(15)
The information processing device according to (3), wherein the setting change processing unit performs processing to change the setting of the imaging device in response to an input from a photographer, regardless of whether the trigger has occurred.
(16)
determining whether the state of the subject has reached a predetermined pre-state;
An information processing method for generating a trigger for changing settings of an imaging device when it is determined that the state of the subject has reached the pre-state.
(17)
determining whether the state of the subject has reached a predetermined pre-state;
An information processing program that causes a computer to execute an information processing method for generating a trigger for changing settings of an imaging device when it is determined that the state of the subject has reached the pre-state.

200: Information processing device 201: State determination unit 202: Trigger processing unit 203: Setting change processing unit 204: Notification processing unit

Claims (14)

a state determination unit that determines whether the state of the subject has reached a predetermined pre-state;
a trigger processing unit that generates a trigger for changing a setting of the imaging device when the state determination unit determines that the state of the subject has reached the pre-state;
a setting change processing unit that performs processing to change the setting of the imaging device after a predetermined time has elapsed since the occurrence of the trigger;
Equipped with
the pre-state is a state of the subject before the subject becomes a capture target state, which is a state that is a target of capture using the imaging device,
The predetermined time is an adjustment time for adjusting the length of time until the image capture state is reached after the setting change process of the image capture device is completed.
Information processing device. The information processing apparatus according to claim 1 , wherein the adjustment time is settable by a user. The information processing apparatus according to claim 1 , wherein the adjustment time can be set to any length equal to or greater than 0 seconds. The information processing device according to claim 1, wherein the state determination unit determines that the current state of the subject has become the prior state when a determination value indicating the degree of coincidence between the current state of the subject in the image generated by the imaging device and the prior state exceeds a threshold value. The information processing device according to claim 1 , wherein the state of the subject is any one of or a combination of the movement, posture, position, and speed of the subject. The information processing apparatus according to claim 1 , wherein the trigger processing unit generates a plurality of the triggers in response to a plurality of states of the subject. The information processing apparatus according to claim 1 , further comprising a notification processing unit that performs processing for notifying the photographer after the trigger occurs. The information processing apparatus according to claim 7 , wherein the notification processing unit performs a process of notifying the photographer to change settings of the imaging device after the trigger occurs. The information processing apparatus according to claim 7 , wherein the notification processing section performs processing for notifying the photographer of the current settings of the imaging device. The information processing apparatus according to claim 7 , wherein the notification processing section performs a process of notifying the photographer that it is time to change a setting of the imaging device after the trigger occurs. The information processing apparatus according to claim 1 , wherein automatic photographing processing of the imaging device is performed after the setting change processing unit changes the setting of the imaging device. The information processing apparatus according to claim 1 , wherein the setting change processing section performs processing to change the setting of the imaging apparatus in response to an input from a photographer, regardless of whether the trigger has occurred or not. determining whether the state of the subject has reached a predetermined pre-state;
When it is determined that the state of the subject has reached the pre-state, a trigger for changing the settings of the imaging device is generated;
performing a process of changing the settings of the imaging device after a predetermined time has elapsed since the trigger was generated;
the pre-state is a state of the subject before the subject becomes a capture target state, which is a state that is a target of capture using the imaging device,
The predetermined time is an adjustment time for adjusting the length of time until the image capture state is reached after the setting change process of the image capture device is completed.
Information processing methods. determining whether the state of the subject has reached a predetermined pre-state;
When it is determined that the state of the subject has reached the pre-state, a trigger for changing the settings of the imaging device is generated;
performing a process of changing the settings of the imaging device after a predetermined time has elapsed since the trigger was generated;
the pre-state is a state of the subject before the subject becomes a capture target state, which is a state that is a target of capture using the imaging device,
The predetermined time is an adjustment time for adjusting the length of time until the image capture state is reached after the setting change process of the image capture device is completed.
An information processing program that causes a computer to execute an information processing method.