JP7638451B2 - Behavior determination device, behavior determination method, and behavior determination program - Google Patents

Description

The present disclosure relates to a behavior determination device, a behavior determination method, and a behavior determination program.

In recent years, land-based aquaculture, which can stably produce aquatic organisms, has been attracting attention. In conventional land-based aquaculture, it was necessary for experienced workers to grasp the health of the aquatic organisms, adjust the water quality, and operate equipment related to feeding, etc., while observing the aquatic organisms, relying on intuition to carry out such tasks.
Patent Document 1 discloses a technology for automating a land-based aquaculture system with the aim of reducing the labor required for these manual tasks. The technology disclosed in Patent Document 1 determines the three-dimensional position of each aquatic organism, such as fish, in an aquarium from a captured image, and individual characteristic data of each organism, such as its size and body shape, tracks the organism based on the determined position and characteristic data, and monitors the growth status of the organism based on the tracking results. Furthermore, the technology is characterized by having a means for controlling the amount of feed and growth environment conditions using statistically managed data to achieve optimal growth status.

JP 2003-250382 A

The technology disclosed in Patent Document 1 does not take into consideration the timing of feeding. Therefore, this technology has the problem that feeding is not always carried out at an appropriate time, and therefore efficient feeding may not be possible.

The present disclosure aims to provide data that enables an automatic feeding device to feed the cultivated aquatic organisms at appropriate times.

The behavior determination device according to the present disclosure comprises:
a behavior determination unit that determines a classification of a behavior of a target organism using behavior features that are calculated based on time transitions of at least one of the positions and bending of each joint of a target organism, the behavior features corresponding to the behavior of the target organism, and a reference behavior model that is a model indicating a classification of the behavior of the target organism according to the behavior features;
The device is equipped with a growth condition control unit that generates data for determining whether or not the device that feeds the target organism will feed the target organism, in accordance with the determined classification of the behavior of the target organism.

According to the present disclosure, the behavior determination unit determines the behavior of the target organism in accordance with the behavioral feature, and the growth condition control unit generates data for a device that feeds the target organism to decide whether or not to feed the target organism in accordance with the behavior of the target organism. Here, the device corresponds to an automatic feeding device, and the automatic feeding device can feed the target organism at an appropriate timing by deciding whether or not to feed the target organism based on the data. Therefore, according to the present disclosure, data can be provided that enables the automatic feeding device to feed the cultivated aquatic organism at an appropriate timing.

FIG. 1 is a diagram showing an example of the configuration of a behavior determination system 9 according to a first embodiment. 5A to 5C are diagrams for explaining the processing of a skeleton extraction unit 23 according to the first embodiment. 1A and 1B are diagrams for explaining the processing of the behavior determination unit 27 in the first embodiment, in which (a) shows a specific example when the target organism is a shrimp, and (b) shows a specific example when the target organism is a catfish. FIG. 2 is a diagram for explaining a feeding behavior model according to the first embodiment. FIG. 2 is a diagram for explaining a non-feeding behavior model according to the first embodiment. 1 is a diagram for explaining a normal behavior model according to the first embodiment; FIG. 2 is a diagram for explaining an abnormal behavior model according to the first embodiment. FIG. 2 is a diagram showing an example of the hardware configuration of a behavior determining device 10 according to the first embodiment. 4 is a flowchart showing the operation of the behavior determining device 10 according to the first embodiment. FIG. 13 is a diagram showing an example of a hardware configuration of a behavior determining device 10 according to a modification of the first embodiment. FIG. 13 is a diagram showing a configuration example of a behavior determination system 9 according to a second embodiment. 13A and 13B are diagrams for explaining the processing of the learning unit 301 in the second embodiment, where FIG. 13A shows an example of creating a feeding behavior model, FIG. 13B shows an example of creating a non-feeding behavior model, and FIG. 13C shows an example of creating an abnormal behavior model. 10 is a flowchart showing the operation of a remaining feed amount acquisition section 200 and a learning section 301 according to the second embodiment. 13 is a flowchart showing the operation of the corpse number acquisition unit 400 and the learning unit 301 according to the second embodiment.

In the description of the embodiments and in the drawings, the same elements and corresponding elements are given the same symbols. Descriptions of elements given the same symbols are omitted or simplified as appropriate. Arrows in the drawings primarily indicate the flow of data or the flow of processing. In addition, "part" may be interpreted as "circuit," "process," "procedure," "processing," or "circuitry" as appropriate.

Embodiment 1.
Hereinafter, the present embodiment will be described in detail with reference to the drawings.

***Configuration Description***
1 shows an example of the configuration of a behavior determination system 9 according to the present embodiment. As shown in FIG. 1, the behavior determination system 9 includes a camera 1, a behavior determination device 10, an automatic feeding device 60, and a growth environment control device 70.
The behavior determination device 10 has a function of determining the behavior of an aquatic organism. In the present application, the aquatic organism typically refers to an aquatic organism cultivated in an aquarium. As shown in FIG. 1, the behavior determination device 10 includes an image acquisition unit 21, a target organism detection unit 22, a skeleton extraction unit 23, a target organism tracking unit 24, a motion feature amount calculation unit 25, a bending feature amount calculation unit 26, a behavior determination unit 27, and a growth condition control unit 28, and stores a reference behavior model 31. In addition, the behavior determination device 10 is connected to a camera 1, an automatic feeding device 60, and a growth environment control device 70.
The camera 1 is a device for photographing aquatic organisms, and there may be a plurality of cameras. The camera 1 outputs video data representing images photographed by the camera 1. The video data is data representing a plurality of frames, and as a specific example, is data of a moving image.
The automatic feeding device 60 is a device that feeds target organisms, and is a device that automatically feeds aquatic organisms. The target organisms are aquatic organisms having joints, and specific examples thereof include fish and crustaceans.
The growth environment control device 70 is a device that controls the growth environment of aquatic organisms.

The video acquisition unit 21 acquires the video data output by the camera 1.

The target organism detection unit 22 detects a target organism in each frame indicated by the video data acquired by the video acquisition unit 21. The target organism detection unit 22 may detect the location of the target organism in each frame by using a trained model based on AI (Artificial Intelligence) technology. As a specific example, the trained model is a model that receives data indicating a frame as input and outputs the result of detecting the target organism.
When there are multiple frames in which a target organism is detected, the target organism detection unit 22 generates, as detection information, information indicating the target organism detected in each frame in which the target organism is detected.

The skeleton extraction unit 23 extracts the skeleton of the target organism indicated by the detection information generated by the target organism detection unit 22, and generates skeleton information indicating the extracted skeleton. At this time, typically, the skeleton extraction unit 23 extracts each joint of the target organism in each frame indicated by the video data, and generates information indicating each extracted joint as skeleton information. At this time, the video data indicates a plurality of frames in which the target organism is shown. The skeleton information may be information indicating the position and bending of each joint, or may be information indicating parts such as the eyes, each fin, the tail, and the legs. The skeleton extraction unit 23 may change the skeleton to be extracted depending on the type of the target organism. In addition, when the target organism is a crustacean, the target organism tracking unit 24 extracts the joints of the target organism as the skeleton.
Fig. 2 is a diagram for explaining the processing of the skeleton extraction unit 23. When the target organism is a fish, as a specific example, the skeleton extraction unit 23 extracts each part of the skeleton of the target organism, that is, the left eye, the right eye, the left pectoral fin, the right pectoral fin, the pelvic fin, the center of the dorsal fin, the anal fin, the base of the caudal fin, and the tip of the caudal fin, as shown in Fig. 2.

The target organism tracking unit 24 tracks the target organism based on the skeletal information, and generates information indicating the tracking results as tracking information. Specifically, the target organism tracking unit 24 tracks the target organism based on multiple frames indicated by the video data acquired by the video acquisition unit 21 and the skeletal information generated by the skeletal extraction unit 23, and generates tracking information indicating the tracking results. At this time, the target organism tracking unit 24 sets, as a specific example, a point within the head of the target organism as the reference point for tracking. As a specific example, the point in question is a point corresponding to the mouth or eyes, etc. The tracking information indicates, as a specific example, the distance and time traveled by the target organism.

The movement characteristic amount calculation unit 25 calculates the movement characteristic amount of the target organism based on the tracking information generated by the target organism tracking unit 24. The movement characteristic amount is a characteristic amount related to the movement of the target organism, and corresponds to at least one of the movement, position, and posture of the target organism. As a specific example, the movement characteristic amount is composed of at least one of a value indicating the position of the target organism, a value indicating the posture of the target organism, a value indicating the swimming distance of the target organism, a value indicating the swimming speed of the target organism, and a value indicating the acceleration of the target organism while swimming. The movement characteristic amount calculation unit 25 may change the characteristic amount calculated according to the items of the reference behavior model 31.

The bending feature amount calculation unit 26 calculates the bending feature amount of the target organism based on the skeleton information generated by the skeleton extraction unit 23. The bending feature amount calculation unit 26 may use the tracking information generated by the target organism tracking unit 24 when calculating the bending feature amount. The bending feature amount is a feature amount related to the bending of the joints of the target organism, and is a feature amount corresponding to at least one of the bending of the joints of the target organism and the joints of the target organism that have moved with the bending of the target joint. As a specific example, the bending feature amount is composed of at least one of a value indicating each of the bending direction and degree of each of at least some of the joints of the target organism, a value indicating each of the angular velocity and angular acceleration at the time of bending of each of the bent joints of the joints of the target organism, and a value indicating each of the movement amount, movement speed, and movement direction of each of the joints that have moved with the bending of the joints of the target organism. The bending feature amount calculation unit 26 may change the feature amount calculated according to the item of the reference behavior model 31.
As a specific example, the flexion feature amount calculation unit 26 calculates the movement trajectory of each joint of the target organism, and calculates the movement distance and movement direction of each joint based on the calculated movement trajectory. Note that the nature of the movement trajectory calculated by the flexion feature amount calculation unit 26 basically differs depending on the type, age, etc. of the target organism.

The behavior determination unit 27 determines the behavior of the target organism using the behavior feature amount and the reference behavior model 31. The behavior feature amount is a feature amount calculated based on the time progression of at least one of the position and bending of each joint of the target organism, and corresponds to the behavior of the target organism. As a specific example, the behavior feature amount is composed of at least one of a movement feature amount and a bending feature amount. Specifically, the behavior determination unit 27 determines what type of behavior the target organism is by applying the calculated movement feature amount and displacement feature amount to the reference behavior model 31.
As a specific example, behavior determination unit 27 determines the behavior of the target organism as feeding behavior when the swimming speed of the target organism is faster than a predetermined speed, the orientation of the target organism's head is within a predetermined angle from the vertical upward direction, and the target organism wags its tail at intervals shorter than a predetermined interval. Feeding behavior is the behavior of the target organism when it desires food.
As another specific example, the behavior determination unit 27 determines the behavior of the target organism as abnormal when the swimming speed of the target organism is slower than a predetermined speed and the abdomen of the target organism is vertically above the dorsal fin of the target organism. Abnormal behavior is behavior that is not normal, and specific examples include behavior of the target organism when the target organism is in a weakened state, a high stress state, or an ill state. Normal behavior is behavior of the target organism when the target organism is in good health.

FIG. 3 is a diagram for explaining the process of the behavior determining unit 27. As shown in FIG.
3A shows a specific example in which the target organism is a shrimp. When the shrimp floats to the water surface and the shrimp's legs move a lot, the behavior determination unit 27 determines the shrimp's behavior as a feeding behavior. On the other hand, when the shrimp stays on the water bottom and the shrimp moves a little, the behavior determination unit 27 determines the shrimp's behavior as a non-feeding behavior. Non-feeding behavior is behavior that is not a feeding behavior.
3B shows a specific example in which the target organism is a catfish. When the catfish frequently moves its tail, the behavior determination unit 27 determines that the behavior of the catfish is normal. On the other hand, when the catfish hardly moves its tail, the behavior determination unit 27 determines that the behavior of the catfish is abnormal.

The growth condition control unit 28 generates data for the device that feeds the target organism to determine whether or not to feed the target organism, according to the behavior of the target organism determined by the behavior determination unit 27. The data may be data indicating whether or not to feed the target organism, data indicating the amount of feeding, data indicating whether or not there is a target organism that is performing a behavior determined to be a feeding behavior, data indicating each of the total number of target organisms performing a behavior determined to be a feeding behavior and the total number of target organisms in the aquarium, or data indicating the processing results of each unit included in the behavior determination device 10. When the data indicates the amount of feeding, the automatic feeding device 60 determines not to feed the target organism when the amount of feeding is 0, and determines to feed the target organism when the amount of feeding is not 0.
Furthermore, the growth condition control unit 28 may generate data indicating the amount of adjustment of each of the automatic feeding device 60 and the growth environment control device 70 based on the determination result of the behavior determination unit 27, and output the generated data. As a specific example, the growth condition control unit 28 calculates at least one of the feeding timing and feeding amount for the target organism according to the determined behavior of the target organism, and also generates data used to adjust the growth environment of the target organism according to the determined behavior of the target organism.

The reference behavior model 31 is a model that indicates a classification of the behavior of a target organism according to behavioral features, and is used to classify the behavior of an aquatic organism. The reference behavior model 31 may be a model expressed in a table format, or may be a classifier based on machine learning.
4 to 7 show specific examples of the reference behavior model 31. The reference behavior model 31 will be described below with reference to Figs. 4 to 7. Note that the feeding behavior model, the non-feeding behavior model, the normal behavior model, and the abnormal behavior model are each a specific example of a model included in the reference behavior model 31.

Figure 4 shows a specific example of a feeding behavior model. The feeding behavior model is a model used to determine whether the behavior of a target organism is feeding behavior, and as a specific example, is a model that shows an index for classifying the behavior of the target organism as feeding behavior according to behavioral features. By using the feeding behavior model shown in Figure 4, the behavior determination unit 27 can determine whether the behavior of the target organism is feeding behavior based on the swimming speed, head angle, swimming depth, and caudal fin angle change of the target organism. Here, the swimming speed, head angle, and swimming depth correspond to the movement features, and the caudal fin angle change corresponds to the bending feature.

Figure 5 shows a specific example of a non-feeding behavior model. The non-feeding behavior model is a model used to determine whether the behavior of a target organism is non-feeding behavior, and as a specific example, is a model that shows an index for classifying the behavior of a target organism as non-feeding behavior according to behavioral features. By using the non-feeding behavior model shown in Figure 5, the behavior determination unit 27 can determine whether the behavior of a target organism is non-feeding behavior based on the swimming speed, head angle, swimming depth, and caudal fin angle change of the target organism. Non-feeding behavior is behavior that is not feeding behavior, and as a specific example, is behavior that a target organism takes when full.

Figure 6 shows a specific example of a normal behavior model. The normal behavior model is a model used to determine whether the behavior of a target organism is normal or not, and as a specific example, is a model showing an index for classifying the behavior of the target organism as normal behavior according to the behavior feature. By using the normal behavior model shown in Figure 6, the behavior determination unit 27 can determine whether the behavior of the target organism is normal or not based on the swimming speed, pectoral fin position, and number of caudal fin angle changes of the target organism. Here, the pectoral fin position corresponds to the movement feature, and the number of caudal fin angle changes corresponds to the bending feature.

Figure 7 shows a specific example of an abnormal behavior model. The abnormal behavior model is a model used to determine whether the behavior of a target organism is abnormal or not, and as a specific example, is a model that shows an index for classifying the behavior of the target organism as abnormal behavior according to behavioral features. By using the abnormal behavior model shown in Figure 7, the behavior determination unit 27 can determine whether the behavior of the target organism is abnormal or not based on the swimming speed, pectoral fin position, and number of caudal fin angle changes of the target organism.

Figure 8 shows an example of the hardware configuration of the behavior determination device 10 according to this embodiment. The behavior determination device 10 is composed of a computer and includes hardware such as a processor 20, a storage device 30, a communication device 40, and an input/output interface 50. The behavior determination device 10 may be composed of multiple computers.

The processor 20 is an integrated circuit (IC) that performs arithmetic processing and controls the hardware of the computer. Specific examples of the processor 20 include a central processing unit (CPU), a digital signal processor (DSP), and a graphics processing unit (GPU).
The behavior determination device 10 may include a plurality of processors that replace the processor 20. The plurality of processors share the role of the processor 20.

The storage device 30 may be a volatile storage device, a non-volatile storage device, or a combination of these. A specific example of a volatile storage device is a RAM (Random Access Memory). A specific example of a non-volatile storage device is a ROM (Read Only Memory), a HDD (Hard Disk Drive), or a flash memory.

The communication device 40 is a receiver and a transmitter. As a specific example, the communication device 40 is a communication chip or a NIC (Network Interface Card).

The input/output interface 50 is a port to which an input device and an output device are connected. A specific example of the input/output interface 50 is a USB (Universal Serial Bus) terminal. A specific example of the input device is a keyboard and a mouse. A specific example of the output device is a display.

Each part of the behavior determination device 10 may use the input/output interface 50 and the communication device 40 as appropriate when communicating with other devices, etc. The communication device 40 and the input/output interface 50 acquire signals output by the camera 1, etc.

The storage device 30 stores a reference behavior model 31 and a behavior determination program. The behavior determination program is a program that causes a computer to realize the functions of each part of the behavior determination device 10. The processor 20 reads out and executes the behavior determination program stored in the storage device 30, thereby operating as each part of the behavior determination device 10. The functions of each part of the behavior determination device 10 are realized by software.

Data used when executing the behavior determination program and data obtained by executing the behavior determination program are appropriately stored in the storage device 30. Each part of the behavior determination device 10 uses the storage device 30 as appropriate. Note that the terms "data" and "information" may have the same meaning. The storage device 30 may be independent of the computer.

The behavior determination program may be recorded on a computer-readable non-volatile recording medium. Specific examples of the non-volatile recording medium include an optical disk or a flash memory. The behavior determination program may be provided as a program product.

*** Operation Description ***
The operation procedure of the behavior determination device 10 corresponds to a behavior determination method, and the program that realizes the operation of the behavior determination device 10 corresponds to a behavior determination program.

Figure 9 is a flowchart showing an example of the operation of the behavior determination device 10. The operation of the behavior determination device 10 will be explained with reference to Figure 9.

(Step S1)
The video acquisition unit 21 acquires, as target video data, the video data output by the camera 1. The video acquisition unit 21 may acquire, as target video data, only data within a certain time range from the video data output by the camera 1.

(Step S2)
The target organism detection unit 22 detects a target organism from each frame represented by the target video data. The target organism detection unit 22 may use only a portion of the frames represented by the target video data.

(Step S3)
If a target living thing is detected from the image represented by the target image data, the behavior determination device 10 proceeds to step S4, otherwise the behavior determination device 10 returns to step S1.
In addition, when a target living organism is detected from the image represented by the target image data, it is assumed that one or more target living organisms are detected.

(Step S4)
The behavior determination device 10 selects, as a selected target organism, one of the one or more target organisms that has not yet been selected in the iterative processing from step S4 to step S10. Note that the selected target organism is assumed to have been detected in multiple frames.

(Step S5)
The skeleton extraction unit 23 extracts the skeleton of the selected target organism in each frame in which the selected target organism is detected among the frames indicated by the target video data, and generates information indicating the extracted skeleton as target skeleton information.

(Step S6)
The target organism tracking unit 24 tracks the selected target organism based on the target skeletal information, and generates information indicating the tracking results as target tracking information.

(Step S7)
The motion feature amount calculation unit 25 calculates the motion feature amount of the selected target organism based on the target tracking information. As a specific example, the motion feature amount calculation unit 25 calculates the swimming distance and swimming time of the target organism based on the target tracking information, and sets the calculated results as the motion feature amount.
In this step, the curvature feature amount calculation section 26 may calculate the curvature feature amount of the selected target organism based on the target skeletal information.

(Step S8)
The behavior determination unit 27 compares the feature amount calculated in step S7 with a reference behavior model 31 prepared in advance.

(Step S9)
The behavior determination section 27 determines the behavior of the selected target creature based on the comparison result in step S8.

(Step S10)
The behavior determination device 10 repeatedly executes steps S5 to S9 the number of times corresponding to the number of target living things detected from the target video data.

(Step S11)
The growth condition control unit 28 generates data indicating the amount of adjustment of each of the automatic feeding device 60 and the growth environment control device 70 based on the judgment result of the behavior judgment unit 27 in step S9, and outputs the generated data to each of the automatic feeding device 60 and the growth environment control device 70.
As a specific example, when a target organism exhibits a behavior determined to be a feeding behavior in step S9, the automatic feeding device 60 determines that it is feeding time and executes feeding. At this time, the automatic feeding device 60 may change the amount of feeding depending on the proportion of target organisms exhibiting the behavior determined to be a feeding behavior in step S9. As a specific example, the automatic feeding device 60 has a reference feeding amount, which is a reference feeding amount for each target organism, in a database not shown, and obtains the number of target organisms exhibiting the behavior determined to be a feeding behavior in step S9 and the number of target organisms detected in step S3, and calculates the proportion of target organisms exhibiting the behavior determined to be a feeding behavior by dividing the number of target organisms exhibiting the behavior determined to be a feeding behavior by the number of detected target organisms, and sets the feeding amount to 1.2 times the reference feeding amount when the calculated proportion is 80% or more.
As another specific example, the automatic feeding device 60 has a feeding amount index that indicates a higher value than the value indicated by the feeding behavior index, and when a feature value corresponding to a target organism exhibiting behavior determined to be feeding behavior is equal to or greater than the value indicated by the feeding amount index, the feeding amount is set to 1.2 times the reference feeding amount. As a specific example, the feeding behavior index indicates a swimming speed of 15 cm/s, and the feeding amount index indicates a swimming speed of 25 cm/s. The feeding behavior index is an index used to determine whether the behavior of the target organism is feeding behavior. The feeding amount index is an index used to determine the feeding amount.
As a specific example, the growth environment control device 70 first sets the water temperature or oxygen supply amount in the aquarium high when there is one or more target organisms whose behavior is determined to be abnormal in step S9. After that, the behavior determination device 10 executes the behavior determination flow shown in FIG. 9 again after a certain time has elapsed, and when there is no target organism whose behavior is determined to be abnormal in step S9, the growth environment control device 70 sets the water temperature or oxygen supply amount in the aquarium low. The reason for first setting the water temperature or oxygen supply amount in the aquarium high is that, in general, the higher the water temperature and the higher the oxygen concentration, the more active (actively moving) the fish are. The growth environment control device 70 may first set the water temperature or oxygen supply amount in the aquarium low, and then set the water temperature or oxygen supply amount in the aquarium high.

***Description of Effect of First Embodiment***
As described above, according to this embodiment, the behavior of the target organism can be understood in more detail than by only tracking the target organism by determining the type of behavior of the target organism based on the swimming behavior and leg movements of the target organism, etc. Also, according to this embodiment, the skeleton of the target organism is extracted according to the type of the target organism, and the behavior of the target organism is determined based on the behavior of the target organism tracked based on the extracted skeleton and a model for classifying the behavior of the target organism, thereby making it possible to understand the behavior according to the ecology of the target organism.
Furthermore, according to this embodiment, efficient feeding can be performed based on the results of understanding the behavior of the target organism. That is, according to this embodiment, since feeding can be performed at an appropriate timing, it is possible to prevent the water quality from deteriorating by feeding the target organism when it is full, and to prevent not feeding the target organism when it is hungry, and it is possible to adjust the amount of feeding according to the health condition of the target organism.

***Other configurations***
<Modification 1>
FIG. 10 shows an example of the hardware configuration of the behavior determining device 10 according to this modified example.
The behavior determination device 10 includes a processing circuit 19 instead of the processor 20 or instead of the processor 20 and the storage device 30 .
The processing circuitry 19 is hardware that realizes at least a portion of each unit of the behavior determination device 10 .
The processing circuitry 19 may be dedicated hardware, or may be a processor that executes a program stored in a storage device 30 .

When processing circuitry 19 is dedicated hardware, processing circuitry 19 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.
The behavior determination device 10 may include a plurality of processing circuits that replace the processing circuit 19. The plurality of processing circuits share the role of the processing circuit 19.

In the behavior determination device 10, some functions may be realized by dedicated hardware and the remaining functions may be realized by software or firmware.

The processing circuitry 19 is illustratively implemented in hardware, software, firmware, or a combination thereof.
The processor 20, the storage device 30, and the processing circuit 19 are collectively referred to as the “processing circuitry.” In other words, the functions of the functional components of the behavior determination device 10 are realized by the processing circuitry.
The behavior determination device 10 according to other embodiments may also have a similar configuration to this modified example.

Embodiment 2.
The following mainly describes the differences from the above-described embodiment with reference to the drawings.

***Configuration Description***
11 shows an example of the configuration of a behavior determination system 9 according to this embodiment. As shown in FIG. 11 , the behavior determination device 10 according to this embodiment further includes a remaining food amount acquisition unit 200, a learning unit 301, and a corpse number acquisition unit 400.
The remaining food amount acquisition section 200 comprises a video acquisition section 201 , a remaining food detection section 202 , and a remaining food amount calculation section 203 .
The corpse number acquisition unit 400 includes an image acquisition unit 401 , a corpse detection unit 402 , and a corpse number calculation unit 403 .

The image acquisition unit 201 is similar to the image acquisition unit 21.

The remaining food detection unit 202 detects remaining food in the tank after feeding from each frame shown in the video data acquired by the video acquisition unit 201. After feeding is after the target organism has eaten the provided food, and as a specific example, after the time normally required for the target organism to finish eating the food has elapsed from the time of feeding. The remaining food detection unit 202 may determine remaining food using a background subtraction method, or may determine where remaining food is present in each frame using a trained model based on AI technology. As a specific example, the trained model is a model that receives data showing a frame as input and outputs the result of detecting remaining food.

The remaining food amount calculation unit 203 calculates the amount of food detected by the remaining food detection unit 202.

The image acquisition unit 401 is similar to the image acquisition unit 21.

The corpse detection unit 402 detects a corpse in the tank from each frame of the video data acquired by the video acquisition unit 401. The corpse detection unit 402 may determine the presence of a corpse using motion features and a background subtraction method, or may determine where in each frame the corpse is located using a trained model based on AI technology. As a specific example, the trained model is a model that receives data indicating a frame as input and outputs the result of detecting a corpse.

The corpse number calculation unit 403 calculates the number of corpses detected by the corpse detection unit 402. As a specific example, when a corpse is detected using the background subtraction method, the corpse number calculation unit 403 calculates the number of corpses based on the total number of pixels estimated to reflect the corpse, and when a corpse is detected using AI technology, the corpse number calculation unit 403 calculates the number of corpses based on the number of detection frames.

The learning unit 301 generates a model based on the movement feature amount and the curvature feature amount, and data indicating the results of detection by the remaining food detection unit 202 or the carcass detection unit 402 .
As a specific example, the learning unit 301 uses the non-feeding behavior data and the feeding behavior data to learn a model that classifies the behavior of the target organism into feeding behavior and non-feeding behavior according to the behavioral feature of the target organism. The non-feeding behavior data is data when the amount of remaining food is equal to or greater than a reference remaining food amount, data showing the behavioral feature of the learning aquatic organism before feeding, and data labeled as data corresponding to non-feeding behavior. The remaining food amount is the amount of food remaining after the feeding time has elapsed from the feeding time. The feeding time is a time set as the time required for the learning aquatic organism to eat the fed food. The learning aquatic organism is an aquatic organism corresponding to the target organism. The learning aquatic organism may be the target organism itself, or may be an aquatic organism having the same or similar properties as the target organism. The feeding behavior data is data when the amount of remaining food is less than a reference remaining food amount, data showing the behavioral feature of the learning aquatic organism before feeding, and data labeled as data corresponding to feeding behavior. In this example, when the amount of food calculated by the remaining food amount calculation unit 203 is less than a certain amount, the learning unit 301 learns the feeding behavior of the target organism based on the movement features and curvature features before feeding and the amount of remaining food after feeding, and may use a model that shows the movement features and curvature features corresponding to the feeding behavior as at least a part of the reference behavior model 31.
As another specific example, the learning unit 301 uses the abnormal behavior data and the normal behavior data to learn a model that classifies the behavior of the target organism into either normal behavior or abnormal behavior according to the behavior feature of the target organism. Here, in this example, it is assumed that an aquarium is used to raise a group of aquatic organisms composed of a plurality of aquatic organisms corresponding to the target organism. Each of the plurality of aquatic organisms corresponding to the target organism may be an aquatic organism having the same or similar properties as the target organism. The abnormal behavior data is data when the carcasses of aquatic organisms constituting an abnormal number or more of aquatic organisms in the aquarium are detected, and is data indicating the behavior feature of each aquatic organism constituting the aquatic organisms before the carcasses of aquatic organisms constituting an abnormal number or more of aquatic organisms in the aquarium are detected, and is labeled as data corresponding to abnormal behavior. The normal behavior data is data when the corpses of aquatic organisms constituting the aquatic organisms group that are less than an abnormal number are detected in the aquarium, and is data indicating behavioral features of each aquatic organism constituting the aquatic organisms group before the corpses of the aquatic organisms constituting the aquatic organisms group that are less than an abnormal number are detected in the aquarium, and is data that is labeled as data corresponding to normal behavior. In this example, when the number of corpses calculated by the corpse number calculation unit 403 is less than a certain number, the learning unit 301 learns abnormal behavior of the target organism based on the movement features and curvature features before the detection of corpses and the number of corpses, and may use a model indicating the movement features and curvature features corresponding to abnormal behavior as at least a part of the reference behavior model 31.

FIG. 12 is a diagram for explaining the processing of the learning unit 301. As shown in FIG.
12A shows a specific example of creating a feeding behavior model by the learning unit 301. When the amount of remaining food detected after feeding is less than a certain amount, i.e., when the target product has eaten a sufficient amount of the provided food, the learning unit 301 creates a feeding behavior model in which the behavior of the target organism before feeding is classified as feeding behavior.
12B shows a specific example of creating a non-feeding behavior model by the learning unit 301. When the amount of remaining food detected after feeding is greater than a certain amount, i.e., when the target product has not eaten enough of the provided food, the learning unit 301 creates a non-feeding behavior model in which the behavior of the target organism before feeding is classified as non-feeding behavior.
12C shows a specific example of creating an abnormal behavior model by the learning unit 301. When the number of detected target organism corpses is greater than a certain number, the learning unit 301 creates an abnormal behavior model in which the behavior of the target organism before the detection of more than a certain number of corpses is classified as abnormal behavior.
The learning unit 301 may generate a normal behavior model when the number of detected target creature corpses is less than a certain number.

*** Operation Description ***
13 is a flow chart showing an example of the operation of the remaining feed amount acquisition section 200 and the learning section 301. The operation of the remaining feed amount acquisition section 200 and the learning section 301 will be described with reference to FIG.

(Step S21)
The image acquisition unit 201 acquires the image data output by the camera 1 as target image data. This step is the same as step S1. Note that the target image data indicates an image captured inside an aquarium in which a target organism is kept.

(Step S22)
If the target video data is data after feeding, the behavior determination apparatus 10 proceeds to step S23. Otherwise, the behavior determination apparatus 10 returns to step S21.
Note that, in cases where the behavior determination apparatus 10 is aware that the target video data is data taken after feeding, the behavior determination apparatus 10 may skip this step.

(Step S23)
The remaining food detection unit 202 detects remaining food from each frame represented by the target video data.

(Step S24)
If remaining food is detected in step S23, the behavior determination apparatus 10 proceeds to step S25. Otherwise, the behavior determination apparatus 10 proceeds to step S26.

(Step S25)
The remaining food amount calculation section 203 calculates the amount of remaining food detected in step S24.

(Step S26)
The learning unit 301 acquires information indicating each of the locomotion amount feature amount and the curvature feature amount before feeding.

(Step S27, Step S28, Step S29)
If the amount of remaining food calculated in step S25 is equal to or greater than a certain amount, the learning unit 301 labels the information acquired in step S26 as non-feeding behavior. Otherwise, the learning unit 301 labels the information acquired in step S26 as feeding behavior. The learning unit 301 may label each piece of information about an individual.
The data labeled in steps S28 and S29 corresponds to learning data, the data labeled as non-feeding behavior corresponds to non-feeding behavior data, and the data labeled as feeding behavior corresponds to feeding behavior data.

(Step S30)
If the amount of learning data is sufficient, the behavior determination apparatus 10 proceeds to step S31, otherwise, the behavior determination apparatus 10 returns to step S21.

(Step S31)
The learning unit 301 uses the learning data labeled in steps S28 and S29 to learn the relationship between the behavior of the target organism and the label.

(Step S32)
Based on the result of learning in step S31, the learning unit 301 creates a feeding behavior model as a model to be included in the reference behavior model 31. Note that the result of learning in step S31 may be the feeding behavior model itself.

Figure 14 is a flowchart showing an example of the operation of the corpse number acquisition unit 400 and the learning unit 301. The operation of the corpse number acquisition unit 400 and the learning unit 301 will be explained with reference to Figure 14.

(Step S41)
The image acquisition unit 401 acquires, as target image data, the image data output by the camera 1. This step is similar to step S21.

(Step S42)
The corpse detection unit 402 detects the corpse of a target organism from each frame represented by the target video data.

(Step S43)
If a corpse is detected in step S42, the behavior determination apparatus 10 proceeds to step S44. Otherwise, the behavior determination apparatus 10 proceeds to step S45.

(Step S44)
Corpse number calculation section 403 calculates the number of corpses detected in step S43.

(Step S45)
The learning unit 301 acquires information indicating each of the motion feature and the curvature feature before the detection of a corpse. The learning unit 301 may acquire only information regarding an individual estimated to be a corpse. Specific examples of an individual estimated to be a corpse include an individual floating on the water surface, or an individual whose surface color has changed.

(Step S46, Step S47, Step S48)
If the number of corpses calculated in step S45 is equal to or greater than a certain number, the learning unit 301 labels the information acquired in step S46 as abnormal behavior. Otherwise, the learning unit 301 labels the information acquired in step S46 as normal behavior. The learning unit 301 may label each piece of information about an individual.
The data labeled in steps S47 and S48 corresponds to learning data, the data labeled as abnormal behavior corresponds to abnormal behavior data, and the data labeled as normal behavior corresponds to normal behavior data.

(Step S49)
If the amount of learning data is sufficient, the behavior determining apparatus 10 proceeds to step S50, otherwise, the behavior determining apparatus 10 returns to step S41.

(Step S50)
The learning unit 301 uses the learning data labeled in steps S47 and S48 to learn the relationship between the behavior of the target organism and the label.

(Step S51)
Based on the result of learning in step S50, the learning unit 301 creates an abnormal behavior model as a model to be included in the reference behavior model 31. Note that the result of learning in step S50 may be the abnormal behavior model itself.

***Description of Effect of Second Embodiment***
As described above, according to this embodiment, a model used to classify the behavior of a target organism can be generated based on the behavior of the target organism and the state inside the aquarium.

***Other embodiments***
The above-described embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted.
In addition, the embodiments are not limited to those shown in the first and second embodiments, and various modifications are possible as necessary. The procedures described using the flowcharts and the like may be modified as appropriate.

1 Camera, 9 Behavior determination system, 10 Behavior determination device, 19 Processing circuit, 20 Processor, 21 Video acquisition unit, 22 Target organism detection unit, 23 Skeleton extraction unit, 24 Target organism tracking unit, 25 Movement feature calculation unit, 26 Curvature feature calculation unit, 27 Behavior determination unit, 28 Growth condition control unit, 30 Storage device, 31 Reference behavior model, 40 Communication device, 50 Input/output interface, 60 Automatic feeding device, 70 Growth environment control device, 200 Remaining food amount acquisition unit, 201 Video acquisition unit, 202 Remaining food detection unit, 203 Remaining food amount calculation unit, 301 Learning unit, 400 Corpse number acquisition unit, 401 Video acquisition unit, 402 Corpse detection unit, 403 Corpse number calculation unit.

Claims (10)

a behavior determination unit that determines whether the behavior of the target organism is a feeding behavior using a behavior feature that is a feature calculated based on time transitions of at least one of the position and bending of each joint of the target organism, the behavior feature corresponding to the behavior of the target organism, and a reference behavior model that is a model that indicates a classification of the behavior of the target organism according to the behavior feature;
A behavior determination device comprising: a growth condition control unit that generates data for a device that feeds the target organism to determine whether or not to feed the target organism based on the result of determining whether the behavior of the target organism is feeding behavior . The behavior feature amount is
A movement feature corresponding to at least one of the movement, position, and posture of the target living thing;
The behavior determination device according to claim 1 , further comprising at least one of a bending feature amount which is a feature amount corresponding to at least one of the bending of the joint of the target organism and a joint among the joints of the target organism which moves in conjunction with the bending of the target joint. The behavior determination device further includes:
a skeleton extraction unit that extracts each joint of the target organism in each frame represented by video data representing a plurality of frames in which the target organism is captured, and generates information representing each of the extracted joints as skeleton information;
a target organism tracking unit that tracks the target organism based on the skeletal information and generates information indicating a tracking result as tracking information;
a motion feature amount calculation unit that calculates a motion feature amount of the target living thing based on the tracking information;
The behavior determining device according to claim 2 , further comprising a curvature feature amount calculation unit that calculates a curvature feature amount of the target organism based on the skeletal information. The behavior determination device according to any one of claims 1 to 3, wherein the growth condition control unit generates data used to adjust the growth environment of the target organism depending on the result of determining whether the behavior of the target organism is feeding behavior or not . The behavior determination device according to any one of claims 1 to 3, wherein the reference behavior model includes at least one of a feeding behavior model showing an index for classifying the behavior of the target organism into feeding behavior, which is the behavior of the target organism when the target organism desires food, according to the behavior feature amount, and a non-feeding behavior model showing an index for classifying the behavior of the target organism into non-feeding behavior, which is the behavior other than the feeding behavior, according to the behavior feature amount. The behavior determination device further includes:
6. The behavior determination device of claim 5, further comprising a learning unit that learns a model that classifies the behavior of the target organism into either the feeding behavior or the non-feeding behavior according to the behavioral features of the target organism using non-feeding behavior data, which is data when the amount of remaining food, which is the amount of food remaining after the feeding time, which is the time required for the target organism to ingest the fed food, has elapsed from the feeding time, and which indicates the behavioral features of the target organism before feeding, and which is labeled as data corresponding to the non-feeding behavior, and feeding behavior data, which is data when the amount of remaining food is less than the standard remaining feed amount, and which indicates the behavioral features of the target organism before feeding, and which is labeled as data corresponding to the feeding behavior. The behavior determination device according to any one of claims 1 to 3, wherein the reference behavior model includes at least one of a normal behavior model showing an index for classifying the behavior of the target organism into normal behavior, which is the behavior of the target organism when the health condition of the target organism is good, according to the behavior feature amount, and an abnormal behavior model showing an index for classifying the behavior of the target organism into abnormal behavior, which is the behavior that is not the normal behavior, according to the behavior feature amount. The behavior determination device further includes:
When a group of aquatic organisms consisting of a plurality of aquatic organisms corresponding to the target organisms is kept in an aquarium,
8. The behavior determination device according to claim 7, further comprising a learning unit that uses abnormal behavior data, which is data when an abnormal number or more of corpses of aquatic organisms constituting the aquatic organisms are detected in the aquarium, indicating behavioral features of each aquatic organism constituting the aquatic organisms before the abnormal number or more of corpses of the aquatic organisms constituting the aquatic organisms are detected in the aquarium, and the data is labeled as data corresponding to the abnormal behavior, and normal behavior data, which is data when an abnormal number or more of corpses of aquatic organisms constituting the aquatic organisms are detected in the aquarium, indicating behavioral features of each aquatic organism constituting the aquatic organisms before the abnormal number or more of corpses of the aquatic organisms constituting the aquatic organisms are detected in the aquarium, and the data is labeled as data corresponding to the normal behavior, to learn a model that classifies the behavior of the target organism into either the normal behavior or the abnormal behavior according to the behavioral features of the target organism. a computer determines whether or not the behavior of the target organism is a feeding behavior using a behavior feature calculated based on time transitions of at least one of the position and bending of each joint of the target organism, the behavior feature being a feature corresponding to the behavior of the target organism, and a reference behavior model including a model indicating a classification of the behavior of the target organism according to the behavior feature;
A behavior determination method in which the computer generates data for determining whether or not an apparatus that feeds the target organism will feed the target organism based on the result of determining whether or not the behavior of the target organism is feeding behavior . a behavior determination process for determining whether or not the behavior of the target organism is a feeding behavior using behavior features calculated based on time transitions of at least one of the positions and bending of each joint of the target organism, the behavior features corresponding to the behavior of the target organism, and a reference behavior model formed of a model showing a classification of the behavior of the target organism according to the behavior features;
A behavior determination program that causes a behavior determination device, which is a computer, to execute a growth condition control process that generates data for the device that feeds the target organism to decide whether or not to feed the target organism, based on the result of determining whether the behavior of the target organism is feeding behavior .

Images