JP7779568B2 - Behavioral analysis device, behavioral analysis method, behavioral analysis program, and recording medium - Google Patents

Description

The present invention relates to a behavioral analysis device, a behavioral analysis method, a behavioral analysis program, and a recording medium.

To develop labor-saving herd management techniques that are compatible with the expansion of dairy farming operations, there is a demand for methods that can automatically monitor the behavior of each individual cow within a herd. One practical technique for monitoring each individual cow within a herd is to attach wearable sensors such as accelerometers to each individual. One example of such technology is to attach collar-type sensors to cows to gauge their activity levels, calculate their activity, rumination, and rest times, and use this information to detect cows that require attention, such as those in heat or suspected of illness.

However, with such technology, the larger the cattle herd, the higher the implementation costs and the greater the labor required to manage the equipment. The costs of implementing and operating such technology are estimated to be several tens of thousands of yen per collar-type sensor used to monitor cattle activity, and the usage fee is several thousand yen per head per year.

In contrast, technology for monitoring cows based on video analysis has the advantage of reducing costs and being easier to manage, as it does not require sensors to be attached to the cows. Examples of technology for monitoring cows based on video analysis include the technologies described in Non-Patent Documents 1 and 2. Non-Patent Document 1 describes how hoof disease can be detected by analyzing the cow's gait from depth camera footage. Non-Patent Document 2 also describes how changes in posture, which are signs of delivery, can be detected from photographed images of the cow.

Furthermore, examples of technologies for identifying and tracking individuals include those described in Patent Documents 1 and 2. Patent Document 1 describes a configuration that detects people moving within a specific area, acquires still images of the people, determines whether they are residents or non-residents, and tracks their behavior over time. Furthermore, Patent Document 2 describes a configuration that detects moving objects in video and extracts features for identifying people.

Patent Publication No. 2017-224249 Patent Publication No. 2019-169843

Sho Sunagawa et al., Information Processing Society of Japan Technical Report, Vol. 2017-CVIM-206, No. 2 (2017) Yusuke Okimoto et al., Proceedings of the 32nd National Conference of the Japanese Society for Artificial Intelligence (2018)

However, the technologies described in Non-Patent Documents 1 and 2 are designed to monitor a single individual, and are therefore not suitable for analyzing the behavior of multiple individual cows.

On the other hand, the technologies described in Patent Documents 1 and 2 monitor multiple individuals and describe the use of a machine-learned model for individual identification, but make no mention of collecting training data for machine learning. In order to create an individual identification technology that can be used universally for various herds of cattle with different individual compositions and imaging conditions, an individual identification system is required that uses a trained model that can be easily constructed for each herd.

One aspect of the present invention has been made to solve the above-mentioned problems, and its purpose is to realize a highly versatile video-based behavior analysis system that can be applied to a variety of individuals.

In order to solve the above problems, the present invention includes the following aspects.
1) The behavioral analysis device includes an image acquisition unit that acquires an image including an individual, a detection unit that detects the individual in the image, a position identification unit that identifies the position of a body part of the detected individual, an information acquisition unit that acquires time-series information that indicates changes in the identified position within a predetermined period of time, and a behavioral analysis unit that analyzes the behavioral pattern of the individual by referring to the acquired time-series information.
2) In the behavioral analysis device described in 1), the behavioral pattern of the individual analyzed by the behavioral analysis unit is selected from the group consisting of recumbent resting, recumbent ruminating, standing resting, standing ruminating, and feeding.
3) In the behavioral analysis device described in 1) or 2), the position identification unit uses an image including an individual as input data, and inputs the image acquired by the image acquisition unit as input data into a position identification model that has been machine-trained to identify the position of a body part of the individual in the image, thereby identifying the position that is output.
4) In the behavioral analysis device according to any one of 1) to 3), the position identification unit acquires coordinates of the position of a body part of the individual.
5) In the behavioral analysis device described in any one of 1) to 4), the behavioral analysis unit uses time-series information as input data, and inputs the time-series information acquired by the information acquisition unit as input data into an analysis model that has been machine-learned to estimate a behavioral type corresponding to the time-series information, thereby acquiring a behavioral type as an analysis result.
6) In the behavioral analysis device described in any one of 1) to 5), the position identification unit identifies the position by referring to the posture of the individual that is output by inputting image data of the individual as input data into a posture estimation model that has been machine-trained to detect the skeleton of the individual from an image representing a part of the individual's body and estimate the posture.
7) The behavioral analysis device according to any one of 1) to 6) further includes an identification unit that identifies the individual detected by the detection unit.
8) The behavioral analysis method includes an image acquisition step of acquiring an image including an individual, a detection step of detecting the individual in the image, a position identification step of identifying a position of a body part of the detected individual, an information acquisition step of acquiring time-series information representing a change in the identified position within a predetermined period of time, and a behavioral analysis step of analyzing the behavioral pattern of the individual by referring to the acquired time-series information.
9) The behavioral analysis program is a behavioral analysis program for causing a computer to function as the behavioral analysis device described in any one of 1) to 7), and causes the computer to function as the image acquisition unit, the detection unit, the position identification unit, the information acquisition unit, and the behavioral analysis unit.
10) A computer-readable recording medium has the behavioral analysis program described in 9) recorded thereon.

One aspect of the present invention makes it possible to realize a highly versatile video-based behavior analysis system that can be applied to a variety of individuals.

1 is a block diagram showing a configuration of a main part of a learning data generation device according to an aspect of the present invention; 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. 1 is a diagram illustrating a learning data generation process of a learning data generation device according to an aspect of the present invention. FIG. 10 is a flowchart showing a learning data generation process performed by the learning data generation device. 1 is a block diagram showing a configuration of a main part of a learning device according to an aspect of the present invention; 10 is a flowchart showing a learning process performed by the learning device. 1 is a block diagram showing a configuration of a main part of an individual identification device according to an embodiment of the present invention; 10 is a flowchart showing an individual identification process performed by the individual identification device. 1 is a block diagram showing a configuration of a main part of a behavior analysis device according to an aspect of the present invention. 10 is a flowchart showing a behavior analysis process performed by the behavior analysis device. 1 is a diagram illustrating an example of individual identification used in a learning data generation device according to an embodiment of the present invention. FIG. FIG. 1 is a diagram illustrating an example of an object detection machine learning model used to detect individuals in an image in a learning data generation device according to an aspect of the present invention. FIG. 1 is a diagram illustrating an example of a skeleton detection machine learning model used to detect individuals in an image in a learning data generation device according to one embodiment of the present invention. 10A and 10B are diagrams illustrating an example of a form in which a body region is identified in the learning data generation device according to an aspect of the present invention. 1 is a diagram illustrating an example of a posture estimation model used to identify an image of the entire body from an image showing a part of the body in a learning data generation device according to an aspect of the present invention. FIG. 1 is a block diagram showing a configuration of a main part of a behavioral analysis system according to an embodiment of the present invention. FIG. 10 is a diagram illustrating examples of body parts of an individual identified by the behavioral analysis device. FIG. 2 is a diagram illustrating an example of an individual's behavioral pattern analyzed by the behavioral analysis device. 10 is a flowchart showing a behavioral analysis process performed by the behavioral analysis device.

[Learning data generation device]
A learning data generation device according to one aspect of the present invention includes an image acquisition unit that acquires an image including an individual, a moving object detection unit that detects a moving object in the acquired image, a framing unit that frames a body region of the detected moving object that includes the entire body of the individual, an extraction unit that extracts a body region image of the framed body region, and a generation unit that generates learning data for an individual identification model that identifies individuals from the extracted body region image.

Figure 1 is a block diagram showing the main components of a training data generation device according to one embodiment of the present invention. The training data generation device 10 includes an image acquisition unit 11, a moving object detection unit 12, a framing unit 13, an extraction unit 14, a training data generation unit (generation unit) 15, and a training data storage unit 16. The training data generation device 10 generates training data used to build an individual identification model through machine learning.

(individual)
The individual identification model, which is machine-learned using training data generated by the training data generation device 10, is a machine-learned model that identifies individuals in images of multiple individuals. The individuals identified by the individual identification model may be humans, animals, other moving objects, etc., but are preferably individuals managed in groups within a closed space. The individuals identified by the individual identification model are preferably livestock individuals raised in large numbers, even more preferably cattle, even more preferably dairy cows, and most preferably Holstein dairy cows. While the following describes Holstein dairy cows as an example of individuals to be identified, the present invention is not limited thereto. Other dairy cows or other animals can be identified in the same manner as Holstein dairy cows by utilizing differences in patterns, body shapes, or simple markers. For example, the present invention can also be applied to koi carp, pigs, cats, dogs, etc. An example of a configuration using simple markers is a configuration in which a label such as a number is attached to a collar, as shown in FIG. 16 . FIG. 16 is a diagram illustrating an example of individual identification used in a training data generation device according to one aspect of the present invention. By using such simple markers, it is possible to easily identify individuals even of species that are difficult to identify by pattern, and the present invention can be applied to individuals of various species.

The individuals kept in large numbers may be free-roaming indoors, as shown in Figure 2, or may be tethered and kept on a chain or the like. Figure 2 is a diagram illustrating the training data generation process of a training data generation device according to one aspect of the present invention. The individuals kept in large numbers may also be kept in a controlled outdoor area. In this embodiment, an example will be described in which the individuals are cows.

(Image acquisition unit 11)
The image acquisition unit 11 acquires images including individual cattle. The image acquisition unit 11 acquires images including the individual cattle from the camera 20 that captured the image of the individual cattle via an input/output unit (not shown). The image acquisition unit 11 acquires images including the individual cattle from the camera 20 by wireless communication or wired communication. The image acquisition unit 11 may acquire images every time the camera 20 captures an image of the individual cattle, or may acquire images at predetermined time intervals.

The camera 20 may photograph the individual cattle from above, or from an obliquely upward angle. The learning data generation device 10 can suitably use not only images of the individual cattle photographed from above, but also images photographed from an obliquely upward angle. As an example, if the individual cattle are raised indoors, the camera 20 may be installed on the ceiling or on a wall. Furthermore, if the individual cattle are raised outdoors, the camera 20 may be installed by fixing it to a pole, fence, etc.

The image including the individual cattle is an image of the individual cattle, and may be a visible light image, an infrared image taken with a thermal camera or the like, or a three-dimensional image obtained with a three-dimensional laser sensor or the like. By using an infrared image or a three-dimensional image taken with a laser sensor, it is possible to obtain images at night without using lighting. Furthermore, the image including the individual cattle may be a still image or a video. The image acquisition unit 11 preferably acquires images including the individual cattle from multiple cameras 20, and more preferably, the multiple cameras 20 acquire images of the individual cattle from different directions. The image acquisition unit 11 sends the acquired image including the individual cattle to the moving object detection unit 12.

(Moving object detection unit 12)
The moving object detection unit 12 detects moving objects in the acquired images including the individual cattle. The moving object detection unit 12 detects the movement of objects in the images including the individual cattle and detects the moving objects as moving objects. The moving object detection unit 12 detects moving objects in the images by, for example, inter-frame difference processing that compares multiple image frames captured at different times, or background difference processing that compares the captured image with a background image acquired in advance. The moving object detection unit 12 acquires position information of the detected moving object in the image and sends it to the enclosing unit 13.

(Framed portion 13)
The framing unit 13 frames a body region including the entire body of a bovine individual among the detected moving objects. When the detected moving object is a bovine individual, the framing unit 13 frames the body region of the bovine individual. The framing unit 13 identifies the body region of the bovine individual in an image of a partial region corresponding to the acquired position information of the moving object and frames the body region.

If a frame is added to an image of a partial region corresponding to the position information of a moving object without specifying the body region of the bovine individual, the frame may be added to only a part of the body, such as the head or tail of the bovine individual, as shown in Figure 3. In addition, a frame may be added to a moving object other than the bovine individual.

On the other hand, the framing unit 13 identifies and frames the body region of the individual cow in the image of the partial region corresponding to the position information of the moving object, so that the entire body of the individual cow is framed so that it is inside the frame, as shown in Figure 4. Figure 4 is a diagram illustrating the learning data generation process of a learning data generation device according to one aspect of the present invention.

The body region of the individual bovine animal in the framing unit 13 can be identified, for example, by using a trained model that has been machine-learned to identify the body region of the individual bovine animal, or by methods such as feature point matching and pattern matching. That is, the framing unit 13 applies a frame to the body region output by inputting image data of the detected moving object as input data into a discrimination model that has been machine-learned to extract a body region including the entire body of the individual bovine animal from an image containing the individual bovine animal. Such a machine-learned discrimination model can be obtained by applying known learning algorithms such as support vector machines, random forests, k-means methods, and convolutional neural networks to training data such as that shown in Figure 5, which contains accumulated images of the entire bodies of thousands to tens of thousands of individual bovine animals. Figure 5 is a diagram illustrating the training data generation process of a training data generation device according to one embodiment of the present invention.

Furthermore, in cases where the captured image contains only part of the individual cow, where the individual cows are overlapping, or where part of the individual cow is hidden by an obstacle such as a pillar or wall, the framing unit 13 generates a body region image from the image representing part of the individual cow and adds a frame to the generated body region image. In other words, the framing unit 13 inputs image data of the detected moving object as input data into a generative model that has been machine-learned to generate an image representing the entire body of an individual from an image representing part of the individual's body, and uses the image representing the entire body of the individual that is output as the body region, and adds a frame to the body region.

Such generative models can include known generative models such as variational autoencoders (VAEs) and generative adversarial networks (GANs). For example, when using a variational autoencoder, as shown in Figure 6, a numerical value representing the features of image data is obtained by convolution in the encoder from image data of a portion of a cow's body. Then, an image of the entire body of the cow is reconstructed by deconvolving the numerical value representing the features. Figure 6 is a diagram illustrating the training data generation process of a training data generation device according to one aspect of the present invention. For example, as shown in Figure 6, a machine-learned generative model can be obtained by applying a known generative model to training data accumulated from images of the entire bodies of thousands to tens of thousands of cows. In this way, by using a generative model trained to reconstruct a body region image of the entire body from an image of a portion of the body of a cow, an image of the entire body of the cow can be reconstructed using an image of a portion of the body of the cow as input image data. Note that in the example shown in Figure 6, the data output from the generative model may be only a frame surrounding the entire body; patterns, colors, etc. may not be output.

Figures 7 and 8 are diagrams illustrating the training data generation process of a training data generation device according to one embodiment of the present invention. When image data representing a portion of a bovine body is input to a machine-learned generative model, as shown in Figure 7, a value Z representing the characteristics of the image data is obtained by convolution in the decoder. The value Z is a value representing the characteristics of the input image, such as information about the scale of the image or the extent of the body to which the area included in the image corresponds. The obtained value Z is then restored by deconvolution to generate and output a body region image representing the body region of the bovine body. Furthermore, when image data representing a portion of a bovine body that includes features not included in the training data is input to this machine-learned generative model, as shown in Figure 7, a body region image of the bovine body is generated and output. In this way, even when unknown bovine image data is input to the generative model, data framing the entire body of the bovine body is output. In this way, even when image data representing a portion of a bovine body that includes features not included in the training data is input to the generative model, data framing the entire body of the bovine body is output. Furthermore, even when the scale of an image showing a part of a cow's body is larger than that of Figure 7, as in the input data shown in Figure 8, an image of the body region of the cow surrounded by a frame of a size according to the scale of the input image is output.

In this way, when the image of the moving object is only part of a cow, the framing unit 13 generates the body region of the cow and applies a frame by using a generative model that has been machine-learned to generate an image representing the entire body of the cow from an image representing part of the body.

(Extraction part 14)
The extraction unit 14 extracts the body region image with the frame added. The extraction unit 14 removes the image outside the frame from the image captured by the camera 20 and cuts out the body region image inside the frame. The extraction unit 14 sends the extracted body region image to the learning data generation unit 15.

(Learning Data Generation Unit 15)
The training data generation unit 15 generates training data for an individual identification model that identifies individuals from the extracted body region images. The training data may be unlabeled, unsupervised data, and may include, for example, hundreds to tens of thousands of pieces of body region image data. The training data generation unit 15 may generate training data by classifying and accumulating the body region images for each individual. The training data generation unit 15 may generate training data from the accumulated body region images when a predetermined number of body region images have been accumulated or when an instruction to generate training data is received from a user.

The classification of body region images for each individual by the learning data generation unit 15 can be performed, for example, by referring to information such as the similarity of the body region images and their positional relationship with previous and next images in the time series. Such classification of body region images for each individual may be performed using a classification estimation model that has been machine-learned to estimate the body region images based on their similarity or their positional relationship with previous and next images in the time series.

The training data generation unit 15 generates training data corresponding to the population identified by the individual identification model. For example, if the individual identification model, which performs machine learning using the training data, identifies population X, which includes 10 cattle individuals, from cattle individual No. 1 to cattle individual No. 10, the training data generation unit 15 generates training data including body region image data of the 10 cattle individuals included in population X. The training data generation unit 15 associates the generated training data with information about the corresponding population and stores it in the training data storage unit 16.

(Learning Data Storage Unit 16)
The generated learning data is stored in the learning data storage unit 16. The learning data storage unit 16 may also store data such as images acquired by the image acquisition unit 11 and position information in the images of moving objects detected by the moving object detection unit 12. The learning data storage unit 16 may also store a program indicating a series of processes required for the operation performed by the learning data generation device 10.

(Learning data generation process)
An example of the learning data generation process performed by the learning data generation device 10 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the learning data generation process performed by the learning data generation device 10 of Fig. 1.

First, in step S91, the image acquisition unit 11 acquires an image including an individual cow. Note that the learning data generation process by the learning data generation device 10 may begin by accepting input of captured image data from the camera 20.

Next, the moving object detection unit 12 detects a moving object in the acquired image containing the individual cow (step S92). Subsequently, in step S93, the framing unit 13 frames the body region of the individual cow among the detected moving objects. Next, in step S94, the extraction unit 14 extracts the framed body region image. Then, the learning data generation unit 15 generates learning data from the body region image (step S94). The learning data generation unit 15 stores the generated learning data in the learning data storage unit 16, and the learning data generation process ends.

In this way, the training data generation device 10 automatically generates training data used to build an individual identification model through machine learning by extracting and storing body region images of individual cattle detected as moving objects. This makes it easy to generate training data for building an individual identification model. Therefore, training data can be easily generated for each individual population, with various individual populations having different individual structures and shooting conditions. Furthermore, the training data generated by the training data generation device 10 includes body region images of individual cattle in various orientations. Therefore, by identifying individuals using an individual identification model trained using such training data, it becomes possible to identify individual cattle regardless of the orientation of the individual cattle in the image to be identified.

A traditional problem with machine learning for image recognition is the time and effort required to prepare training data. The training data generation device 10 automatically accumulates training data used in machine learning for individual identification models. This makes it possible to easily build individual identification models that are compatible with each dairy farming site, where the individual composition of cattle populations and photography conditions vary, for example, enabling the general-purpose use of individual identification models.

[Learning device]
A learning device according to one embodiment of the present invention includes a learning data acquisition unit that acquires learning data generated by a learning data generation device according to one embodiment of the present invention, and a learning unit that constructs an individual identification model that identifies individuals by using the acquired learning data to perform machine learning to identify individuals contained in an image.

Figure 10 is a block diagram showing the main components of a learning device according to one embodiment of the present invention. The learning device 100 includes a learning data acquisition unit 101, a learning unit 102, and a learning model storage unit 103. The learning device 100 constructs an individual identification model that identifies individuals.

(Learning Data Acquisition Unit 101)
The training data acquisition unit 101 acquires training data stored in the training data storage unit 16 of the training data generation device 10 via an input/output unit (not shown). The training data acquisition unit 101 acquires training data corresponding to a population to be identified by an individual identification model that is machine-learned in the training unit 102. For example, if the individual identification model is a trained model that identifies cattle individuals of population X that includes 10 cattle individuals, cattle individual No. 1 to cattle individual No. 10, the training data acquisition unit 101 acquires training data corresponding to population X. The training data acquisition unit 101 sends the acquired training data to the training unit 102.

(Learning unit 102)
The learning unit 102 uses the acquired learning data to construct an individual identification model that identifies individuals. The learning unit 102 uses the acquired learning data to perform machine learning of the individual identification model by applying a known learning algorithm such as a support vector machine, a random forest, a k-means method, or a convolutional neural network.

The machine-learned individual identification model constructed by the learning unit 102 is a trained model that outputs identification information that identifies an individual when image data including the individual is input as input data. For example, when the individual identification model identifies a cow individual in population X, the individual identification model outputs which cow individual in population X (cow individual No. 1, cow individual No. 2, ..., or cow individual No. 10) the cow individual in the input image data corresponds to. The learning unit 102 stores the machine-learned individual identification model in the learning model storage unit 103.

(Learning model storage unit 103)
The learning model storage unit 103 stores an individual identification model that has undergone machine learning. The learning model storage unit 103 may also store learning data acquired by the learning data acquisition unit 101. The learning model storage unit 103 may also store a program that indicates a series of processes required for the operation of the learning device 100.

(Learning process)
An example of the learning process performed by the learning device 100 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the learning process performed by the learning device 100 of Fig. 10.

First, in step S111, the training data acquisition unit 101 acquires training data from the training data generation device 10. Next, the training unit 102 uses the acquired training data to train an individual identification model that identifies individuals (step S112). The training unit 102 then stores the trained individual identification model in the training model storage unit 103 (step S113), and ends the training process.

In this way, the learning device 100 constructs an individual identification model through machine learning using learning data generated for each population. This makes it possible to easily construct an individual identification model that corresponds to each dairy farm site, where the individual composition of cattle populations and photography conditions vary. As a result, the individual identification model can be used for general purposes.

[Individual Identification Device]
An individual identification device according to one embodiment of the present invention includes an image acquisition unit that acquires an image including an individual, and an identification unit that outputs individual identification information of the individual included in the image by inputting data of the acquired image as input data into the individual identification model constructed by a learning device according to one embodiment of the present invention.

Figure 12 is a block diagram showing the main components of an individual identification device according to one embodiment of the present invention. The individual identification device 120 includes an image acquisition unit 121, an identification unit 122, and an identification image storage unit 123. The individual identification device identifies individuals in a population using an individual identification model.

(individual)
The individual identified by the individual identification device 120 may be a human, an animal, another moving object, etc., but is preferably an individual managed in groups in a closed space. The individual identified by the individual identification model is more preferably a livestock individual raised in large numbers, even more preferably a cattle individual, even more preferably a dairy cow individual, and most preferably a Holstein dairy cow individual. Note that the raised large numbers of individuals may be individuals that are allowed to roam free indoors, as shown in FIG. 2, or individuals that are kept tethered by a chain or the like. The raised large numbers of individuals may also be individuals that are raised in a managed outdoor area. In this embodiment, a case in which the individuals are cattle individuals will be described as an example.

(Image acquisition unit 121)
The image acquisition unit 121 acquires images including the individual bovine cattle. The image acquisition unit 121 acquires images including the individual bovine cattle from the camera 20 that captured the image of the individual bovine cattle via an input/output unit (not shown). The image acquisition unit 121 acquires images including the individual bovine cattle from the camera 20 by wireless communication or wired communication. The image acquisition unit 121 may acquire an image every time the camera 20 captures an image of the individual bovine cattle, or may acquire an image at predetermined time intervals.

The camera 20 may photograph the individual cattle from above, or from an obliquely upward angle. The individual identification device 120 can suitably use not only images of the individual cattle photographed from above, but also images photographed from an obliquely upward angle. As an example, if the individual cattle are raised indoors, the camera 20 may be installed on the ceiling or on a wall. Furthermore, if the individual cattle are raised outdoors, the camera 20 may be installed by fixing it to a pole, fence, etc.

The image including the individual cattle is an image of the individual cattle, and may be a visible light image, an infrared image taken with a thermal camera or the like, or a three-dimensional image obtained with a three-dimensional laser sensor or the like. By using an infrared image or a three-dimensional image taken with a laser sensor, it is possible to obtain images at night without using lighting. Furthermore, the image including the individual cattle may be a still image or a video. The image acquisition unit 121 sends the obtained image including the individual cattle to the identification unit 122.

(Identification unit 122)
The identification unit 122 inputs the data of the acquired image as input data into the individual identification model constructed by the learning device 100, and outputs individual identification information of the individual cattle in the image. The identification unit 122 acquires the individual identification model stored in the learning model storage unit 103 of the learning device 100 via an input/output unit (not shown). The individual identification model is a machine-learned model corresponding to the population identified by the individual identification device 120. For example, if the population identified by the individual identification device 120 is population X including 10 individual cattle, from cattle individual No. 1 to cattle individual No. 10, the individual identification model is a machine-learned model corresponding to population X.

The identification unit 122 associates the image data with individual identification information indicating which individual bovine animal is included in the acquired image, and stores the associated image data in the identified image storage unit 123. Furthermore, as shown in FIG. 4, the identification unit 122 may also attach the individual identification information to the image data and display it on a display unit (not shown). For example, as shown in FIG. 4, the identification unit 122 displays the individual bovine animal in the image by attaching a bovine individual number that identifies the individual bovine animal.

The image identified by the identification unit 122 may be an image obtained by extracting a body region image including the entire body of the individual bovine animal from the image acquisition unit 121. That is, the individual identification device 120 may further include a framing unit that adds a frame to the body region of the individual bovine animal in the image acquired by the image acquisition unit 121, and an extraction unit that extracts the framed body region image. The individual identification device 120 may also include a moving object detection unit that detects moving objects in the image acquired by the image acquisition unit 121. The identification unit 122 then inputs the body region image extracted by the extraction unit as input data into an individual identification model, thereby outputting individual identification information for the individual bovine animal in the image. Note that the image identified by the identification unit 122 may be the image itself acquired by the image acquisition unit 121 from the camera 20, for example, if the individual bovine animal in the image is alone.

In this way, the individual identification device 120 may use, as the image to be identified, a body region image that has been subjected to processing similar to the learning data generation processing performed by the learning data generation device 10, rather than the image itself acquired from the camera 20. This prevents the image to be identified from including parts unnecessary for individual identification, enabling more accurate individual identification.

(Identification image storage unit 123)
The identified image storage unit 123 stores an image of an individual identified. The identified image storage unit 123 may also store a program indicating a series of processes required for the operation of the individual identification device 120.

(Individual identification processing)
An example of the individual identification process performed by the individual identification device 120 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing the individual identification process performed by the individual identification device 120 of Fig. 12.

First, in step S131, the image acquisition unit 121 acquires an image including an individual cow. Note that the individual identification process by the individual identification device 120 may begin by accepting input of captured image data from the camera 20.

Next, the identification unit 122 reads out the trained individual identification model from the learning device 100 (step S132). Subsequently, in step S133, the identification unit 122 inputs the image data into the read individual identification model. Then, in step S134, the identification unit 122 associates the individual identification information of the individual cow output by the individual identification model with the image, stores the associated information in the identification image storage unit 123, and terminates the individual identification process.

In this way, the individual identification device 120 identifies individuals using an individual identification model that corresponds to each population, making it possible to identify individuals that are suited to each dairy farm site, where the individual composition of the cattle population and the photographing conditions vary. Furthermore, by using an individual identification model that has been trained by the learning device 100, it is possible to identify cattle regardless of the orientation of the individual cattle in the image to be identified. The individual identification device 120 allows for general-purpose use of the individual identification model.

[Behavior analysis device]
A behavioral analysis device according to one embodiment of the present invention includes an image acquisition unit that acquires images including individuals, an identification unit that outputs individual identification information for the individuals included in the images by inputting data from the acquired images as input data into an identification model for identifying individuals constructed by machine learning to identify the individuals included in the images using learning data generated by the learning data generation device, an identification image acquisition unit that acquires time-series images of the identified individuals, and an analysis unit that analyzes the behavior of the individuals by referring to the acquired images. That is, the behavioral analysis device includes an identification image acquisition unit that acquires time-series images of the individuals identified by the individual identification device according to one embodiment of the present invention, and an analysis unit that analyzes the behavior of the individuals by referring to the acquired images.

Figure 14 is a block diagram showing the main components of a behavior analysis device according to one embodiment of the present invention. The behavior analysis device 140 includes an identification image acquisition unit 141, an analysis unit 142, and an analysis result storage unit 143. The behavior analysis device tracks and analyzes the behavior of identified individuals.

(Identification image acquisition unit 141)
The identification image acquisition unit 141 acquires, via an input/output unit (not shown), images of individual cattle stored in the identification image storage unit 123 of the individual identification device 120. The identification image acquisition unit 141 acquires time-series images of individual cattle identified by the individual identification device 120. The identification image acquisition unit 141 sends the acquired time-series images of individual cattle to the analysis unit 142.

(Analysis Department 142)
The analysis unit 142 analyzes the behavior of the individual cattle by referring to the time-series images of the individual cattle. The time-series images of the individual cattle can be used to track the behavior of the individual cattle. The analysis unit 142 analyzes the behavior of the individual cattle over a predetermined period of time by tracking the behavior of the individual cattle based on the time-series images of the individual cattle. The analysis unit 142 analyzes, for example, the amount of activity, range of activity, and behavior pattern of the individual cattle over a predetermined period of time by tracking the behavior of the individual cattle based on the time-series images of the individual cattle. The analysis unit 142 stores the analysis results of the individual cattle in the analysis result storage unit 143.

The analysis unit 142 may predict the physical condition, estrus state, disease, etc. of an individual cow by analyzing its behavioral patterns, such as the amount of time it spends sitting, standing, or moving around. The analysis unit 142 may also manage the feeding of an individual cow by analyzing its behavioral patterns, such as the number of times and times it drinks water and the number of times and times it eats feed.

The analysis unit 142 may acquire position coordinates in the image of the individual bovine animal from the image of the individual bovine animal and convert these position coordinates into spatial coordinates of the space in which the individual bovine animal exists, thereby tracking and analyzing the range and behavioral patterns of the individual bovine animal within space. The conversion from position coordinates to spatial coordinates can be performed, for example, by converting the acquired position coordinates into spatial coordinates of the space in which the individual bovine animal exists, using information on a reference point on the image whose spatial coordinates are known. If the image of the individual bovine animal referenced by the analysis unit 142 is a body region image that includes the entire body of the individual bovine animal, the position coordinates of the individual bovine animal's feet can be acquired, thereby enabling the behavior of the individual bovine animal within space to be accurately tracked. The position coordinates or spatial coordinates of the individual bovine animal's feet may be displayed on a display unit (not shown) together with the body region image of the individual bovine animal, as shown in FIG. 4.

(Behavioral analysis processing)
An example of behavior analysis processing by behavior analysis apparatus 140 will be described with reference to Fig. 15. Fig. 15 is a flowchart showing behavior analysis processing by behavior analysis apparatus 140 of Fig. 14.

First, in step S151, the identification image acquisition unit 141 acquires time-series image data of the individual cow. Next, the analysis unit 142 tracks the behavior of the individual cow using the time-series images of the individual cow and analyzes the behavior of the individual cow (step S152). The analysis unit 142 stores the analysis results in the analysis result storage unit 143 and terminates the behavior analysis process.

In this way, the behavior analysis device 140 analyzes the behavior of individuals by tracking images of individuals identified using an individual identification model corresponding to each population, making it possible to perform accurate and easy individual behavior analysis that is suited to each dairy farm site, for example, where the individual composition of the cattle population and the shooting conditions vary. Note that behavior analysis systems equipped with the learning data generation device 10 shown in FIG. 1, the learning device 100 shown in FIG. 10, the individual identification device 120 shown in FIG. 12, and the behavior analysis device 140 shown in FIG. 14 are also included within the scope of the present invention.

[Method for generating learning data and method for analyzing behavior]
A training data generation method according to one aspect of the present invention includes an image acquisition step of acquiring an image including an individual, a moving object detection step of detecting a moving object in the acquired image, a framing step of framing a body region of the detected moving object that includes the entire body of the individual, an extraction step of extracting a body region image of the framing-added body region, and a generation step of generating training data for a discriminative model that identifies the individual from the extracted body region image. As an example, the training data generation method according to one aspect of the present invention is realized by the training data generation device according to one aspect of the present invention described above. Therefore, the description of the training data generation method according to one aspect of the present invention follows the description of the training data generation device according to one aspect of the present invention described above.

A behavioral analysis method according to one aspect of the present invention includes an image acquisition step of acquiring images containing individuals; an identification step of inputting data from the acquired images as input data into an identification model for identifying individuals, constructed by machine learning to identify individuals included in the images using learning data generated by the learning data generation method described above, thereby outputting individual identification information for the individuals included in the images; an identification image acquisition step of acquiring time-series images of the identified individuals; and an analysis step of analyzing the behavior of the individuals by referring to the acquired images. As an example, the behavioral analysis method according to one aspect of the present invention is realized by the behavioral analysis device according to one aspect of the present invention described above. Therefore, the description of the behavioral analysis method according to one aspect of the present invention follows the description of the behavioral analysis device according to one aspect of the present invention described above.

[Software implementation example]
The control blocks of the learning data generation device 10, the learning device 100, the individual identification device 120, and the behavioral analysis device 140 may be realized by logic circuits (hardware) formed on an integrated circuit (IC chip) or the like, or may be realized by software.

In the latter case, each of the above-mentioned devices includes a computer that executes instructions from a program, which is software that realizes each function. This computer includes, for example, one or more processors and a computer-readable recording medium that stores the program. The object of the present invention is achieved when the processor in the computer reads and executes the program from the recording medium. The processor may be, for example, a CPU (Central Processing Unit). The recording medium may be a "non-transitory tangible medium," such as a ROM (Read Only Memory), tape, disk, card, semiconductor memory, or programmable logic circuit. The device may also include a RAM (Random Access Memory) for expanding the program. The program may also be supplied to the computer via any transmission medium capable of transmitting the program (such as a communications network or broadcast waves). One aspect of the present invention may also be realized when the program is embodied through electronic transmission in the form of a data signal embedded in a carrier wave.

In other words, the learning data generation device, learning device, individual identification device, and behavioral analysis device 140 according to each aspect of the present invention may be realized by a computer. In this case, the scope of the present invention also includes programs that cause the computer to operate as the respective parts (software elements) of each of the devices, thereby realizing each of the devices on the computer, and computer-readable recording media on which such programs are recorded.

Specifically, the following programs also fall within the scope of the present invention:
A program for causing a computer to function as a learning data generation device according to each aspect of the present invention, the program comprising: an image acquisition step for acquiring an image including an individual; a moving object detection step for detecting a moving object in the acquired image; a framing step for adding a frame to a body region including the entire body of the individual among the detected moving objects; an extraction step for extracting a body region image of the body region to which the frame has been added; and a generation step for generating learning data for a discriminative model that identifies individuals from the extracted body region image:
A program for causing a computer to function as a behavioral analysis device according to any one of the aspects of the present invention, the program comprising: an image acquisition step of acquiring images including individuals; an identification step of inputting data from the acquired images as input data into an identification model for identifying individuals, the identification model being constructed by machine learning to identify individuals included in the images using learning data generated by the learning data generation device according to any one of the aspects of the present invention, thereby outputting individual identification information for the individuals included in the images; an identification image acquisition step of acquiring time-series images of the identified individuals; and an analysis step of analyzing the behavior of the individuals by referring to the acquired images.
A program for causing a computer to function as a learning device according to each aspect of the present invention, the program comprising: a learning data acquisition step for acquiring learning data generated by the learning data generation device according to each aspect of the present invention; and a learning step for constructing an identification model for identifying individuals by machine learning using the acquired learning data to identify individuals included in an image.
A program for causing a computer to function as an individual identification device relating to each aspect of the present invention, characterized in that the program executes an image acquisition step of acquiring an image including an individual, and an identification step of inputting data of the acquired image as input data into the identification model constructed by the learning device relating to each aspect of the present invention, thereby outputting individual identification information of the individual included in the image.

[Modification]
A learning data generation device according to another aspect of the present invention includes an image acquisition unit that acquires an image including an individual, a detection unit that detects the individual in the acquired image, a body region identification unit that identifies a body region including the entire body of the detected individual, an extraction unit that extracts a body region image of the identified body region, and a generation unit that generates learning data for a discrimination model that identifies individuals from the extracted body region image.

In other words, the method for detecting individuals in an image in a learning data generation device is not limited to a moving object detection method, and other methods for detecting individuals may be used. Other examples of detecting individuals in an image include methods that utilize machine learning. Therefore, a configuration in a learning data generation device that uses a detection unit that detects individuals in an image using machine learning instead of a moving object detection unit is also included in the scope of the present invention.

When using machine learning to detect individuals in an image, it is preferable to create a machine learning model suitable for detecting individuals depending on the direction in which the image was taken. For example, when using an image of an individual taken from diagonally above, a machine learning model created to be suitable for detecting individuals in an image taken from diagonally above is used. Methods for detecting individuals in an image using machine learning include methods for detecting objects and methods for detecting the skeleton of an individual using machine learning. Such a detection unit can also be used in an individual identification device according to one aspect of the present invention.

When using a machine learning model created to detect objects from images, as shown in Figure 17, image data of individual cows is input as input data into the object detection machine learning model, and the detection result of the cow individual part in the image is output. When using a machine learning model created to detect skeletons from images, as shown in Figure 18, image data of individual cows is input as input data into the skeleton detection machine learning model. This detects the skeleton positions of individual cows in the image and organizes them for each individual, and the detection result of the cow individual part in the image is output.

In addition, in the training data generation device, the method for identifying the body region including the entire body of a detected individual is not limited to the method of adding a frame, and other methods for identifying the body region of an individual may be used. Another example of identifying the body region of an individual is a method using image segmentation, as shown in FIG. 19. FIG. 19 is a diagram illustrating an example of how body regions are identified in a training data generation device according to one aspect of the present invention. As shown in FIG. 19, a body region is identified using image segmentation in an image including a cow individual, and, for example, position coordinates indicating only the identified body region range are obtained. In this way, a configuration in which a training data generation device uses a body region identification unit that identifies the body region of an individual using image segmentation instead of a frame is also included in the scope of the present invention. Furthermore, such a body region identification unit can also be used in an individual identification device according to one aspect of the present invention.

Furthermore, in the learning data generation device, the method for identifying an image of the entire body from an image representing a portion of the individual's body is not limited to the method of restoring images representing other portions of the body, and may be realized by other methods for identifying an image of the entire body from an image representing a portion of the body. Another example of identifying an image of the entire body from an image representing a portion of the body is a method that refers to a posture estimated based on skeletal detection using machine learning.

Figure 20 shows an example of identifying an image of the entire body from an image showing a portion of the body by referencing the posture estimated based on skeleton detection. Figure 20 is a diagram illustrating an example of a posture estimation model used to identify an image of the entire body from an image showing a portion of the body in a learning data generation device according to one aspect of the present invention. As shown in Figure 20, when image data containing only a portion of an individual cow is acquired, the positions of the parts not included in the image (invisible parts) are identified by referencing the posture estimated from the parts included in the image (visible parts).

For example, in the image data of Figure 20, only seven parts are visible: the nose, left eye, right eye, left ear, right ear, left shoulder, and left hip. When this image data is input into the posture estimation model, machine learning is used to detect the skeleton of the visible parts. Skeletal frame information including the positions of the 10 invisible parts (right shoulder, right hip, left front ankle, left front hoof, left hind ankle, left hind hoof, right front ankle, right front hoof, right hind ankle, right hind hoof) is then output as the posture estimation result. Note that the parts of the cow shown here are examples, and the types and numbers of parts included and not included in the image are not limited to these and can be set as appropriate.

This makes it possible to identify the entire body region and use it in analyzing the individual's behavior, even when there is an obstruction in the image and only a portion of the individual is included. In this way, information about the body parts included in the image is used as information for estimating parts not included in the image. Furthermore, the position of each body part detected in images before and after the target image in the time series may also be used as input data, for example.

In this way, the scope of the present invention also includes a configuration in which, in a learning data generation device, a body region identification unit identifies a body region representing an individual's entire body by referring to the individual's posture output by inputting image data of the detected moving object as input data into a posture estimation model that has been machine-trained to detect the skeleton of the individual from an image representing part of the individual's body and estimate the posture. Such a body region identification unit can also be used in an individual identification device according to one aspect of the present invention.

[Behavioral Analysis System]
The behavioral type (behavioral pattern) of each individual animal is important information for animal management. For example, feeding behavior and resting behavior are indicators of whether maintenance behavior is being ensured. Furthermore, in ruminant livestock, rumination, among other behavioral patterns, is considered useful information that serves as a barometer reflecting feed intake, signs of various diseases, recovery status, or stress conditions. Previously, systems for determining behavioral patterns using wearable sensors and the like have been put into practical use, but have problems in terms of cost and management effort. Meanwhile, conventional image analysis has had difficulty detecting behaviors that require behavioral monitoring, such as rumination. According to one embodiment of the present invention, by using image analysis to determine an animal's behavioral pattern, behavioral pattern data can be obtained easily and labor-savingly. Furthermore, according to one embodiment of the present invention, even behavioral patterns that require behavioral monitoring can be detected.

With reference to Figure 21, a behavioral analysis system according to one aspect of the present invention will be described. Figure 21 is a block diagram showing the main components of a behavioral analysis system 200 according to one aspect of the present invention. The behavioral analysis system 200 includes a behavioral analysis device 220. The behavioral analysis system 200 also includes a camera 20, a learning data generation device 210, and a learning device 100.

(Behavioral analysis device 220)
The behavioral analysis device 220 includes an image acquisition unit 221, a detection unit 222, an identification unit 223, a position identification unit 224, an information acquisition unit 225, and a behavioral analysis unit 226. The behavioral analysis device 220 further includes an analysis result storage unit 227. The behavioral analysis device 220 detects and analyzes the behavioral patterns of individuals.

<Individual>
The individuals whose behavioral patterns are analyzed by the behavioral analysis device 220 may be humans, animals, other mobile objects, etc., but may also be individuals managed in groups in a closed space, or individuals managed individually. Such individuals include livestock individuals raised in large numbers and livestock individuals raised alone. Livestock individuals may be ruminant animals such as cattle, for example, dairy cows, more specifically, Holstein dairy cows. Individuals raised in large numbers may be individuals allowed to roam free indoors, or individuals kept tethered by chains or the like. Additionally, individuals raised in large numbers may be individuals raised in a managed outdoor area. In this embodiment, an example will be described in which the individuals are cattle.

<Image Acquisition Unit 221>
The image acquisition unit 221 acquires an image including an individual cattle photographed by the camera 20. The image acquisition unit 221 is configured in the same manner as the image acquisition unit 121 of the individual identification device 120 described above. Therefore, the description of the image acquisition unit 121 of the individual identification device 120 will be used to explain the image acquisition unit 221, and a detailed description thereof will be omitted. The camera 20 is also the same as the camera 20 described in relation to the individual identification device 120, and a detailed description thereof will be omitted.

<Detection unit 222>
The detection unit 222 detects individual cattle in the image acquired by the image acquisition unit 221. Methods for detecting individual cattle in an image include a moving object detection method and a method using machine learning. When using a moving object detection method, the detection unit 222 can be configured in the same manner as the moving object detection unit 12 of the learning data generation device 10. When using machine learning, a machine learning model such as the object detection machine learning model shown in FIG. 17 and the skeleton detection machine learning model shown in FIG. 18 can be used. Note that when a single individual cattle is included in the image, detection of the individual cattle is not necessary, and position identification, which will be described later, may be performed on the acquired image.

<Identification unit 223>
The identification unit 223 identifies the individual cattle detected by the detection unit 222. The identification unit 223 is configured in the same manner as the identification unit 122 of the individual identification device 120 described above. Therefore, the description of the identification unit 122 of the individual identification device 120 will be used to explain the identification unit 223, and a detailed description thereof will be omitted. Note that when a single individual cattle is included in the image, it is not necessary to identify the individual cattle, and position identification, which will be described later, may be performed on the acquired image.

<Position specifying unit 224>
The position identifying unit 224 identifies the positions of the identified body parts of the individual. The positions of the body parts of the individual identified by the position identifying unit 224 are, for example, the nose tip, mouth, forelegs, hooves, etc. Figure 22 shows an example of the position identifying unit 224 identifying body parts of a bovine individual. The left side of Figure 22 shows an example in which the nose tip, mouth, forelegs, and hooves of a sitting bovine individual are identified, and the right side shows an example in which the nose tip, mouth, forelegs, and hooves of a standing bovine individual are identified. Note that the number of body parts of the individual to be identified is not limited to this and may be fewer or more.

The position identification unit 224 may use machine learning to identify the position of an individual's body part. For example, an image including an individual is used as input data, and an image acquired by the image acquisition unit is input as input data into a position identification model that has been machine-learned to identify the position of the individual's body part in the image, thereby identifying the output position. By inputting an image including an individual into the position identification model, an image in which the individual's body part is identified can be output, as shown in Figure 22.

The position identification unit 224 may identify the position by acquiring the coordinates of the position of the individual's body part. The position identification unit 224 acquires the position coordinates of the individual's body part in the image. The position identification unit 224 may convert the acquired position coordinates into spatial coordinates of the space in which the individual cow exists. The position identification unit 224 acquires position coordinates in multiple frames of images captured at different times.

The position identification unit 224 may identify the position by referring to the posture of the individual that is output by inputting the detected image data of the individual as input data into a posture estimation model that has been machine-learned to detect the skeleton of the individual from an image representing a portion of the individual's body and estimate the posture. The posture estimation model used in the position identification unit 224 is the same as the posture estimation model shown in FIG. 20. The position identification unit 224 may also use a generative model that has been machine-learned to generate an image representing the entire body of the individual from an image representing a portion of the individual's body that is used in the enclosing unit 13 of the training data generation device 10. This makes it possible to identify the individual's body parts and obtain position coordinates even when there is an obstruction in the image and only a portion of the individual's body is included.

In addition, when the image data contains only a portion of the individual's body, the position identification unit 224 may identify the individual's body part in the image of the portion of the individual's body and acquire position coordinates, without generating an image representing the individual's entire body. In this case, for example, the individual's body part may be identified using a position identification model that has been trained to be able to identify the individual's body part from the image of the portion of the individual's body. Behavioral analysis can be performed even if the image of the portion of the individual's body is used as is, without reconstructing an image of the entire body from the image of the portion of the individual's body.

<Information acquisition unit 225>
The information acquisition unit 225 acquires time-series information that represents changes in the identified positions within a predetermined time. The time-series information acquired by the information acquisition unit 225 is information that represents changes in parts of the individual cow obtained from successive images within a predetermined time, and may be the average distance between each part and the average displacement between image frames of each part, which are obtained based on the position coordinates acquired by the position identification unit 224.

<Behavioral analysis department 226>
The behavioral pattern analysis unit 226 analyzes the behavioral pattern of the individual cattle by referring to the acquired time-series information. By referring to the time-series information, the behavioral pattern analysis unit 226 can detect the movements of the individual cattle and can also track the movements of the individual cattle. This makes it possible to analyze the behavioral pattern from the movements and movements of the individual cattle at a predetermined timing or within a predetermined period of time. The behavioral pattern analysis unit 226 stores the analysis results of the behavioral patterns of the individual cattle in the analysis result storage unit 217.

The behavioral patterns of individual cattle analyzed by the behavioral pattern analysis unit 226 are types of behavior, such as lying down resting, lying down ruminating, standing upright resting, standing upright ruminating, and feeding. As shown in Figure 23, by previously acquiring information that associates time-series information obtained from consecutive images within a predetermined period of time with the behavioral patterns of individual cattle corresponding to that time-series information, it is possible to analyze the behavioral patterns of individual cattle based on the time-series information.

The behavioral pattern analysis unit 226 may analyze the behavioral pattern of an individual cow using an analytical model constructed using training data in which time-series information is associated with the behavioral pattern of the individual cow. By inputting time-series information as input data into such an analytical model, the behavioral pattern of the individual cow is output. As shown in FIG. 23, the behavioral pattern analysis unit 226 may display the analyzed behavioral pattern of the individual cow on a display unit (not shown) together with an image of the individual cow. Furthermore, if the image data contains only part of the individual's body, the behavioral pattern analysis unit 226 may analyze the individual's behavioral pattern using only the position coordinate information of the identified part of the body. In this case, the individual's behavioral pattern may be analyzed using an analytical model that has been trained to be able to analyze the behavioral pattern even if there is missing data in the position coordinates of the body part.

<Learning Data Generation Device 210>
The learning data generation device 210 included in the behavioral analysis system 200 includes an image acquisition unit 11 that acquires images including individuals, a detection unit 212 that detects the individuals in the acquired images, an identification unit 213 that identifies the body region of the detected individuals, an extraction unit 14 that extracts a body region image of the identified body region, and a learning data generation unit 15 that generates learning data for a discrimination model that identifies individuals from the extracted body region image. The learning data generation device 210 further includes a learning data storage unit 16 that stores the generated learning data.

The detection unit 212 detects individuals in an image using a moving object detection method, a method using machine learning, or the like. Methods for detecting individuals in an image using machine learning include methods for detecting objects using machine learning and methods for detecting the skeleton of an individual. The identification unit 213 identifies the body region of the detected individual using a method for adding a frame to the body region including the entire body of the detected individual, a method using image segmentation, or the like.

The image acquisition unit 11, extraction unit 14, and training data generation unit 15 are identical to those components in the training data generation device 10 shown in Figure 1. That is, the training data generation device 210 is configured in the same way as the training data generation device 10. Therefore, the explanation of the training data generation device 210 will be based on the explanation of the training data generation device 10, and a detailed explanation will be omitted.

The training data generation device 210 can be used not only to generate training data for constructing a discrimination model, but also to generate training data for constructing an analysis model that the behavioral type analysis unit 226 uses to analyze behavioral types.

<Learning device 100>
The learning device 100 constructs a discriminative model, an analytical model, etc. using the training data generated by the training data generation device 210. The learning device 100 included in the behavior-based analysis system 200 is the same as the learning device 100 shown in Fig. 10. Therefore, the description of the learning device 100 shown in Fig. 10 will be used to explain the learning device 100 included in the behavior-based analysis system 200, and a detailed description thereof will be omitted.

<Behavioral analysis processing>
An example of the behavior-based analysis process performed by the behavior-based analysis device 220 will be described with reference to Fig. 24. Fig. 24 is a flowchart showing the behavior-based analysis process performed by the behavior-based analysis device 220 of Fig. 23 .

First, in step S241, the image acquisition unit 221 acquires an image including an individual cow. Note that the behavioral analysis process by the behavioral analysis device 220 may be initiated by accepting input of captured image data from the camera 20, or by accepting operational input from the user to start the behavioral analysis process.

Next, the detection unit 222 inputs the image data into a skeleton detection machine learning model and detects individual cattle in the image data (step S242). The identification unit 223 then inputs the image data of the detected individual cattle into an identification model, obtains individual identification information for the individual cattle, and identifies the individual cattle (step S243).

The position identification unit 224 inputs the image data of the identified bovine individual into a position identification model and obtains position coordinates that identify the body parts of the bovine individual (step S244). The information acquisition unit 225 then obtains the specified position coordinates for several consecutive frames and calculates the average distance between each part and the average displacement between frames of each part as time-series information (step S245).

The behavioral pattern analysis unit 226 inputs the time-series information into the analysis model and analyzes the behavioral patterns of the individual cows (step S246). The behavioral pattern analysis unit 226 stores the analysis results in the analysis result storage unit 227 and terminates the behavioral pattern analysis process.

In this way, the behavioral analysis device 220 can analyze an individual's behavioral pattern by identifying the location of an individual's body parts from image data and tracking changes in that location. This allows for labor-saving and easy behavioral analysis, and can also detect behavioral patterns that require understanding of actions such as rumination.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

10 Learning data generation device, 11 Image acquisition unit, 12 Moving object detection unit, 13 Framing unit, 14 Extraction unit, 15 Learning data generation unit (generation unit), 100 Learning device, 101 Learning data acquisition unit, 102 Learning unit, 120 Individual identification device, 121 Image acquisition unit, 122 Identification unit, 140 Behavioral analysis device, 141
Identification image acquisition unit, 142 Analysis unit, 200 Behavioral analysis system, 220 Behavioral analysis device, 221 Image acquisition unit, 222 Detection unit, 223 Identification unit, 224 Position identification unit, 225 Information acquisition unit, 226 Behavioral analysis unit

Claims (12)

an image acquisition unit that acquires an image including an individual animal that is a member of a group of animals ;
a detection unit that detects the individual in the image;
a position specifying unit that specifies the position of the detected body part of the individual;
an information acquisition unit that acquires time-series information representing changes in the identified position within a predetermined time;
a behavioral pattern analysis unit that analyzes the behavioral pattern of the individual by referring to the acquired time-series information. an image acquisition unit that acquires an image including an individual;
a detection unit that detects the individual in the image;
a position specifying unit that specifies the position of the detected body part of the individual;
an information acquisition unit that acquires time-series information representing changes in the identified position within a predetermined time;
a behavior pattern analysis unit that refers to the acquired time-series information and analyzes the behavior pattern of the individual;
It is equipped with
A behavioral analysis device, wherein the behavioral pattern of the individual analyzed by the behavioral analysis unit is selected from the group consisting of recumbent resting, recumbent ruminating, standing resting, standing ruminating, and feeding. an image acquisition unit that acquires an image including an individual;
a detection unit that detects the individual in the image;
a position specifying unit that specifies the position of the detected body part of the individual;
an information acquisition unit that acquires time-series information representing changes in the identified position within a predetermined time;
a behavior pattern analysis unit that refers to the acquired time-series information and analyzes the behavior pattern of the individual;
It is equipped with
The position identification unit is a behavioral analysis device that identifies the position by referring to the posture of an individual that is output by inputting image data of the individual as input data into a posture estimation model that has been machine-trained to detect the skeleton of the individual from an image representing part of the individual's body and estimate the posture. 4. The behavioral analysis device according to claim 1, wherein the position identification unit identifies the position output by inputting the image acquired by the image acquisition unit as input data into a position identification model that has been machine-trained to use an image including an individual as input data and identify the position of a body part of the individual in the image. The behavioral analysis device according to claim 1 , wherein the position specifying unit acquires coordinates of a position of a body part of the individual. 6. The behavioral analysis device according to claim 1, wherein the behavioral type analysis unit inputs time-series information acquired by the information acquisition unit as input data into an analysis model that has been machine-learned to estimate a behavioral type corresponding to the time-series information, and acquires, as an analysis result, a behavioral type that is output by inputting the time-series information acquired by the information acquisition unit. an identification unit that identifies the individual detected by the detection unit;
The behavioral analysis device according to claim 1 , further comprising: an image acquisition step of acquiring an image including an individual animal that is a multi-headed animal ;
a detecting step of detecting the individual in the image;
a location determination step of determining the location of the detected body part of the individual;
an information acquisition step of acquiring time-series information representing a change in the identified position within a predetermined time;
a behavioral pattern analysis step of analyzing the behavioral pattern of the individual by referring to the acquired time-series information. an image acquisition step of acquiring an image including the individual;
a detecting step of detecting the individual in the image;
a location determination step of determining the location of the detected body part of the individual;
an information acquisition step of acquiring time-series information representing a change in the identified position within a predetermined time;
a behavioral pattern analysis step of analyzing the behavioral pattern of the individual by referring to the acquired time-series information;
Including,
A behavioral pattern analysis method, wherein the behavioral pattern of the individual analyzed in the behavioral pattern analysis step is selected from the group consisting of recumbent resting, recumbent ruminating, standing resting, standing ruminating, and feeding. an image acquisition step of acquiring an image including the individual;
a detecting step of detecting the individual in the image;
a location determination step of determining the location of the detected body part of the individual;
an information acquisition step of acquiring time-series information representing a change in the identified position within a predetermined time;
a behavioral pattern analysis step of analyzing the behavioral pattern of the individual by referring to the acquired time-series information;
Including,
In the position identification step, the position is identified by referring to the posture of the individual output by inputting image data of the individual as input data into a posture estimation model that has been machine-learned to detect the skeleton of the individual from an image representing a part of the individual's body and estimate the posture. A behavioral analysis program for causing a computer to function as the behavioral analysis device described in any one of claims 1 to 7, the behavioral analysis program causing a computer to function as the image acquisition unit, the detection unit, the position identification unit, the information acquisition unit, and the behavioral analysis unit. A computer-readable recording medium on which the behavioral analysis program according to claim 11 is recorded.