JP6966038B2 - Animal behavior estimation device, animal behavior estimation method and program - Google Patents

Description

The present invention relates to an animal behavior estimation device, an animal behavior estimation method and a program.

Conventionally, there is a technique of estimating behavior based on the displacement of the position of a human joint or the like obtained from a moving image (see, for example, Patent Document 1).

Japanese Patent No. 6525181

In the case of humans, most of the body parts such as joints, which are sample points used for estimating behavior, are often obtained from images without being shielded by the torso or the like. In the case of humans, the skeleton looks the same whether viewed from the front or the back. However, in the case of cows, the torso is large, and the parts such as the joints of the limbs, which are the sample points used for estimating behavior, are shielded by the torso because they are on the opposite side of the viewpoint, because they are inside the torso. It may not be sufficiently obtained from the image because it cannot be detected from the appearance. Furthermore, quadrupeds such as cows are generally asymmetrical and more complex than humans. Therefore, it was difficult to apply the behavior estimation for humans to cattle as it is because there are many invisible sample points.

In view of the above circumstances, it is an object of the present invention to provide an animal behavior estimation device, an animal behavior estimation method and a program capable of improving the estimation accuracy of animal behavior based on images.

One aspect of the present invention is a feature point that can be visually recognized from the appearance of the estimation target animal in the image data from the image data obtained by the estimation target animal that is the behavior estimation target animal, and the estimation target animal in the image data. Based on a feature point detection unit that detects feature points that cannot be seen from the outside, and time-series feature point position information that indicates the positions of a plurality of types of the feature points detected by the feature point detection unit for each of the time-series image data. It is an animal behavior estimation device including a behavior estimation unit that estimates the behavior of the estimation target animal.

One aspect of the present invention is the above-mentioned animal behavior estimation device, and among the plurality of types of feature points used for estimating the behavior of an animal based on the position of the feature point detected by the feature point detection unit. The behavior estimation unit further includes a feature point position estimation unit that estimates the position of a type of feature point that was not detected by the feature point detection unit, and the behavior estimation unit detects the feature point detection unit for each of the time-series image data. The behavior of the estimated target animal is estimated based on the position of the feature point and the time-series feature point position information indicating the position of the feature point estimated by the feature point position estimation unit.

One aspect of the present invention is the above-mentioned animal behavior estimation device, and when a plurality of estimation target animals are photographed in the image data, the feature point detection unit determines the type and position of the detected feature points. Based on this, the estimated target animal to which the feature point detected from the image data belongs is determined.

One aspect of the present invention is the above-mentioned animal behavior estimation device, and the behavior estimation unit uses machine-learned behavior using information on the positions of a plurality of types of feature points in a time series and information on correct behavior as learning data. The behavior of the estimated target animal is estimated based on the estimation model and the time-series feature point position information.

One aspect of the present invention is the above-mentioned animal behavior estimation device, and the correct behavior information is information indicating the type of behavior, information indicating whether it is normal or not, or normal or abnormal. It is information indicating whether or not.

One aspect of the present invention is the above-mentioned animal behavior estimation device, in which the feature point detection unit determines the image data captured by the animal and the type and position of the feature points specified by the user in the image data. Feature points are detected from the image data using a machine-learned feature point detection model as training data.

One aspect of the present invention is the above-mentioned animal behavior estimation device, and the feature point position estimation unit is a machine that uses information on the type and position of feature points specified by a user in image data captured by an animal as learning data. Based on the learned feature point position estimation model and the position of the feature point detected by the feature point detection unit, the position of a type of feature point not detected by the feature point detection unit is estimated.

One aspect of the present invention is the animal behavior estimation device described above, further comprising a notification unit that outputs a notification when the behavior estimated by the behavior estimation unit is an abnormal behavior. The abnormal behavior of the target for which the notification is output may be specified by the user.

One aspect of the present invention is the animal behavior estimation device described above, wherein the animal is a quadruped, and the types of feature points include a face, tongue, back, tail and joints. Facial features include the crown, eyes, ears, nose and mouth.

One aspect of the present invention is a feature point that can be visually recognized from the appearance of the estimation target animal in the image data from the image data obtained by the estimation target animal that is the behavior estimation target animal, and the estimation target animal in the image data. Based on a feature point detection step that detects feature points that cannot be seen from the outside, and time-series feature point position information that indicates the positions of a plurality of types of the feature points detected in the feature point detection step for each of the time-series image data. This is an animal behavior estimation method including a behavior estimation step for estimating the behavior of the estimation target animal.

One aspect of the present invention is a program for making a computer function as any of the above-mentioned animal behavior estimation devices.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to improve the estimation accuracy of animal behavior based on images.

It is a figure which shows the whole structure example of the animal behavior estimation system by one Embodiment of this invention. It is a block diagram which shows the structure of the learning apparatus by the same embodiment. It is a block diagram which shows the structure of the 1st animal behavior estimation apparatus by the same embodiment. It is a block diagram which shows the structure of the 2nd animal behavior estimation apparatus by the same embodiment. It is a figure which shows the sample point of the cow by the same embodiment. This is an example of a screen displayed by the learning device according to the same embodiment. It is a figure which shows the example of the time-series feature point position information by the same embodiment. It is a figure which shows the example of the feature point detected and estimated by the 1st animal behavior estimation apparatus by the same embodiment. It is a figure which shows the example which arranged the feature point detected and estimated by the 1st animal behavior estimation apparatus by the same embodiment in time series. It is a flow figure which shows the processing of the learning apparatus by the same embodiment. It is a flow chart which shows the animal behavior estimation processing of the animal behavior estimation system by the same embodiment. It is a block diagram which shows the structure of the animal behavior estimation apparatus by the same embodiment. It is a block diagram which shows the structure of the animal behavior estimation apparatus by the same embodiment. It is a flow chart which shows the animal behavior estimation processing of the animal behavior estimation apparatus by the same embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The animal behavior estimation device of the embodiment of the present invention estimates the behavior of an animal to be behavior-estimated based on the detected feature points. As this feature point, a sample point capable of expressing the posture of the animal is used. The animal to be the target of the behavior estimation according to the present embodiment is an animal other than a human being, for example, a quadruped animal such as a cow. In the following, a case where the animal to be the target of behavior estimation is a cow will be described as an example. The behavior estimation is an estimation of the type of behavior of what kind of behavior it is, or an estimation of whether or not it is a normal behavior.

As a conventional technique, there is a system for detecting abnormal behavior of livestock such as cattle among animals by analyzing the entire moving image. However, in this conventional technique, in the case of an animal whose whole body is black, such as Japanese Black cattle, the discrimination ability is lowered depending on the color, and there is a problem in terms of accuracy. In addition, when multiple cows are stacked, it is difficult to separate and recognize each individual. In addition to the fact that quadrupeds are more complicated to move than humans, when the body is large like a cow, the movement of the limbs on the opposite side of the viewpoint is blocked, and the movement of the limbs is often not observable. Therefore, even if the conventional technique of analyzing the entire moving image to estimate the behavior or the conventional technique of estimating the behavior in time series using the joints detected from the moving image is applied, the part that cannot be seen visually. It was difficult to estimate the behavior of animals in which there are many animals.

In the present embodiment, the sample points used for estimating the behavior (movement) of an animal having complicated limb movements are determined including the sample points of the limbs that are difficult to see or are invisible. Then, the user adds an annotation (annotation) for each sample point on the image of the animal. By adding the sample points of the invisible part to the sample points of the visible part, the posture of the animal can be transparently expressed in two or three dimensions. In the present embodiment, the feature point detection model for detecting the feature points to be the sample points from the image is learned by machine learning using the sample points annotated in this way as the training data. By machine learning using the same learning data, a feature point position estimation model for estimating the position of another feature point from the position of some feature points may be further learned. In addition, by using the annotation of the sample points corresponding to each feature point, even when a plurality of animals are photographed in the image, the feature points are separated for each individual to obtain an expression of each behavior. In the present embodiment, the movement of the animal represented by the position of each feature point in the time series and the correct behavior are used as learning data, and the behavior estimation for estimating the behavior from the position of each feature point in the time series. Learn the model. For example, from the position of each feature point obtained from one still image, it is difficult to determine whether the cow is in that posture as a result of sitting or as a result of falling. By using the change in the position of each feature point in the time series as an input, it is possible to determine whether the cow is a normal behavior of sitting or an abnormal behavior of falling.

After learning each model, the animal behavior estimation device of the present embodiment features a portion of the animal to be behavior-estimated (hereinafter referred to as an estimation target animal) that is visible using the feature point detection model from the captured image. A point is detected, and the feature point of the invisible part is estimated using the detected feature point and the feature point position estimation model. The animal behavior estimation device of the present embodiment adds the estimated feature points to the feature points of the visible part, and estimates the behavior of the animal using the time-series position change of those feature points and the behavior estimation model. ..

In the case of a quadruped, unlike a human, the movement is more complicated than that of a human because the movements of left and right and back and forth are not targeted. Therefore, in the present embodiment, not only the visible part but also the invisible part Is also learned by the skeleton represented by the feature points and used for behavior estimation. In addition, the sample points used as feature points include sample points of parts not found in humans such as the tongue and tail, and sample points such as the back that are not important for estimating human behavior. As a result, it is possible to dramatically improve the estimation accuracy of animal behavior, including behavior peculiar to animals.

FIG. 1 is a diagram showing an overall configuration example of an animal behavior estimation system 1 according to an embodiment of the present invention. The animal behavior estimation system 1 includes a learning device 100, a first animal behavior estimation device 200, a second animal behavior estimation device 300, a camera 400, and a user device 500. The learning device 100, the first animal behavior estimation device 200, the second animal behavior estimation device 300, and the user device 500 are connected to the network 600. In the figure, the first animal behavior estimation device 200, the camera 400, and the user device 500 are shown one by one, but the number of the first animal behavior estimation device 200, the camera 400, and the user device 500 is arbitrary. For example, a business operator that provides a service for detecting abnormal behavior of an animal owns a learning device 100 and a second animal behavior estimation device 300, and a user of the service is a first animal behavior estimation device 200, a camera 400, and a user. It owns the device 500. The learning device 100 and the second animal behavior estimation device 300 may be physically the same device. Further, the first animal behavior estimation device 200 may also serve as the user device 500.

The learning device 100 learns various models used for estimating the behavior of the animal to be estimated. The first animal behavior estimation device 200 acquires the positions of the feature points in the time series of the estimation target animal from the moving image or the still image of the estimation target animal, and estimates whether or not the behavior of the estimation target animal is normal. do. The second animal behavior estimation device 300 estimates the type of behavior of the estimated target animal that the first animal behavior estimation device 200 estimates is not normal behavior.

The camera 400 is an imaging device that captures moving images or still images. The camera 400 may be, for example, a webcam, or a sensing camera such as a ToF (Time of Flight) camera or a depth camera. ToF cameras and depth cameras are sensing cameras that can measure the depth distance for each pixel. Further, the camera 400 may capture a moving image or a still image observed with visible light, or may capture a moving image or a still image observed with a wavelength other than visible light. The camera 400 outputs the moving image data of the estimated target animal to the first animal behavior estimation device 200. For example, in the case of livestock such as cattle, the camera 400 is installed in a barn or a grazing yard. The video data consists of time-series frames. Each frame is still image data. Information indicating the shooting time or shooting order is added to each frame included in the moving image data, or information capable of acquiring the shooting time or shooting order of each frame is added to the moving image data. The camera 400 may output still image data taken at different timings to the first animal behavior estimation device 200. In the following, the frame included in the moving image data and the still image data are collectively referred to as image data. Further, in the following, a case where the camera 400 captures moving image data will be described as an example. The user device 500 is, for example, a user's personal computer, a smartphone, a tablet terminal, or the like. The user device 500 displays the notification received from the second animal behavior estimation device 300 on the screen or outputs it by voice.

FIG. 2 is a block diagram showing the configuration of the learning device 100. The learning device 100 is realized by, for example, one or more computer devices. The learning device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a learning unit 150.

The communication unit 110 transmits / receives data to / from another device via the network 600. The input unit 120 is configured by using an existing input device such as a keyboard, a pointing device (mouse, tablet, etc.), a button, and a touch panel. The input unit 120 is operated by the user when inputting the user's instruction to the learning device 100. The input unit 120 may receive a user's instruction from a user's computer device (not shown) connected via a network. The display unit 130 is an image display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro Luminescence) display. The display unit 130 displays the data. The display unit 130 may transmit data to a computer device of a user (not shown) connected via a network and display the data.

The storage unit 140 includes a learning image storage unit 141, a learning data storage unit 142, a feature point detection model storage unit 143, a feature point position estimation model storage unit 144, and an action estimation model storage unit 145.

The learning image storage unit 141 stores data of a moving image or a still image for learning in which an animal is photographed. The learning data storage unit 142 stores the learning data. The learning data includes feature point learning data and behavior estimation learning data. The feature point learning data is data in which the image data is associated with the type and position of the feature points in the image data. The behavior estimation learning data is data in which the positions of each type of feature point in the time series are associated with the behavior of the correct animal. As feature points, sample points used to estimate animal behavior are used.

The feature point detection model storage unit 143 stores the feature point detection model. The feature point detection model is a model for detecting the position of a feature point of an animal included in the image data by inputting a feature amount obtained from the image data. As the feature amount, for example, a pixel value, a brightness, or the like can be used. The feature point position estimation model storage unit 144 stores the feature point position estimation model. The feature point detection model is a model for identifying the positions of other types of feature points by inputting the positions of each of some types of feature points among a plurality of types of predetermined feature points.

The behavior estimation model storage unit 145 stores the behavior estimation model. The behavior estimation model is a model for outputting the estimation result of the behavior of the animal by inputting the time series feature point position information. The time-series feature point position information indicates the time-series feature point position information. The feature point position information indicates the position of each feature point of each type. A tensor, JSON (JavaScript (registered trademark) Object Notation), a matrix, or the like can be used to describe the time-series feature point position information. The behavior estimation model of the present embodiment includes a normal behavior determination model and a behavior type estimation model. The normal behavior determination model inputs the time-series feature point position information and outputs the estimation result of whether or not the behavior is normal. The action type estimation model inputs the time-series feature point position information and outputs the estimation result of the action type.

The learning unit 150 includes a learning data generation unit 151, a feature point detection model learning unit 152, a feature point position estimation model learning unit 153, a behavior estimation model learning unit 154, and a distribution unit 155.

The learning data generation unit 151 adds feature point type and position information to the image data read from the learning image storage unit 141 according to the user's input using the input unit 120 to generate the feature point learning data. Write to the learning data storage unit 142. Further, the learning data generation unit 151 performs behavior estimation in which the time-series feature point position information and the correct behavior of the animal from which the time-series feature point position information is obtained are associated with each other according to the user's input using the input unit 120. The learning data is generated and written to the learning data storage unit 142.

The feature point detection model learning unit 152 learns a feature point detection model for each type of feature point by machine learning using the feature point learning data stored in the learning data storage unit 142, and the feature point detection model storage unit 152. Write to 143. The feature point position estimation model learning unit 153 learns the feature point position estimation model by machine learning using the feature point position information for each individual animal obtained from the feature point learning data stored in the learning data storage unit 142. It is written in the feature point position estimation model storage unit 144. The behavior estimation model learning unit 154 learns a normal behavior determination model and a behavior type estimation model by machine learning using the behavior estimation learning data stored in the learning data storage unit 142, and writes them in the behavior estimation model storage unit 145. The distribution unit 155 distributes the learned feature point detection model, feature point position estimation model, and normal behavior determination model to the first animal behavior estimation device 200, and distributes the learned behavior type estimation model to the second animal behavior estimation device 300. Deliver to.

As the learning device 100, an annotation tool of the prior art can be used. For example, a computer device equipped with an existing application for annotating human feature points may be used as the learning device 100.

FIG. 3 is a block diagram showing the configuration of the first animal behavior estimation device 200. The first animal behavior estimation device 200 is based on, for example, hardware such as an ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array), or a computer device such as a tablet terminal or a personal computer. It will be realized. The first animal behavior estimation device 200 includes a communication unit 210, a storage unit 220, an image input unit 230, a processing unit 240, and a display unit 250.

The communication unit 210 transmits / receives data to / from another device via the network 600. The storage unit 220 includes a feature point detection model storage unit 221, a feature point position estimation model storage unit 222, a normal behavior determination model storage unit 223, an image storage unit 224, and an analysis information storage unit 225. The feature point detection model storage unit 221 stores the feature point detection model learned by the learning device 100. The feature point position estimation model storage unit 222 stores the feature point position estimation model learned by the learning device 100. The normal behavior determination model storage unit 223 stores the normal behavior determination model learned by the learning device 100. The image storage unit 224 stores moving image data in which the estimated target animal is photographed. The analysis information storage unit 225 stores the time-series feature point position information of the target for which the animal behavior is estimated and the information for generating the time-series feature point position information.

The image input unit 230 inputs the moving image data captured by the camera 400 and writes it in the image storage unit 224. When the shooting time or shooting order is not added to the frame included in the moving image data, the image input unit 230 adds the shooting time or shooting order information calculated using the information set in the moving image data to the frame. .. The image input unit 230 may sequentially input data of still images taken from the camera 400 at different timings and write them to the image storage unit 224.

The processing unit 240 includes an update unit 241, a feature point detection unit 242, a feature point position estimation unit 243, a determination unit 244, an estimation request unit 245, and a status display processing unit 246.

The update unit 241 uses the feature point detection model, the feature point position estimation model, and the normal behavior determination model delivered from the learning device 100 as the feature point detection model storage unit 221 and the feature point position estimation model storage unit 222, and the normal behavior determination, respectively. Write to the model storage unit 223. The feature point detection model storage unit 221, the feature point position estimation model storage unit 222, and the normal behavior determination model storage unit 223 have the feature point detection model, the feature point position estimation model, and the normal behavior determination learned by the learning device 100, respectively. If the model is stored in advance and is not updated, the processing unit 240 may not include the updating unit 241.

The feature point detection unit 242 detects the feature points of the animal to be estimated in each image data by using the feature point detection model stored in the feature point detection model storage unit 221 and obtains the position of the feature points. The feature points detected by the feature point detection unit 242 are referred to as detection feature points. When a plurality of estimation target animals are photographed in the image data, the feature point detection unit 242 classifies the detection feature points by individual based on the type and position of the detection feature points.

The feature point position estimation unit 243 features each individual based on the position of the detection feature point detected by the feature point detection unit 242 and the feature point position estimation model stored in the feature point position estimation model storage unit 222. The position of the feature point of the type not detected by the point detection unit 242 is estimated. The feature point whose position is estimated by the feature point position estimation unit 243 is referred to as an estimated feature point. The feature point position estimation unit 243 adds the information on the position of the estimated feature point to the information on the position of the detected feature point detected by the feature point detection unit 242 to generate the feature point position information, and writes it in the analysis information storage unit 225. ..

The determination unit 244 is an example of a behavior estimation unit that estimates the behavior of an animal using a behavior estimation model. The determination unit 244 uses the normal behavior determination model stored in the normal behavior determination model storage unit 223 as the behavior estimation model. The determination unit 244 generates time-series feature point position information for each individual, in which the feature point position information generated for each of a predetermined number of time-series image data is arranged in chronological order. Based on the time-series feature point position information and the normal behavior judgment model, the judgment unit 244 determines whether the behavior of the estimation target animal is normal or whether the behavior of the estimation target animal is normal or abnormal for each individual. judge.

The estimation request unit 245 transmits the time-series feature point position information determined by the determination unit 244 to be abnormal or abnormal to the second animal behavior estimation device 300, and requests estimation of the type of behavior. The status display processing unit 246 displays the operating state of the first animal behavior estimation device 200, the information notified from the second animal behavior estimation device 300, and the like on the display unit 250. The information notified from the second animal behavior estimation device 300 includes information on the estimation target animal estimated by the second animal behavior estimation device 300 to be abnormal behavior. This information is, for example, data such as numerical values and texts indicating time-series feature point position information, estimated types of abnormal behavior, and the like.

The display unit 250 is an image display device such as a liquid crystal display, an organic EL display, or a CRT display. The display unit 250 displays the data.

FIG. 4 is a block diagram showing the configuration of the second animal behavior estimation device 300. The second animal behavior estimation device 300 is realized by, for example, one or more server computers. The second animal behavior estimation device 300 has a communication unit 310, a storage unit 320, and a processing unit 330.

The communication unit 310 transmits / receives data to / from another device via the network 600. The storage unit 320 includes an action type estimation model storage unit 321, a user information storage unit 322, and an analysis target information storage unit 323. The behavior type estimation model storage unit 321 stores the behavior type estimation model learned by the learning device 100. The user information storage unit 322 stores user information. The user information includes the identification information of the service user and the user address. The user address is the address of the user device 500 that is the output destination of the notification. An identifier such as the address of the first animal behavior estimation device 200 may be used as the identification information of the service user. The analysis target information storage unit 323 stores the time-series feature point position information received from the first animal behavior estimation device 200.

The processing unit 330 includes an update unit 331, an action type estimation unit 332, a notification unit 333, and an abnormality data transmission unit 334. The update unit 331 writes the action type estimation model delivered from the learning device 100 into the action type estimation model storage unit 321. If the behavior type estimation model learned by the learning device 100 is stored in advance in the behavior type estimation model storage unit 321 and is not updated, the processing unit 330 may not include the update unit 331. The behavior type estimation unit 332 is an example of a behavior estimation unit that estimates the behavior of an animal using a behavior estimation model. The behavior type estimation unit 332 uses the behavior type estimation model stored in the behavior type estimation model storage unit 321 as the behavior estimation model. That is, the behavior type estimation unit 332 determines the behavior type of the estimation target animal by using the time-series feature point position information received from the first animal behavior estimation device 200 and the behavior type estimation model. When the action type estimated by the action type estimation unit 332 is an abnormal action, the notification unit 333 notifies the user device 500 of the detection of the abnormal action and the estimated action type. When the type of action estimated by the action type estimation unit 332 is abnormal action, the abnormal data transmission unit 334 includes numerical values, texts, etc. including time-series feature point position information estimated to be abnormal action, the type of abnormal action, and the like. The data is transmitted to the first animal behavior estimation device 200.

FIG. 5 is a diagram showing sample points of cattle as an example among quadrupeds. In this embodiment, the crown, right ear, left ear, right eye, left eye, nose, mouth, middle of cow tongue, tip of cow tongue, right elbow head, left elbow head, right front knee, left front knee, right front connection, left front connection, right This is an example using 26 types of sample points: posterior knee, left posterior knee, right flight segment, left flight segment, right posterior connection, left posterior connection, back, ridge, tail base, middle tail, and tip of tail. The number of points does not have to be 26. The crown, right ear, left ear, right eye, left eye, nose, and mouth are facial features. In FIG. 5, the number added before the name of the type of sample point is an identifier that identifies the type of sample point.

In the case of humans, the position of the shoulder is an important sample point for expressing the posture, but in the case of cattle, the position of the shoulder is not important, so it is not used as the sample point. On the other hand, the back is one of the important sample points for cattle. The direction from the back to the top of the head tells you which direction the head is facing. The line of the back muscle connecting the back and the ridge indicates the direction of the cow. Furthermore, the direction of the limbs can be known from the lines connecting the back and the right olecranon and the left olecranon, which are the bases of the feet. In addition, as an abnormal behavior of cows, for example, there is tongue play. Such peculiar abnormal behavior can be detected by using the sample point of the tongue as a feature point. In addition, the sample point of the tail part is also a sample point that humans do not have.

FIG. 6 is an example of a screen displayed on the display unit 130 by the learning device 100 when the learning data is generated. A user having knowledge of the structure of the cow inputs the position of the sample point in the image of the cow and the position of the sample point for each cow by the input unit 120. In addition to inputting the sample points that are visible from the outside, the user can see the feature points that cannot be seen from the appearance, such as the sample points in the shaded part of the left and right parts, and the part that overlaps with other cows. Enter the sample points in the above and the positions of the sample points in the fuselage as much as possible from the image. It is not necessary to enter the positions of all types of sample points for each cow. The number of sample points differs depending on whether the cow is facing the front or the side, but enter the sample points as much as possible. By inputting the position of the sample point as accurately as possible, the accuracy of estimation is improved. In the case of humans, care must be taken such as considering the left-right balance for eye height, but the same accuracy is required for cows. As shown in FIG. 6, the learning data generation unit 151 connects the input predetermined types of sample points with a straight line for each individual and displays them as a skeleton. The area including one cow in the image is shown by a rectangle.

FIG. 7 is a diagram showing time-series feature point position information. As shown in the figure, the time-series feature point position information is information in which the time, the individual identification information (ID), and the position of each feature point are associated with each other. The individual identification information is information for identifying an animal to be estimated. In FIG. 7, the positions of the feature points are shown by the coordinates on the two-dimensional plane. The position information may be a relative position, an absolute position, or polar coordinates. In the case of a relative position, the position is represented by a coordinate value in xy coordinates with a predetermined pixel of the image as the origin. For example, when the camera 400 generates video data or image data including information on the depth of the image, the position of the feature point may be represented by a coordinate value on three-dimensional coordinates.

FIG. 8 is a diagram showing feature points detected and estimated by the first animal behavior estimation device 200. As shown in FIG. 8, the posture of the cow can be seen from the line connecting the feature points for each individual cow. In particular, the posture and orientation of the cow can be easily understood by connecting the sample points on the back and ridge to other sample points such as the crown and limbs. FIG. 9 is a diagram in which the feature points detected and estimated by the first animal behavior estimation device 200 are arranged in chronological order. In FIG. 9, the posture is shown by connecting the detected feature points for each individual. By arranging those postures in chronological order, the movement of each individual can be understood.

Subsequently, the operation of the animal behavior estimation system 1 will be described.
FIG. 10 is a processing flow showing the learning process in the learning device 100. First, the learning data generation unit 151 of the learning device 100 generates feature point learning data (step S110). The learning data generation unit 151 reads out image data to be used as learning data from the learning image storage unit 141 according to an instruction input by the user by the input unit 120, and displays the image data on the display unit 130. The user inputs the type and position of the feature point on an individual basis by the input unit 120. The learning data generation unit 151 adds the feature point position information indicating the type and position information of each cow's feature point input by the user to the image data to generate the feature point learning data, and causes the learning data storage unit 142 to generate the feature point learning data. Write.

Subsequently, the learning data generation unit 151 generates behavior estimation learning data (step S120). The learning data generation unit 151 acquires the feature point position information of the individual designated by the input unit 120 from each of the time-series feature point learning data designated by the user by the input unit 120. The learning data generation unit 151 corresponds to the time-series feature point position information in which the time-series feature point position information obtained from each feature point learning data is set and the correct behavior of the animal input by the user through the input unit 120. Generate the attached behavior estimation learning data. In the case of behavior estimation learning data (hereinafter referred to as the first behavior estimation learning data) used for learning the normal behavior judgment model, whether the correct behavior is normal behavior or normal behavior and abnormal behavior. In the case of the behavior estimation learning data (hereinafter referred to as the second behavior estimation learning data) used in the behavior type estimation model, the correct action is the behavior type. Types of behavior include, for example, abnormal behavior such as cow injury, cow fighting, cow abuse, cow illness, cow estrus, cow birth, cow fall, cow tongue play, and walking. , Eating food, etc. The learning data generation unit 151 writes the generated behavior estimation learning data into the learning data storage unit 142.

When the learning start instruction of the feature point detection model is input by the input unit 120, the feature point detection model learning unit 152 reads the feature point learning data from the learning data storage unit 142 and uses the image data included in the feature point learning data. The feature point detection model is machine-learned using the feature amount of the obtained image and the feature point position information added to the image data. As the feature point detection model, any conventional technique for detecting feature points such as human eyes and joints from image data can be used. For example, the feature point detection model is CNN (Convolutional neural network). The feature point detection model learning unit 152 writes the machine-learned feature point detection model into the feature point detection model storage unit 143 (step S130).

When the learning start instruction of the feature point position estimation model is input by the input unit 120, the feature point position estimation model learning unit 153 reads out the feature point position information added to the feature point learning data from the learning data storage unit 142. , Machine learning of the feature point position estimation model is performed using a plurality of feature point position information for each individual. The feature point position estimation model learning unit 153 writes the learned feature point position estimation model in the feature point position estimation model storage unit 144 (step S140). For learning the feature point position estimation model, feature point position information including both the position of the feature point visible in the image data and the position of the feature point not visible in the image data may be used. The feature point position estimation model may be trained using only the positions of the feature points that are visible in the image data. When learning the feature point position estimation model using only the positions of the feature points that are visible in the image data, the user determines in step S110 that the feature points are missing or on the background in addition to the type and position of the feature points. Enter information on whether or not it is a difficult feature point.

When the learning start instruction of the normal behavior determination model is input by the input unit 120, the behavior estimation model learning unit 154 reads out the first behavior estimation learning data from the learning data storage unit 142. The behavior estimation model learning unit 154 machine-learns a normal behavior determination model using the read first behavior estimation learning data (step S150). As a normal behavior determination model, for example, SVM (support vector machine) is used. The behavior estimation model learning unit 154 writes the learned normal behavior determination model in the behavior estimation model storage unit 145.

When the learning start instruction of the behavior type estimation model is input by the input unit 120, the behavior estimation model learning unit 154 reads out the second behavior estimation learning data from the learning data storage unit 142. The behavior estimation model learning unit 154 machine-learns the behavior type estimation model using the read second behavior estimation learning data (step S160). For example, the behavior estimation model learning unit 154 creates a behavior type estimation model using SVM for each behavior type. Alternatively, the behavior estimation model learning unit 154 has an input node that inputs time-series feature point position information, and an output node that corresponds to each type of behavior and outputs a value that quantitatively indicates that the behavior is that type. DNN (Deep Neural Network) or the like having the above may be learned as an action type estimation model. The behavior estimation model learning unit 154 writes the learned behavior type estimation model in the behavior estimation model storage unit 145.

The distribution unit 155 distributes the feature point detection model, the feature point position estimation model, and the normal behavior determination model instructed by the input unit 120 to the first animal behavior estimation device 200, and the distribution unit 120 instructs the distribution. The behavior type estimation model is delivered to the second animal behavior estimation device 300 (step S170). The update unit 241 of the first animal behavior estimation device 200 displays the received feature point detection model in the feature point detection model storage unit 221 and the feature point position estimation model in the feature point position estimation model storage unit 222, and displays the normal behavior determination model in the feature point position estimation model storage unit 222. Write to the normal behavior determination model storage unit 223. The update unit 331 of the second animal behavior estimation device 300 writes the behavior type estimation model in the behavior type estimation model storage unit 321.

In addition, the learning device 100 may execute the processing in which the processing order of each step shown in FIG. 10 is possible by exchanging the possible ones. Various models can be updated by repeating the learning process shown in FIG.

FIG. 11 is a flow chart showing the animal behavior estimation process in the animal behavior estimation system 1. The image input unit 230 of the first animal behavior estimation device 200 inputs video data of the behavior estimation target captured by the camera 400 in real time, or sequentially inputs data of still images captured at different timings, and is an image storage unit. I am writing to 224. The first animal behavior estimation device 200 activates the process of FIG. 11 at a predetermined cycle.

The feature point detection unit 242 reads a plurality of image data whose shooting time is between the current time and a time retroactive by a predetermined time from the current time from the image storage unit 224. The feature point detection unit 242 detects each feature point of the estimation target animal in the image data by using the feature amount obtained from the image data and the feature point detection model of each feature point for each read image data. .. The detected feature points may include feature points that cannot be visually recognized from the appearance in addition to the feature points that can be visually recognized from the appearance of the estimated target animal in the image data. The feature point detection unit 242 classifies the detected feature points into a set of detected feature points for each individual based on the detected feature points, that is, the type and position of the detected feature points. For this classification, any conventional technique for obtaining individual feature points from image data taken by a plurality of people can be used. For example, if there are detection feature points of the same type, the feature point detection unit 242 determines that they are feature points of different individuals. Further, the feature point detection unit 242 is within a predetermined range in which the direction or distance from one type of detection feature point to one or a plurality of other types of detection feature points is determined to be one individual feature point. Based on whether or not, it is determined whether or not these detected feature points are the feature points of the same individual. By combining these determinations, the feature point detection unit 242 classifies the detection feature points for each individual. The feature point detection unit 242 generates feature point position information in which the shooting time of the image data is associated with the type and position information of each detected feature point for each individual, and corresponds to the image data or the identification information of the image data. It is attached and written in the analysis information storage unit 225 (step S210).

Further, the feature point detection unit 242 may analyze the image data to obtain identification information of each individual. For example, when an ear tag displaying an individual identification number is attached to the right ear and the left ear of each individual, the individual identification number is stored in the storage unit 220 in advance. The feature point detection unit 242 recognizes characters in a region near the detection feature points of the right ear and the left ear of the image data. When the recognized character matches the individual identification number, the feature point detection unit 242 adds the individual identification number to the feature point position information in which the position of the detected feature point is set.

The feature point position estimation unit 243 is detected by the feature point detection unit 242 for each individual of the image data by using the position of each detection feature point set in the feature point position information and the feature point position estimation model. Calculate the position of the estimated feature point, which is the type of feature point that did not exist. The feature point position estimation unit 243 adds the type and position of each estimated feature point to the feature point position information (step S220).

The determination unit 244 generates time-series feature point position information for each individual, in which the feature point position information associated with the image data read in step S210 is arranged in chronological order (step S230). The determination unit 244 determines for each individual whether or not the behavior is normal by using the time-series feature point position information and the normal behavior determination model (step S240). The determination unit 244 may determine whether the behavior is normal or abnormal. The estimation request unit 245 adds user identification and camera identification information to the time-series feature point position information determined by the determination unit 244 to be abnormal or abnormal, and transmits the user identification and camera identification information to the second animal behavior estimation device 300 to request estimation. (Step S250). The camera identification information is information for identifying the camera 400 that captured the image data from which the time-series feature point position information was obtained.

The processing unit 330 of the second animal behavior estimation device 300 receives the time-series feature point position information from the first animal behavior estimation device 200 and stores it in the analysis target information storage unit 323 (step S310). The action type estimation unit 332 estimates the action type using the time-series feature point position information and the action type estimation model (step S320). When the action type estimation unit 332 determines that the estimated action type is normal or not abnormal (step S330: NO), the action type estimation unit 332 ends the process.

When the action type estimation unit 332 determines that the estimated action type is an abnormal action (step S330: YES), the user identification, individual identification information, and camera read from the time-series feature point position information to the notification unit 333. The identification information and the type of estimated abnormal behavior are output. The notification unit 333 transmits the alarm information in which the individual identification information and the camera identification information and the information of the estimated abnormal behavior type are set to the user address read in response to the user identification (step S340). The user device 500 displays the alarm information received from the notification unit 333 on the screen or outputs it by voice. The user selects the type of abnormal behavior to be notified, and registers the notification target abnormal behavior information indicating the selected abnormal behavior type in the user information storage unit 322 in association with the user identification. May be good. The notification unit 333 transmits alarm information when the estimated type of abnormal behavior is set as the notification target in the notification target abnormal behavior information read in response to the user identification.

The abnormal data transmission unit 334 describes the time-series feature point position information and the type of abnormal behavior in numerical values or text in the first animal behavior estimation device 200 that is the source of the time-series feature point position information determined to be abnormal behavior. The abnormal data information is transmitted (step 350). The status display processing unit 246 of the first animal behavior estimation device 200 displays the abnormal data information transmitted from the second animal behavior estimation device 300 on the display unit 250 (step S260).

According to the above embodiment, by using the sample points that cannot be detected from the image in addition to the sample points detected from the image, it is possible to estimate the animal behavior with high accuracy. Further, since the second animal behavior estimation device 300 can extract only the abnormal behavior of the animal with a small amount of information, the load is light. Therefore, it is possible to provide a service for many users to extract abnormal behavior of animals and notify the users. Further, since the amount of data transmitted from the first animal behavior estimation device 200 to the second animal behavior estimation device 300 can be reduced, the communication cost of the service user can be reduced. Further, since the animal behavior estimation performed by the first animal behavior estimation device 200 is a determination of whether or not it is normal, the processing capacity required for the first animal behavior estimation device 200 can be reduced.

The animal behavior estimation system 1 does not have to use the feature point position estimation model. In this case, the learning device 100 does not have to include the feature point position estimation model storage unit 144 and the feature point position estimation model learning unit 153, and the first animal behavior estimation device 200 includes the feature point position estimation model storage unit 222 and the feature point position estimation model storage unit 222. The feature point position estimation unit 243 may not be provided. The learning device 100 does not perform the process of step S140 of FIG. 10, and the distribution unit 155 does not distribute the feature point position estimation model in step S170. Further, the first animal behavior estimation device 200 does not perform the process of step S220 of FIG. 11, and the determination unit 244 generates time-series feature point position information that does not include the type and position of the estimated feature point in step S230. do.

Further, the first animal behavior estimation device 200 may be configured not to have the normal behavior determination model storage unit 223 and the determination unit 244. In this case, the first animal behavior estimation device 200 does not perform the process of step S240 in FIG. In step S250 of FIG. 11, the estimation request unit 245 makes an estimation request to the second animal behavior estimation device 300 for all the time-series feature point position information. The learning device 100 does not have to learn the normal behavior determination model.

Further, the first animal behavior estimation device 200 and the second animal behavior estimation device 300 may be integrated as shown in FIG. 12 as an animal behavior estimation device. FIG. 12 is a block diagram showing the configuration of the animal behavior estimation device 700. In the figure, the same parts as those of the first animal behavior estimation device 200 shown in FIG. 3 and the second animal behavior estimation device 300 shown in FIG. 4 are designated by the same reference numerals, and the description thereof will be omitted.

The animal behavior estimation device 700 includes a communication unit 210, a storage unit 720, an image input unit 230, a processing unit 740, and a display unit 250. The storage unit 720 includes a feature point detection model storage unit 221, a feature point position estimation model storage unit 222, a normal behavior determination model storage unit 223, an image storage unit 224, an analysis information storage unit 225, and an action type estimation model. It includes a storage unit 321, a user information storage unit 322, and an analysis target information storage unit 323. The processing unit 740 includes an update unit 741, a feature point detection unit 242, a feature point position estimation unit 243, a determination unit 244, an estimation request unit 745, an action type estimation unit 332, a notification unit 333, and a status display. It is provided with a processing unit 746. The update unit 741 uses the feature point detection model, the feature point position estimation model, the normal behavior determination model, and the behavior type estimation model delivered from the learning device 100 in step S170 of FIG. 10, respectively, in the feature point detection model storage unit 221 and the feature. Write to the point position estimation model storage unit 222, the normal action determination model storage unit 223, and the action type estimation model storage unit 321. In step S250 of FIG. 11, the estimation request unit 745 outputs the time-series feature point position information determined by the determination unit 244 to be abnormal or abnormal to the action type estimation unit 332, and requests the action estimation. As a result, the type of action is estimated only for the time-series feature point position information determined to be abnormal or abnormal, so that the load on the animal behavior estimation device 700 can be reduced. The status display processing unit 746 displays the operating status of the animal behavior estimation device 700. In addition, the status display processing unit 746 provides abnormal data information in which the time-series feature point position information determined by the action type estimation unit 332 to be abnormal behavior and the type of abnormal behavior are described in numerical values or text in step S350 of FIG. It is displayed on the display unit 250.

The animal behavior estimation device 700 may not have the feature point position estimation model storage unit 222 and the feature point position estimation unit 243. The animal behavior estimation device 700 does not perform the process of step S220 of FIG. 11, and the determination unit 244 generates time-series feature point position information that does not include the type and position of the estimated feature points in step S230.

The animal behavior estimation device 700 may not have the normal behavior determination model storage unit 223 and the determination unit 244. In this case, the animal behavior estimation device 700 does not perform the process of step S240 in FIG. In step S250 of FIG. 11, the estimation request unit 745 outputs all the time-series feature point position information to the action type estimation unit 332 and requests the action estimation.

Further, the animal behavior estimation device 700 may not have the estimation request unit 745, the behavior type estimation unit 332, and the behavior type estimation model storage unit 321. In this case, the animal behavior estimation device 700 does not perform the processing of steps S250 and S310 to S340 in FIG. After the processing of step S240, the notification unit 333 displays the individual identification of the estimation target animal determined by the determination unit 244 to be abnormal or abnormal on a display unit (not shown) provided in the animal behavior estimation device 700, or the user. Notify device 500.

Further, as shown in FIG. 13, one device may be used to learn the model and estimate the behavior type of the target estimation animal. FIG. 13 is a block diagram showing the configuration of the animal behavior estimation device 800. The animal behavior estimation device 800 estimates the type of behavior for all time-series feature point position information. In FIG. 13, the same parts as the learning device 100 shown in FIG. 2, the first animal behavior estimation device 200 shown in FIG. 3, and the second animal behavior estimation device 300 shown in FIG. 4 are designated by the same reference numerals, and the description thereof will be described. Is omitted.

The animal behavior estimation device 800 includes an input unit 120, a display unit 130, a storage unit 840, a learning unit 850, an image input unit 230, and an estimation processing unit 860. The storage unit 840 includes a learning image storage unit 141, a learning data storage unit 142, a feature point detection model storage unit 143, a feature point position estimation model storage unit 144, an action estimation model storage unit 145, and an image storage unit. It includes a 224, an analysis information storage unit 225, and a user information storage unit 322. The difference between the learning unit 850 and the learning unit 150 shown in FIG. 2 is that the learning unit 850 does not include the distribution unit 155. The behavior estimation model learning unit 154 learns the behavior type estimation model as the behavior estimation model.

The estimation processing unit 860 includes a feature point detection unit 242, a feature point position estimation unit 243, an action type estimation unit 332, a notification unit 861, and a status display processing unit 862. The feature point detection unit 242 uses the feature point detection model stored in the feature point detection model storage unit 143 for detecting the feature points. The feature point position estimation unit 243 uses the feature point position estimation model stored in the feature point position estimation model storage unit 144 for estimating the position of the feature point. The action type estimation unit 332 generates time-series feature point position information, and estimates the action type using the action type estimation model stored in the action estimation model storage unit 145. The notification unit 861 may notify the user device 500 of the type of action estimated by the action type estimation unit 332, or may display it on the display unit 130. When the image input unit 230 receives the moving image data or the image data of the still image to which the user identification is given from the camera 400 of each user, the notification unit 861 uses the usage specified by the moving image data or the still image data. Determine the notification destination based on the person identification. The status display processing unit 862 describes the operating state of the animal behavior estimation device 800, the time-series feature point position information estimated to be abnormal by the behavior type estimation unit 332, and the type of abnormal behavior in numerical values or text. The abnormal data information is displayed on the display unit 250.

With the above configuration, the learning unit 850 of the animal behavior estimation device 800 performs the processing of steps S110 to S160 of the learning process shown in FIG. The animal behavior estimation device 800 performs the animal behavior estimation process as follows.

FIG. 14 is a flow chart showing the animal behavior estimation process in the animal behavior estimation device 800. The image input unit 230 of the animal behavior estimation device 800 inputs the moving image data of the behavior estimation target captured by the camera 400 in real time, or sequentially inputs the still image data captured at different timings to the image storage unit 224. I'm writing. The image input unit 230 of the animal behavior estimation device 800 may input moving image data obtained by photographing the behavior estimation target from each of the cameras 400 of a plurality of users via the network 600 and write it in the image storage unit 224. In this case, camera identification information and user identification are set in the moving image data. The camera identification information may be used as user identification. The animal behavior estimation device 800 activates the process of FIG. 14 at a predetermined cycle.

The processing of steps S810 and S820 is the same as the processing of steps S210 and S220 shown in FIG. That is, the feature point detection unit 242 reads out a plurality of image data whose shooting time is between the current time and a time retroactive by a predetermined time from the current time from the image storage unit 224. The feature point detection unit 242 detects each feature point of the estimation target animal in the image data by using the feature amount obtained from the image data and the feature point detection model of each feature point for each read image data. .. The feature point detection unit 242 generates feature point position information in which the shooting time of the image data is associated with the type and position information of each detected feature point for each individual, and corresponds to the image data or the identification information of the image data. It is attached and written in the analysis information storage unit 225 (step S810). Further, the feature point detection unit 242 adds the individual identification information obtained by analyzing the image data to the feature point position information. The feature point position estimation unit 243 calculates the position of the estimated feature point for each individual of the image data by using the position of each detected feature point set in the feature point position information and the feature point position estimation model. .. The feature point position estimation unit 243 adds the type and position of each estimated feature point to the feature point position information (step S820).

The action type estimation unit 332 generates time-series feature point position information for each individual, in which the feature point position information associated with the image data read in step S810 is arranged in chronological order (step S830). The action type estimation unit 332 estimates the action type using the time-series feature point position information and the action type estimation model (step S840). When the action type estimation unit 332 determines that the estimated action type is normal or not abnormal (step S850: NO), the action type estimation unit 332 ends the process.

When the action type estimation unit 332 determines that the estimated action type is an abnormal action (step S850: YES), the user identification, individual identification information, and camera read from the time-series feature point position information in the notification unit 861. The identification information and the type of estimated abnormal behavior are output. The notification unit 861 reads out the user address from the user information storage unit 322 in response to the user identification added to the image data from which the time-series feature point position information is obtained. The notification unit 861 transmits the alarm information set with the individual identification information and the camera identification information and the information of the type of the estimated abnormal behavior to the user address (step S860). The user device 500 displays the alarm information received from the notification unit 333 on the screen or outputs it by voice. When the user owns the animal behavior estimation device 800, the notification unit 861 may display the alarm information on the display unit 130. The status display processing unit 862 displays the abnormal data information in which the time-series feature point position information determined by the action type estimation unit 332 as the abnormal behavior and the type of the abnormal behavior are described in numerical values or texts on the display unit 250 (step S870). ). The process of step S870 may also serve as a process of displaying alarm information on the display unit 130 in step S860.

The estimation processing unit 860 of the animal behavior estimation device 800 may further include a determination unit 244 and an estimation request unit 245. In this case, the storage unit 840 further includes a normal behavior determination model storage unit 223. Then, the behavior estimation model learning unit 154 further learns the normal behavior determination model and writes it in the behavior estimation model learning unit 154. The determination unit 244 determines whether or not the behavior of the estimated target animal is normal, or whether or not the behavior is normal or abnormal, using the normal behavior determination model stored in the behavior estimation model learning unit 154. The estimation request unit 245 requests the behavior type estimation unit 332 to estimate the behavior type only for the time-series feature point position information of the estimation target animal determined by the determination unit 244 to be abnormal or abnormal.

The estimation processing unit 860 of the animal behavior estimation device 800 may include a determination unit 244 instead of the behavior type estimation unit 332. In this case, the behavior estimation model learning unit 154 learns the normal behavior determination model as the behavior estimation model and writes it in the behavior estimation model learning unit 154. The determination unit 244 determines whether or not the behavior of the estimated target animal is normal, or whether or not the behavior is normal or abnormal, using the normal behavior determination model stored in the behavior estimation model learning unit 154. The notification unit 861 displays the individual identification information of the estimated target animal determined by the determination unit 244 as abnormal or abnormal and the alarm information set with the camera identification information on the display unit 130 provided in the animal behavior estimation device 800, or , Notify the user device 500.

The animal behavior estimation device 800 may not include the feature point position estimation model storage unit 144, the feature point position estimation model learning unit 153, and the feature point position estimation unit 243. The learning unit 850 does not perform the process of step S140 in FIG. 10, and the action type estimation unit 332 generates time-series feature point position information that does not include the type and position of the estimated feature points in step S830.

The learning device 100, the first animal behavior estimation device 200, the second animal behavior estimation device 300, the animal behavior estimation device 700, and the animal behavior estimation device 800 described above are a CPU (Central Processing Unit) and a GPU (Graphics) connected by a bus. It is equipped with a processor such as a Processing Unit), a memory, an auxiliary storage device, etc., and realizes the above functions by executing a program by the processor. All or part of the functions of the learning device 100, the first animal behavior estimation device 200, the second animal behavior estimation device 300, the animal behavior estimation device 700, and the animal behavior estimation device 800 may be made of ASIC, PLD, FPGA, or the like. It may be realized by using hardware. The program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a flexible disk, a magneto-optical disk, a portable medium such as a ROM, a CD-ROM, a USB memory, an SSD, or a storage device such as a hard disk built in a computer system. The program may be transmitted over a telecommunication line.

Further, the learning device 100, the first animal behavior estimation device 200, the second animal behavior estimation device 300, the animal behavior estimation device 700, and the animal behavior estimation device 800 may be realized by a plurality of computer devices, respectively. In this case, each functional unit of the learning device 100, the first animal behavior estimation device 200, the second animal behavior estimation device 300, the animal behavior estimation device 700, and the animal behavior estimation device 800 is realized by any of these plurality of computer devices. Can be arbitrary. Further, the same functional unit may be realized by a plurality of computer devices. For example, the storage unit 140 of the learning device 100, the storage unit 720 of the animal behavior estimation device 700, and the storage unit 840 of the animal behavior estimation device 800 may be realized by a computer device different from other functional units, and the second animal. The behavior type estimation unit 332 of the behavior estimation device 300, the animal behavior estimation device 700, and the animal behavior estimation device 800 may be realized by a plurality of computer devices to perform distributed processing.

According to the embodiment described above, the animal behavior estimation device includes a feature point detection unit and a behavior estimation unit. The feature point detection unit is a feature point that can be visually recognized from the appearance of the estimation target animal in the image data and a feature that cannot be visually recognized from the appearance of the estimation target animal in the image data from the image data obtained by the estimation target animal that is the behavior estimation target animal. Detect points. The image data may be either frame data or still image data included in the moving image data. Feature points are used to estimate the behavior of an animal, and if the animal is a quadruped, the types of feature points include face, tongue, dorsal, tail and joints. The behavior estimation unit estimates the behavior of the animal to be estimated based on the time-series feature point position information indicating the position of the feature point detected by the feature point detection unit for each of the time-series image data.

In addition, the animal behavior estimation device may further include a feature point position estimation unit. The feature point position estimation unit is based on the position of the feature point detected by the feature point detection unit, and is a feature point of a type not detected by the feature point detection unit among a plurality of types of feature points used for estimating the behavior of the animal. Estimate the position of. The behavior estimation unit is an estimation target animal based on the time-series feature point position information indicating the position of the feature point detected by the feature point detection unit and the position of the feature point estimated by the feature point position estimation unit for each time-series image data. Estimate the behavior of.

When a plurality of estimation target animals are photographed in the image data, the feature point detection unit determines the estimation target animal to which the feature points detected from the image data belong based on the type and position of the detected feature points. do.

The behavior estimation unit uses machine-learned behavior estimation models and time-series feature point position information as learning data for the position information of multiple types of feature points in the time series and the behavior of the animal to be estimated. May be estimated. The correct behavior information is information indicating the type of behavior, information indicating whether or not the behavior is normal, or information indicating whether the behavior is normal or abnormal.

The feature point position estimation unit is a feature point position estimation model machine-learned using the information of the type and position of the feature point specified by the user in the image data captured by the animal as learning data, and the feature point detection unit detected by the feature point detection unit. The position of a feature point of a type not detected by the feature point detection unit may be estimated based on the position of.

The feature point detection unit uses a feature point detection model machine-learned using the image data obtained by the animal and the type and position of the feature points specified by the user in the image data as learning data, and features from the image data. Detect points.

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs and the like within a range that does not deviate from the gist of the present invention.

1 ... Animal behavior estimation system,
100 ... Learning device,
110, 210, 310 ... Communication unit,
120 ... Input section,
130, 250 ... Display,
140, 220, 320, 720, 840 ... Storage unit,
141 ... Learning image storage unit,
142 ... Learning data storage,
143, 221 ... Feature point detection model storage unit,
144, 222 ... Feature point position estimation model storage unit,
145 ... Behavior estimation model memory unit,
150, 850 ... Learning Department,
151 ... Learning data generator,
152 ... Feature point detection model learning unit,
153 ... Feature point position estimation model learning unit,
154 ... Behavior estimation model learning department,
155 ... Distribution Department,
200 ... First animal behavior estimation device,
220 ... Memory unit,
223 ... Normal behavior judgment model storage unit,
224 ... Image storage unit,
225 ... Analytical information storage unit,
230 ... Image input unit,
240, 330, 740 ... Processing unit,
241, 331, 741 ... Update section,
242 ... Feature point detector,
243 ... Feature point position estimation unit,
244 ... Judgment unit,
245, 745 ... Estimate request unit,
246, 746, 862 ... Status display processing unit,
300 ... Second animal behavior estimation device,
320 ... Memory unit,
321 ... Behavior type estimation model storage unit,
322 ... User information storage unit,
323 ... Analysis target information storage unit,
332 ... Behavior type estimation unit,
333, 861 ... Notification section,
334 ... Abnormal data transmitter,
400 ... camera,
500 ... User device,
600 ... network,
700, 800 ... Animal behavior estimator,
860 ... Estimate processing unit

Claims (10)

From the image data obtained by the estimation target animal, which is the behavior estimation target animal, the feature points that can be visually recognized from the appearance of the estimation target animal in the image data and the feature points that cannot be visually recognized from the appearance of the estimation target animal in the image data. Feature point detector to detect and
A behavior estimation unit that estimates the behavior of the animal to be estimated based on time-series feature point position information indicating the positions of a plurality of types of the feature points detected by the feature point detection unit for each of the time-series image data.
Equipped with a,
The animal is a cow
The types of the feature points are face, multiple tongues, multiple tails, back, right anterior connection, left anterior connection, left posterior connection, left posterior connection, right anterior knee, left anterior knee, right posterior knee, left posterior knee, right. Including hock and left knee,
Animal behavior estimation device. Based on the position of the feature point detected by the feature point detection unit, the position of the type of feature point not detected by the feature point detection unit is estimated among the plurality of types of feature points used for estimating the behavior of the animal. Further equipped with a feature point position estimation unit
The behavior estimation unit indicates the position of the feature point detected by the feature point detection unit and the position of the feature point estimated by the feature point position estimation unit for each of the image data in the time series. Estimate the behavior of the estimated target animal based on
The animal behavior estimation device according to claim 1. When a plurality of estimation target animals are photographed in the image data, the feature point detection unit performs the estimation to which the feature points detected from the image data belong based on the type and position of the detected feature points. Determine the target animal,
The animal behavior estimation device according to claim 1 or 2. The behavior estimation unit uses the information on the positions of a plurality of types of feature points in the time series and the information on the correct behavior as learning data, and the estimation target is based on the behavior estimation model machine-learned and the time-series feature point position information. Estimate animal behavior,
The animal behavior estimation device according to any one of claims 1 to 3. The correct behavior information is information indicating the type of behavior, information indicating whether it is normal or not, or information indicating whether it is normal or abnormal.
The animal behavior estimation device according to claim 4. The feature point detection unit uses the feature point detection model machine-learned using the image data obtained by the animal and the type and position of the feature points specified by the user in the image data as learning data. Detect feature points from
The animal behavior estimation device according to any one of claims 1 to 5. The behavior estimated by the behavior estimation unit is one or more of cow injury, cow fight, cow abuse, cow illness, cow estrus, cow birth, cow fall, and cow tongue play. Including abnormal behavior,
The animal behavior estimation device according to any one of claims 1 to 4. Further comprising a notification portion that the action is estimated behavior by the estimation unit outputs a notification when it is the abnormal behavior,
The animal behavior estimation device according to claim 7. From the image data obtained by the estimation target animal, which is the behavior estimation target animal, the feature points that can be visually recognized from the appearance of the estimation target animal in the image data and the feature points that cannot be visually recognized from the appearance of the estimation target animal in the image data. Feature point detection step to be detected, estimation step,
A behavior estimation step that estimates the behavior of the animal to be estimated based on the time-series feature point position information indicating the position of the feature point detected in the feature point detection step for each of the time-series image data.
Have a,
The animal is a cow
The types of the feature points are face, multiple tongues, multiple tails, back, right anterior connection, left anterior connection, left posterior connection, left posterior connection, right anterior knee, left anterior knee, right posterior knee, left posterior knee, right. Including hock and left knee,
Animal behavior estimation method. Computer,
The program for functioning as the animal behavior estimation device according to any one of claims 1 to 8.

Images