JP7603483B2 - Control system, learning system, device control method and program - Google Patents

Description

This disclosure relates to a control system, a learning system, an equipment control method, and a program.

An operation determination device has been proposed that includes an indirect operation unit that generates indirect operation data based on the trajectory of the electronic key when an operator performs an indirect operation to operate an operation target while carrying the electronic key, and a control unit that determines that an indirect operation has been performed if the indirect operation data matches within an allowable range with teacher data in which a graph of the trajectory of the electronic key is made into a template. Here, if the control unit determines that an indirect operation has not been performed because the indirect operation data deviates from the allowable range of the teacher data, but a direct operation that can give the same instruction as the indirect operation is performed within a preset period, it determines that the operator has performed the operation again and learns so that the indirect operation can be performed using the indirect data, and updates the teacher data.

JP 2019-197450 A

However, in an operation determination device that determines the operation content corresponding to an indirect operation, i.e., a gesture operation, as described in Patent Document 1, there is a demand for improving the recognition accuracy of the gesture operation so that the operation target can be appropriately operated.

This disclosure has been made in consideration of the above-mentioned reasons, and aims to provide a control system, a learning system, a device control method, and a program that can appropriately control a device through gesture operations.

In order to achieve the above object, the control system according to the present disclosure comprises:
a neural network storage unit that stores a neural network for identifying one type of operation content corresponding to a gesture image of the same type from gesture image information showing a gesture of the same type of user corresponding to each of a plurality of devices;
an imaging device provided in each of the plurality of devices and configured to image a user of each of the plurality of devices;
an image acquisition unit that acquires, from the imaging device, gesture image information indicating a gesture of the user captured by the imaging device;
a selection unit that selects, from the neural networks stored in the neural network storage unit, a neural network corresponding to an apparatus provided with an imaging device that has generated the gesture image information acquired by the image acquisition unit;
a neural network calculation unit that calculates a judgment value corresponding to the gesture image information acquired by the image acquisition unit, using the neural network selected by the selection unit;
an operation identification unit that identifies operation identification information that identifies an operation on the device associated with the determination value calculated by the neural network calculation unit;
a device control unit that controls a device provided with an imaging device that generates the gesture image information, based on the operation identification information identified by the operation identification unit;
The neural network update unit updates the neural network selected by the selection unit based on the gesture image information and operation identification information corresponding to a judgment value calculated from the image information each time the image acquisition unit acquires the gesture image information, and identifies, each time a preset neural network update time arrives, a neural network requiring the least amount of calculations from among the neural networks corresponding to each of the plurality of devices stored in the neural network storage unit, and updates all of the neural networks corresponding to each of the plurality of devices to the identified neural network.

According to the present disclosure, the neural network update unit updates the neural network selected by the selection unit based on the gesture image information and the operation identification information corresponding to the judgment value calculated from the image information, each time the image acquisition unit acquires gesture image information. In addition, the neural network update unit identifies the neural network with the least amount of calculations among the neural networks corresponding to each of the multiple devices stored in the neural network storage unit each time a preset neural network update time arrives, and updates all of the neural networks corresponding to each of the multiple devices to the identified neural network. As a result, the neural networks corresponding to each of the multiple devices are periodically updated to the neural network with the least amount of calculations, i.e., the most computationally efficient, thereby shortening the time required to identify the operation content corresponding to the gesture performed by the user and improving the accuracy of identifying the operation content.

1 is a schematic configuration diagram of a control system according to an embodiment of the present disclosure. FIG. 1 is a block diagram showing a hardware configuration of a control system according to an embodiment. A block diagram showing the functional configuration of an air conditioner according to an embodiment. FIG. 1A is a diagram showing an example of an image showing a gesture of a user; FIG. 1B is a diagram showing an example of an image showing a gesture of another user; FIG. 1 is a block diagram showing a functional configuration of a cloud server according to an embodiment. FIG. 1 is an explanatory diagram of the operation of a control system according to an embodiment of the present invention; FIG. 1 is an explanatory diagram of the operation of a control system according to an embodiment of the present invention; FIG. 1 is a diagram showing an example of information stored in an operation identification information storage unit according to an embodiment; FIG. 1 is a sequence diagram showing an operation of a control system according to an embodiment. FIG. 1 is a sequence diagram showing an operation of a control system according to an embodiment. FIG. 1 is a sequence diagram showing an operation of a control system according to an embodiment. A flowchart showing an example of the flow of device control processing executed by an air conditioner according to an embodiment. A flowchart showing an example of the flow of device control processing executed by an air conditioner according to an embodiment. 1 is a flowchart showing an example of the flow of an operation determination process executed by a cloud server according to an embodiment. 1 is a flowchart showing an example of the flow of a weighting coefficient determination process executed by a cloud server according to an embodiment. 1 is a flowchart showing an example of the flow of an operation determination process executed by a cloud server according to an embodiment.

A control system according to an embodiment of the present disclosure will be described below with reference to the drawings. The control system according to the present embodiment includes a neural network storage unit, an imaging device, an image acquisition unit, a selection unit, a neural network calculation unit, an operation identification unit, a device control unit, and a neural network update unit. The neural network storage unit stores a neural network for identifying one type of operation content corresponding to a gesture image of the same type from gesture image information indicating a gesture of the same type of user corresponding to each of a plurality of devices. The imaging device is provided in each of the plurality of devices and images the user of each of the plurality of devices. The image acquisition unit acquires, from the imaging device, gesture image information indicating the gesture of the user captured by the imaging device. The selection unit selects, from the neural networks stored in the neural network storage unit, a neural network corresponding to the device provided with the imaging device that generated the gesture image information acquired by the image acquisition unit. The neural network calculation unit calculates a judgment value corresponding to the gesture image information from the gesture image information acquired by the image acquisition unit using the selected neural network. The operation identification unit identifies operation identification information that identifies an operation on a device associated with the judgment value calculated by the neural network calculation unit. The device control unit controls a device provided with an imaging device that generated the gesture image information, based on the operation identification information identified by the operation identification unit. The neural network update unit updates the neural network selected by the selection unit, based on the gesture image information and the operation identification information corresponding to the judgment value calculated from the image information, every time the image acquisition unit acquires the gesture image information. In addition, the neural network update unit identifies a neural network with the least amount of calculation from among the neural networks corresponding to each of the multiple devices stored in the neural network storage unit, and updates all of the neural networks corresponding to each of the multiple devices to the identified neural network, every time a preset neural network update time arrives.

As shown in FIG. 1, the control system according to this embodiment includes a plurality of air conditioners 1A and 1B (two in FIG. 1) installed in a building H, and a cloud server 2. Operating devices 6A and 6B for operating the air conditioners 1A and 1B, respectively, are installed in the building H. Here, the air conditioners 1A and 1B are each installed in a room in the building H. Also installed in the building H are a router 82 connected to a local network NW2 constructed in the building H, and a data circuit-terminating device 81 connected to the router 82 and a wide area network NW1. The wide area network NW1 is, for example, the Internet. Also, the local network NW2 is, for example, a wired LAN (Local Area Network) or a wireless LAN. The data circuit-terminating device 81 is, for example, a modem, a gateway, etc.

As shown in FIG. 2, the air conditioners 1A and 1B according to this embodiment have a control unit 100 and an imaging device 181. The air conditioner 1 also has a compressor (not shown) and a blower fan (not shown) that operate based on a control signal input from the control unit 100. The cloud server 2, the imaging devices 181 of the air conditioners 1A and 1B, and the control unit 100 constitute a learning system that updates the weight coefficients of the neural network, i.e., trains the neural network, as described below. The imaging device 181 has a camera using a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The imaging device 181 generates an image signal corresponding to an image captured by the camera and outputs it to the imaging interface 108 of the control unit 100 described below.

The control unit 100 includes a CPU (Central Processing Unit) 101, a main memory 102, an auxiliary memory 103, a neuro engine 104, a clock 105, a local communication unit 106, a wireless module 107, an imaging interface 108, and a bus 109 that interconnects these. The main memory 102 is made of a volatile memory and is used as a working area for the CPU 101. The auxiliary memory 103 is made of a non-volatile memory such as a semiconductor flash memory, and stores programs for implementing various functions of the control unit 100. The neuro engine 104 is hardware dedicated to arithmetic processing using a neural network having a preset number of nodes and layers. The clock 105 is, for example, a real-time clock. The local communication unit 106 is connected to the local network NW2, and sends various information notified from the CPU 101 to the local network NW2 and notifies the CPU 101 of various information received from the local network NW2. Wireless module 107 wirelessly communicates with manipulation devices 6A, 6B using a communication method conforming to a short-range wireless communication standard such as Bluetooth (registered trademark), and upon receiving operation information indicating the operation content performed by the user on manipulation devices 6A, 6B from manipulation devices 6A, 6B, notifies CPU 101 of the operation information. Note that wireless module 107 may also perform infrared communication with manipulation devices 6A, 6B based on the IrDA (Infrared Data Association) standard, for example. When an image signal is input from imaging device 181, imaging interface 108 generates image information corresponding to the signal and notifies CPU 101.

The CPU 101 reads out the program stored in the auxiliary storage unit 103 into the main storage unit 102 and executes it, thereby functioning as an image acquisition unit 111, an image correction unit 112, an operation reception unit 113, a device control unit 114, an operation identification information transmission unit 115, an operation mode setting unit 116, a user identification unit 117, an image transmission unit 118, and a device setting unit 119, as shown in Fig. 3. The auxiliary storage unit 103 also has an image storage unit 131, a user information storage unit 132, an operation mode storage unit 133, a device setting storage unit 134, and an operation information storage unit 135. The image storage unit 131 stores gesture image information indicating a gesture of a user captured by the imaging device 181. The image storage unit 131 stores, for example, gesture image information indicating a gesture image GE11 corresponding to the operation of "air conditioner ON", gesture image information indicating a gesture image GE12 corresponding to the operation of "air conditioner OFF", gesture image information indicating a gesture image GE13 corresponding to the operation of "change the temperature setting of the air conditioner", or gesture image information indicating a gesture image GE14 corresponding to the operation of "maintain the temperature setting of the air conditioner", as shown in FIG. 4(A). Alternatively, the image storage unit 131 stores, for example, gesture image information indicating a gesture image GE21 corresponding to the operation of "air conditioner ON", gesture image information indicating a gesture image GE22 corresponding to the operation of "air conditioner OFF", gesture image information indicating a gesture image GE23 corresponding to the operation of "change the temperature setting of the air conditioner", or gesture image information indicating a gesture image GE24 corresponding to the operation of "maintain the temperature setting of the air conditioner", as shown in FIG. 4(B).

Returning to FIG. 3, the user information storage unit 132 stores image feature information indicating the features of the user, such as the contour and dimensions, that appear in the image of the user captured by the imaging device 181, in association with user identification information that identifies the user. The operation mode storage unit 133 stores operation mode information indicating the operation mode when operating the air conditioners 1A and 1B. Here, the operation modes include a normal operation mode in which the operation settings of the air conditioners 1A and 1B can be changed only by the user's operation on the operation devices 6A and 6B, and a gesture operation mode in which the operation settings of the air conditioners 1A and 1B are changed according to the gesture of the user captured by the imaging device 181. The operation modes also include a gesture registration mode for registering a new user gesture in the air conditioners 1A and 1B, that is, for updating the correspondence between the operation identification information and the judgment value information stored in the operation identification information storage unit 233 of the cloud server 2 described later. The device setting storage unit 134 stores device setting information indicating the device setting parameters of the air conditioners 1A and 1B.

Returning to FIG. 3, the image acquisition unit 111 acquires gesture image information indicating a gesture image captured by the imaging device 181, and stores the acquired image information in the image storage unit 131. When the operation mode information stored in the operation mode storage unit 133 indicates the gesture registration mode, the image acquisition unit 111 waits for a waiting time that is preset based on the time measured by the clock unit 105 after the operation mode information indicating the gesture registration mode is stored in the operation mode storage unit 133, and then acquires, from the imaging device 181, gesture image information corresponding to the user's gesture captured by the imaging device 181.

The image correction unit 112 corrects the gesture image information acquired by the image acquisition unit 111. Here, the image correction unit 112 corrects the gesture image information based on imaging conditions such as the performance of the imaging device 181, the installation position of the imaging device 181, the brightness around the imaging device 181, and the background image of the user's gesture image captured by the imaging device 181. For example, the image correction unit 112 roughly sets a boundary between an image portion including the user's gesture image and a background image portion assuming that the user's image is present in the center of the gesture image information, then calculates a new boundary by performing graph cutting based on the color distribution of each image portion, and calculates a new boundary by performing graph cutting again based on the color distribution of each image portion partitioned by the new boundary, thereby extracting the user's gesture image from the background image. Then, the image correction unit 112 performs angle of view correction based on the installation position of the imaging device 181 for the extracted gesture image information showing the gesture image of the user, and then performs averaging or complementation processing to convert the number of pixels of the gesture image to a preset reference number of pixels based on the number of pixels of the imaging device 181, and then performs brightness correction (e.g., backlight correction) based on the brightness around the imaging device 181 before storing the image in the image storage unit 131.

When the wireless module 107 receives operation information transmitted from the operation devices 6A, 6B, the operation reception unit 113 accepts the received operation information. Then, if the operation information relates to updating the device setting parameters of the air conditioners 1A, 1B, the operation reception unit 113 notifies the operation identification information transmission unit 115 and the device setting unit 119 of the operation information. Furthermore, if the operation information relates to changing the operation mode of the air conditioners 1A, 1B, the operation reception unit 113 notifies the operation mode setting unit 116 of the operation information.

When the operation information is notified from the operation reception unit 113, the operation mode setting unit 116 identifies operation mode information based on the notified operation information, and stores the identified operation mode information in the operation mode storage unit 133.

The user identification unit 117 extracts features of the user's contour, dimensions, etc. that appear in the user's image based on the image information acquired by the image acquisition unit 111. The user identification unit 117 then identifies, from the user identification information stored in the user information storage unit 132, the user identification information that corresponds to image feature information that shows the same features as the extracted features, as the user identification information of the user of the air conditioners 1A and 1B. The user identification unit 117 also stores the identified user identification information of the user of the air conditioners 1A and 1B in the user information storage unit 132.

When the operation mode information stored in the operation mode storage unit 133 indicates the gesture operation mode, the device setting unit 119 acquires operation identification information transmitted from the cloud server 2 and calculates device setting parameters based on the acquired operation identification information. Then, the device setting unit 119 stores the device setting information indicating the calculated device setting parameters in the device setting storage unit 134. Furthermore, when the device setting unit 119 is notified of the operation identification information from the operation reception unit 113, it calculates device setting parameters based on the operation identification information and stores the device setting information indicating the calculated device setting parameters in the device setting storage unit 134. Then, the device control unit 114 controls the operation of the compressor and the blower fan based on the device setting information stored in the device setting storage unit 134.

When the operation identification information transmission unit 115 acquires operation information from the operation reception unit 113, it selects operation identification information corresponding to the acquired operation information from the operation identification information stored in the operation information storage unit 135. Then, the operation identification information transmission unit 115 transmits the selected operation identification information to the cloud server 2, and notifies the image transmission unit 118 of operation identification information transmission notification information notifying that the operation identification information will be transmitted.

When new gesture image information is stored in the image storage unit 131 and the operation mode information stored in the operation mode storage unit 133 indicates the gesture operation mode, the image transmission unit 118 transmits the gesture image information, the user information corresponding to the gesture image information stored in the user information storage unit 132, and the device identification information identifying the air conditioners 1A and 1B to the cloud server 2. When the operation mode information stored in the operation mode storage unit 133 indicates the gesture registration mode and the operation identification information transmission notification information is notified from the operation identification information transmission unit 115, the image transmission unit 118 transmits the newly stored gesture image information stored in the image storage unit 131 and the device identification information identifying the air conditioners 1A and 1B to the cloud server 2.

When the operation receiving unit 113 receives a new registration operation for newly registering neural networks corresponding to the air conditioners 1A and 1B in the cloud server 2, the registration request unit 120 identifies the user based on the operation content of the new registration operation. Here, the registration request unit 120 accepts the new registration operation when the user performs a new device registration operation on the operation devices 6A and 6B to register the air conditioners 1A and 1B newly installed in building H, registering device identification information of the newly installed air conditioners 1A and 1B and information about the user. Then, the registration request unit 120 transmits registration request information including the user information of the identified user and the device identification information of the air conditioners 1A and 1B to the cloud server 2.

Returning to FIG. 2, the cloud server 2 comprises a CPU 201, a main memory unit 202, an auxiliary memory unit 203, a clock unit 205, a wide area communication unit 206, and a bus 209 that interconnects these units. The CPU 201 is, for example, a multi-core processor. The main memory unit 202 is composed of a volatile memory, and is used as a working area for the CPU 201. The auxiliary memory unit 203 is composed of a large-capacity non-volatile memory, and stores programs for implementing various functions of the cloud server 2. The clock unit 205 is, for example, a real-time clock. The wide area communication unit 206 is connected to the wide area network NW1.

The CPU 201 reads out the program stored in the auxiliary storage unit 203 into the main storage unit 202 and executes it, thereby functioning as an image acquisition unit 211, a selection unit 212, a neural network calculation unit (hereinafter referred to as the "NN calculation unit") 213, an operation identification unit 214, an operation identification information notification unit 215, a neural network update unit (hereinafter referred to as the "NN update unit") 216, and an operation identification information acquisition unit 217, as shown in FIG. 5. The auxiliary storage unit 203 also has a neural network storage unit (hereinafter referred to as the "NN storage unit") 231, an image storage unit 232, and an operation identification information storage unit 233. The image storage unit 232 stores the gesture image information and user information acquired from the air conditioners 1A and 1B, and the device identification information of the air conditioners 1A and 1B, in association with each other.

The NN storage unit 231 stores a neural network for identifying one type of operation content corresponding to the same type of gesture image from gesture image information showing the same type of gesture of the user corresponding to each of the air conditioners 1A and 1B. Specifically, the NN storage unit 231 stores information indicating the structure of a neural network for identifying an operation performed by a user from the user's gesture image information and information indicating weighting coefficients in the neural network in association with the device identification information of the corresponding air conditioners 1A and 1B. Here, the information indicating the structure of the neural network includes information indicating the number of nodes and layers of the neural network, and weighting coefficients and activation functions corresponding to each node. This neural network has an input layer L10, a hidden layer L20, and an output layer L30 as shown in FIG. 6. The input layer L10 inputs pixel position information and pixel value information included in the gesture image information to the hidden layer L20.

The hidden layer L20 is composed of N (N is a positive integer) layers including a preset number M[j] of nodes x[j,i] (1≦i≦M[j], M[j] is a positive integer). That is, the hidden layer L20 has a structure in which each node column is connected to each other. Here, the output y[j,i] of each node x[j,i] is expressed by the relational expression of the following formula (1).
Here, W[j,i,k] indicates a weighting coefficient, and f(*) indicates an activation function. The weighting coefficient W[j,i,k] corresponds to the neural network coefficient that determines the structure of the neural network described above. In addition, a nonlinear function such as a sigmoid function, a ramp function, a step function, or a softmax function is used as the activation function. In the hidden layer L20, information input to the node is the sum of the outputs of each node in the previous layer multiplied by a weighting coefficient. Then, the output of the activation function with the sum as an argument is transmitted to the next layer. The output layer L30 outputs the output y[j,i] from the final layer of the hidden layer L20 as it is. Then, the neural network storage unit 134 stores information indicating the structure of the neural network expressed by formula (1) and information indicating the weighting coefficient W[j,i,k] of the neural network.

This neural network is a so-called convolutional neural network (CNN), and the hidden layer L20 includes a convolutional layer, a pooling layer, a fully connected layer, and a judgment layer. In the convolutional layer, a convolution operation is performed using a plurality of convolution filters of a preset width. In this convolutional layer, the degree of conformity of features in a partial area of the image indicated by the gesture image information is calculated, and the image is converted into information indicating where the image of which feature is located. In addition, in the convolutional layer, for example, the same number of pieces of information as filters having a two-dimensional array structure are output. In the pooling layer, an average pooling process is performed with a preset width on the output from the convolutional layer. In addition, in the pooling layer, the information having a two-dimensional array structure input from the convolutional layer is averaged and compressed in the two-dimensional direction, so that only information effective for distinguishing the gesture remains. As a result, even if there is a slight difference in the shape of the user's object included in the gesture image, the influence of the difference on the gesture discrimination can be eliminated. Then, the convolution layer and the pooling layer are combined to calculate the feature quantity corresponding to the image information from the image information input from the input layer. In particular, the calculation of the convolution layer and the calculation of the pooling layer are alternately performed multiple times, thereby reducing the amount of calculation. The fully connected layer calculates the sum of values obtained by multiplying each of the multiple feature quantities by a weighting coefficient, converts the calculated sum into a feature variable using an activation function, and outputs it. Here, for example, a sigmoid function is adopted as the activation function. In the fully connected layer, all information output from the pooling layer at the final stage is input and output as information with a one-dimensional structure. In addition, in the fully connected layer, when the number of nodes corresponding to the pooling layer at the final stage increases, the number of divisions of the feature quantity space increases accordingly, and the number of feature functions characterizing each region obtained by dividing the feature quantity space increases accordingly. The judgment layer estimates the operation content corresponding to the image information by converting the feature variables output from the fully connected layer into judgment values within a preset numerical range using, for example, a softmax function. As shown in FIG. 7, the amount of information processed by the NN calculation unit 213 is ND1 in the input layer, but decreases as it progresses to the convolution layer and the pooling layer, and decreases to ND2 just before the fully connected layer. This is because the pooling process averages multiple features and aggregates them into one feature. The information indicating the structure of the neural network includes information indicating the width of the convolution filter and information indicating the width in the average pooling process. The information indicating the weighting coefficients includes information indicating the weighting coefficients of each of the multiple convolution filters used in the convolution layer and information indicating the weighting coefficients in the fully connected layer. The NN storage unit 231 also stores information indicating the initial values of the weighting coefficients used when determining the weighting coefficients of the neural network.

The operation identification information storage unit 233 stores judgment value information indicating the range of judgment values assigned to various operations performed by the user in association with operation identification information that identifies the operation content performed by the user. For example, as shown in FIG. 8, the range of judgment values corresponding to the operation identification information that identifies the operation of "air conditioner ON" is set to NUM11 or more and less than NUM12, and the range of judgment values corresponding to the operation identification information that identifies the operation of "air conditioner OFF" is set to NUM21 or more and less than NUM22. In addition, the range of judgment values corresponding to the operation identification information that identifies the operation of "change temperature setting of air conditioner" is set to NUM31 or more and less than NUM32, and the range of judgment values corresponding to the operation identification information that identifies the operation of "maintain temperature setting of air conditioner" is set to NUM41 or more and less than NUM42. The ranges of judgment values are set so as not to overlap with each other.

Returning to FIG. 5, when the image acquisition unit 211 acquires the gesture image information, user information, and device identification information transmitted from the air conditioners 1A and 1B, it stores these in the image storage unit 232 in association with each other. The image acquisition unit 211 also notifies the selection unit 212 of the acquired device identification information. This device identification information is information that identifies the air conditioners 1A and 1B in which the imaging device 181 that generated the gesture image information acquired by the image acquisition unit 211 is installed.

The selection unit 212 selects a neural network corresponding to the device identification information notified by the image acquisition unit 211 from among the neural networks stored in the NN storage unit 231. Specifically, the selection unit 212 selects information indicating the structure of the neural network corresponding to the device identification information notified by the image acquisition unit 211 and information indicating weighting coefficients. Then, the selection unit 212 sets the weighting coefficients of each of the selected convolution filters to the selected convolution filters. In addition, the selection unit 212 sets the weighting coefficients in the fully connected layer of the selected neural network to a portion corresponding to the fully connected layer of the selected neural network.

The NN calculation unit 213 calculates a judgment value from the gesture image information and user identification information stored in the image storage unit 232 using the neural network selected by the selection unit 212 based on the device identification information. The NN calculation unit 213 also calculates feature amounts from the gesture image information and user information by performing arithmetic processing using parts corresponding to the convolution layer and pooling layer in the neural network. Here, the NN calculation unit 213 calculates feature amounts by repeating convolution calculations using multiple convolution filters and pooling processing a preset number of times for the gesture image information and the user information. Then, the NN calculation unit 213 calculates a judgment value for the feature amounts corresponding to the gesture image information and the user information by performing arithmetic processing using parts corresponding to the fully connected layer and judgment layer in the neural network. The NN calculation unit 213 uses the neural network selected by the selection unit 212 to calculate a judgment value included in the range of "NUM11" or more and less than "NUM12" from image information showing images GE11, GE21 of the same type of gesture corresponding to "air conditioner ON", for example as shown in Figures 4 (A) and (B), and calculates a judgment value included in the range of "NUM21" or more and less than "NUM22" from image information showing images GE12, GE22 of the same type of gesture corresponding to "air conditioner OFF". In addition, the NN calculation unit 213 uses the neural network selected by the selection unit 212 to calculate a judgment value included in the range of "NUM31" to "NUM32" from image information showing images GE13 and GE23 of the same type of gesture corresponding to "change temperature setting" shown in Figs. 4(A) and (B), and calculates a judgment value included in the range of "NUM41" to "NUM42" from images GE14 and GE24 of the same type of gesture corresponding to "maintain temperature setting". The NN calculation unit 213 notifies the operation specification unit 214 and the NN update unit 216 of judgment value information indicating the calculated judgment value. In this way, even if the users shown by the gesture image information are different, the NN calculation unit 213 calculates a judgment value included in the same range of judgment values if the gestures shown by the gesture image information correspond to the same type of gesture operation.

When the operation identification unit 214 is notified of the judgment value information from the NN calculation unit 213, it identifies the operation identification information corresponding to the notified judgment value information from among the operation identification information stored in the operation identification information storage unit 233. Then, the operation identification unit 214 notifies the operation identification information transmission unit 215 of the identified operation identification information. Based on the device identification information stored in the image storage unit 232, the operation identification information transmission unit 215 transmits the operation identification information notified from the operation identification unit 214 to the air conditioner 1A, 1B that is the sender of the gesture image information corresponding to the operation identification information.

Each time the image acquisition unit 211 acquires gesture image information, user information, and device identification information, the NN update unit 216 updates the neural network selected by the selection unit 212 based on the gesture image information and the operation identification information corresponding to the judgment value calculated from the gesture image information. Specifically, the NN update unit 216 first calculates the error between the judgment value notified from the NN calculation unit 213 and the median value of the range of judgment values indicated by the judgment value information corresponding to the operation identification information stored in the operation identification information storage unit 233, and determines the weight coefficient of the neural network by error backpropagation based on the calculated error. Here, the NN update unit 216 determines the weight coefficient of each of the multiple convolution filters used in the convolution layer of the neural network and the weight coefficient of the fully connected layer of the neural network. Then, the NN update unit 216 updates the information indicating the weight coefficients of the neural network selected by the selection unit 212, which is stored in the NN storage unit 231, with information indicating the determined weight coefficients.

In addition, each time a preset neural network update time (hereinafter referred to as the "NN update time") arrives, the NN update unit 216 identifies the neural network with the least amount of calculations from among the neural networks corresponding to the air conditioners 1A and 1B stored in the NN storage unit 231. The NN update unit 216 then updates all of the neural networks corresponding to the air conditioners 1A and 1B to the identified neural network.

Next, the operation of the control system according to the present embodiment will be described with reference to FIG. 9 and FIG. 10. First, as shown in FIG. 9, it is assumed that the air conditioners 1A and 1B are set to the gesture registration mode by the user (step S1). Next, when the air conditioners 1A and 1B determine that a preset waiting time has elapsed after the operation mode is set to the gesture registration mode (step S2), they identify operation identification information based on the operation performed by the user on the operation devices 6A and 6B during the waiting time (step S3). Next, the air conditioners 1A and 1B acquire gesture image information obtained by capturing an image of a gesture performed by the user to operate the air conditioners 1A and 1B (step S4). At this time, the air conditioners 1A and 1B may notify the user that the user's gesture will be captured by blinking an LED (Light Emitting Diode) lamp (not shown) provided on the air conditioner 1 or sounding a buzzer (not shown) provided on the air conditioners 1A and 1B. Thereafter, the air conditioners 1A and 1B correct the acquired gesture image information (step S5). Next, the air conditioners 1A and 1B identify the users of the air conditioners 1A and 1B based on the corrected gesture image information (step S6). Next, the gesture image information acquired by the air conditioners 1A and 1B, user information indicating the identified user, operation identification information identified by the air conditioners 1A and 1B, and device identification information of the air conditioners 1A and 1B are transmitted from the air conditioners 1A and 1B to the cloud server 2 (step S7).

On the other hand, when the cloud server 2 acquires the gesture image information, user information, operation identification information, and device identification information, the cloud server 2 associates the acquired gesture image information, user information, and device identification information with each other and stores them in the image storage unit 232. Next, the cloud server 2 selects a neural network corresponding to the air conditioner 1A, 1B that transmitted the gesture image information, operation identification information, and device identification information based on the device identification information (step S8). Next, the cloud server 2 calculates a judgment value from the gesture image information and user information using the selected neural network (step S9). After that, the cloud server 2 stores judgment value information indicating the calculated judgment value in the operation identification information storage unit 233 in association with the operation identification information acquired from the air conditioners 1A, 1B (step S10).

Next, assume that the user performs an operation mode change operation to change the operation mode of the air conditioners 1A and 1B to the gesture operation mode. In this case, the operation mode of the air conditioner 1 is set to the gesture operation mode (step S11). Next, the air conditioners 1A and 1B acquire gesture image information obtained by capturing an image of a gesture for operating the air conditioners 1A and 1B performed by the user (step S12), and correct the acquired gesture image information (step S13). After that, the air conditioners 1A and 1B identify the user of the air conditioners 1A and 1B based on the gesture image information (step S14). Next, the gesture image information acquired by the air conditioners 1A and 1B, user information indicating the identified user, and device identification information of the air conditioners 1A and 1B are transmitted from the air conditioners 1A and 1B to the cloud server 2 (step S15). Next, the cloud server 2 selects a neural network corresponding to the air conditioner 1A, 1B that transmitted the gesture image information, operation identification information, and device identification information based on the device identification information (step S16). After that, the cloud server 2 uses the selected neural network to calculate a judgment value from the gesture image information and user information (step S17).

Next, the cloud server 2 identifies the operation identification information corresponding to the calculated judgment value from among the operation identification information stored in the operation identification information storage unit 233 (step S18). The identified operation identification information is then transmitted from the cloud server 2 to the air conditioners 1A and 1B that are the transmission sources of the gesture image information, the user information, and the device identification information (step S19). On the other hand, upon acquiring the operation identification information, the air conditioners 1A and 1B calculate device setting parameters corresponding to the acquired operation identification information, generate device setting information indicating the calculated device setting parameters, and update the device setting information stored in the device setting storage unit 131 (step S20). At this time, the air conditioners 1A and 1B control the operation of the compressor and the blower fan based on the updated device setting information stored in the device setting storage unit 134.

The cloud server 2 also determines a weighting coefficient for the selected neural network based on a judgment value corresponding to the identified operation identification information stored in the operation identification information storage unit 233 and a judgment value calculated from the gesture image information, the user information, and the device identification information (step S21). After that, the cloud server 2 updates the weighting coefficient information for the selected neural network stored in the NN storage unit 231 with information indicating the determined weighting coefficient (step S22).

As shown in FIG. 10, when the preset NN update time arrives, the cloud server 2 compares the amount of calculations in the neural networks of the air conditioners 1A and 1B stored in the NN memory unit 231 (step S23). Next, the cloud server 2 identifies the neural network with the least amount of calculations (step S24). Next, the cloud server 2 updates all of the weighting coefficient information of the neural networks corresponding to the air conditioners 1A and 1B to the weighting coefficient information of the identified neural network (step S25).

After the series of processes from step S12 to S17 described above are executed, it is assumed that the cloud server 2 has determined that it has failed to identify the operation identification information (step S26). Here, if the judgment value calculated using the selected neural network is significantly different from the range of judgment values indicated by the judgment value information corresponding to each of the operation identification information stored in the operation identification information storage unit 233 and is not included in any of the judgment value ranges, the cloud server 2 determines that it has failed to identify the operation identification information. In this case, the cloud server 2 newly selects a neural network corresponding to the other air conditioners 1A and 1B that is different from the selected neural network (step S27). Then, the cloud server 2 uses the newly selected neural network to calculate a judgment value from the gesture image information and user information acquired from the air conditioners 1A and 1B (step S28). Thereafter, if the cloud server 2 is able to identify the operation identification information based on the calculated judgment value (step 29), the identified operation identification information is transmitted from the cloud server 2 to the air conditioners 1A and 1B that are the senders of the gesture image information, user information, and device identification information (step S19).

The cloud server 2 also determines a weighting coefficient for the previously selected neural network based on a judgment value corresponding to the specified operation identification information stored in the operation identification information storage unit 233 and a judgment value calculated from the gesture image information, user information, and device identification information using the previously selected neural network (step S30). Next, the cloud server 2 updates the weighting coefficient information for the previously selected neural network stored in the NN storage unit 231 with information indicating the determined weighting coefficient (step S31).

Also, suppose that the user performs a new device registration operation to register the air conditioners 1A and 1B newly installed in the building H. In this case, the operation mode of the air conditioners 1A and 1B is set to the new registration mode (step S32). Next, the air conditioners 1A and 1B identify the user based on the operation content of the new registration operation (step S33). After that, the identified user information and the device identification information of the air conditioners 1A and 1B are transmitted from the air conditioners 1A and 1B to the cloud server 2 (step S34). On the other hand, when the cloud server 2 acquires the identified user information and the device identification information of the air conditioners 1A and 1B, it selects one neural network based on the user information from among the neural networks stored in the NN storage unit 231 (step S35). Next, the cloud server 2 associates the NN structure information indicating the structure of the selected neural network and the weight coefficient information indicating the weight coefficient with the acquired device identification information and newly stores it in the NN storage unit 231 (step S36).

As shown in FIG. 11, suppose that within a preset reference time after the device setting information is updated by executing a series of processes from step S12 to S22, the user performs a switching operation to switch the air conditioners 1A and 1B from the gesture operation mode to the manual operation mode in order to change the device settings of the air conditioners 1A and 1B. That is, suppose that the user performs a correction operation to correct the device settings of the air conditioners 1A and 1B set by the gesture operation. In this case, the air conditioners 1A and 1B accept a correction operation to correct the operation content corresponding to the acquired gesture image, and the operation mode of the air conditioners 1A and 1B is set to the manual operation mode (step S37). Next, when the air conditioners 1A and 1B determine that a preset waiting time has elapsed after the operation mode was set to the manual operation mode (step S38), they identify the operation identification information based on the operation performed by the user on the operation devices 6A and 6B during the waiting time (step S39). The identified operation identification information and the device identification information of the air conditioners 1A and 1B are then transmitted from the air conditioners 1A and 1B to the cloud server 2 (step S40). Meanwhile, upon acquiring the operation identification information and the device identification information of the air conditioners 1A and 1B, the cloud server 2 selects neural networks corresponding to the air conditioners 1A and 1B (step S41). Next, the cloud server 2 determines the weighting coefficients of each of the selected neural networks based on the judgment values corresponding to the identified operation identification information stored in the operation identification information storage unit 233 and the judgment values calculated in step S17 (step S42). Next, the cloud server 2 updates the weighting coefficient information of the neural networks stored in the NN storage unit 231 with information indicating the weighting coefficients determined for each neural network (step S43).

Next, the device control process executed by the air conditioners 1A, 1B according to this embodiment will be described with reference to Figs. 12 and 13. This device control process is started, for example, when the air conditioners 1A, 1B are powered on. Furthermore, when the user performs an operation on the operation devices 6A, 6B to change the operation mode, the operation reception unit 113 receives operation information for changing the operation mode information from the operation devices 6A, 6B and notifies the operation mode setting unit 116 of the operation information. Then, each time the operation reception unit 113 notifies the operation information for changing the operation mode information, the operation mode setting unit 116 updates the operation mode information stored in the operation mode storage unit 133 based on the notified operation information.

First, the operation reception unit 113 determines whether or not a new registration operation for newly registering neural networks corresponding to the air conditioners 1A and 1B in the NN memory unit 231 of the cloud server 2 has been received (step S101). If the operation reception unit 113 determines that a new registration operation has not been received (step S101: No), the process of step S104 described below is executed. On the other hand, if the operation reception unit 113 determines that a new registration operation has been received (step S101: Yes), the registration request unit 120 identifies a user based on the operation content of the new registration operation (step S102). Next, the registration request unit 120 transmits user information of the identified user and device identification information of the air conditioners 1A and 1B to the cloud server 2 (step S103).

Next, the operation reception unit 113 determines whether the air conditioner 1 is set to the gesture registration mode by referring to the operation mode information stored in the operation mode storage unit 133 (step S104). If the operation reception unit 113 determines that the air conditioners 1A and 1B are not set to the gesture registration mode (step S104: No), the processing of step S111 described below is executed. On the other hand, if the operation reception unit 113 determines that the air conditioners 1A and 1B are set to the gesture registration mode (step S104: Yes), it determines whether a preset waiting time has elapsed since the operation reception unit 113 was set to the gesture registration mode (step S105). As long as the operation reception unit 113 determines that the aforementioned waiting time has not yet elapsed since the operation reception unit 113 was set to the gesture registration mode (step S105: No), it repeatedly executes the processing of step S105. On the other hand, when the operation reception unit 113 determines that the aforementioned waiting time has elapsed after the air conditioners 1A and 1B were set to the gesture registration mode (step S105: Yes), if the operation information received from the operation devices 6A and 6B is related to updating the device setting parameters of the air conditioners 1A and 1B, it identifies the operation identification information corresponding to the operation information (step S106). At this time, the operation reception unit 113 notifies the operation identification information transmission unit 115 of the identified operation identification information.

Then, the image acquisition unit 111 acquires gesture image information of the gesture to be registered, which is obtained by imaging the image capture device 181 (step S107). At this time, the image acquisition unit 111 may notify the user that the user's gesture will be imaged by notifying a lighting device (not shown) that lights the LED lamp of command information for blinking the LED lamp provided in the air conditioner 1, or notifying a buzzer drive unit (not shown) that drives the buzzer of command information for sounding the buzzer provided in the air conditioner 1. Next, the image correction unit 112 corrects the gesture image information acquired by the image acquisition unit 111 (step S108). At this time, the image correction unit 112 stores the corrected gesture image information in the image storage unit 131. Next, the user identification unit 117 identifies the user based on the gesture image information stored in the image storage unit 131 (step S109). At this time, the user identification unit 117 stores the user information of the identified user in the user storage unit 132. After that, the image transmission unit 118 transmits the gesture image information, the user information, and the device identification information of the air conditioners 1A and 1B to the cloud server 2, and the operation identification information transmission unit 115 transmits the operation identification information notified by the operation reception unit 113 to the cloud server 2 (step S110).

Next, the image acquisition unit 111 determines whether the operation mode indicated by the operation mode information stored in the operation mode storage unit 133 is the gesture operation mode (step S111). If the image acquisition unit 111 determines that the operation mode information stored in the operation mode storage unit 133 does not indicate the gesture operation mode (step S111: No), the process of step S101 is executed again. On the other hand, if the image acquisition unit 111 determines that the operation mode information stored in the operation mode storage unit 133 indicates the gesture operation mode (step S111: Yes), the image acquisition unit 111 determines whether gesture image information corresponding to a gesture for operating the air conditioners 1A and 1B by the user has been acquired (step S112). Here, if the image acquisition unit 111 determines that gesture image information has not been acquired (step S112: No), the process of step S101 is executed again.

On the other hand, when the image acquisition unit 111 determines that the user has acquired gesture image information for operating the air conditioners 1A and 1B (step S112: Yes), the image correction unit 112 corrects the gesture image information (step S113). Next, the user identification unit 117 identifies the user based on the gesture image information stored in the image storage unit 131 (step S114). After that, the image transmission unit 118 transmits the gesture image information, the user information, and the device identification information of the air conditioners 1A and 1B to the cloud server 2 (step S115). Next, as shown in FIG. 13, the device setting unit 119 determines whether or not the operation identification information transmitted from the cloud server 2 has been acquired within a preset waiting time (step S116). Here, the waiting time is set to, for example, several seconds. When the device setting unit 119 determines that the operation identification information could not be acquired within the aforementioned waiting time (step S116: No), the processing of step S101 is executed again assuming that the gesture operation was not performed. On the other hand, when the device setting unit 119 determines that it has acquired the operation identification information transmitted from the cloud server 2 (step S116: Yes), it calculates device setting parameters based on the acquired operation identification information and stores device setting information indicating the calculated device setting parameters in the device setting storage unit 134 (step S117). Then, the device control unit 114 controls the operation of the compressor and the blower fan based on the device setting information stored in the device setting storage unit 134.

Then, the operation reception unit 113 determines whether the time elapsed since the device setting information stored in the device setting storage unit 134 was updated has passed a preset reference time based on the time information clocked by the clock unit 105 (step S118). If the operation reception unit 113 determines that the time elapsed since the device setting information was updated has passed the aforementioned reference time (step S118: Yes), the process of step S101 is executed again. On the other hand, if the operation reception unit 113 determines that the time elapsed since the device setting information was updated has not yet passed the aforementioned reference time (step S118: No), it determines whether the operation mode information stored in the operation mode storage unit 133 has been changed from the gesture operation mode to the manual operation mode (step S119). That is, the operation reception unit 113 determines whether a correction operation for correcting the operation content corresponding to the gesture image acquired by the image acquisition unit 111 has been accepted. If the operation reception unit 113 determines that the operation mode has not yet been changed to the manual operation mode (step S119: No), the process of step S118 is executed again.

On the other hand, when the operation reception unit 113 determines that the operation mode has been changed to the manual operation mode (step S119: Yes), it determines whether or not a preset waiting time has elapsed since the operation mode was changed to the manual operation mode (step S120). As long as the operation reception unit 113 determines that the aforementioned waiting time has not yet elapsed since the operation mode was changed to the manual operation mode (step S120: No), it repeatedly executes the process of step S120. On the other hand, when the operation reception unit 113 determines that the aforementioned waiting time has elapsed since the operation mode was changed to the manual operation mode (step S120: Yes), it identifies the operation identification information corresponding to the operation information received from the operation devices 6A and 6B (step S121). At this time, the operation reception unit 113 notifies the operation identification information transmission unit 115 of the identified operation identification information. Next, the operation identification information transmission unit 115 transmits the operation identification information notified from the operation reception unit 113 and the device identification information of the air conditioners 1A and 1B to the cloud server 2 (step S122). Next, step S101 is executed again.

Next, the operation determination process executed by the cloud server 2 according to the present embodiment will be described with reference to FIG. 14 to FIG. 16. This operation determination process is started, for example, when the power is turned on to the cloud server 2 and a program for executing the operation determination process is started. First, the image acquisition unit 211 acquires gesture image information, user information, and device identification information from the air conditioners 1A and 1B, and the operation identification information acquisition unit 117 determines whether or not it has acquired the operation identification information from the air conditioners 1A and 1B (step S201). Here, if it is determined that the image acquisition unit 211 has not acquired gesture image information, user information, and device identification information from the air conditioners 1A and 1B, or that the operation identification information acquisition unit 117 has not acquired the operation identification information from the air conditioners 1A and 1B (step S201: No), the process of step S204 described below is executed. On the other hand, it is assumed that the image acquisition unit 211 acquires gesture image information, user information, and device identification information from the air conditioners 1A and 1B, and that the operation identification information acquisition unit 117 determines that it has acquired the operation identification information from the air conditioners 1A and 1B (step S201: Yes). In this case, the selection unit 212 selects a neural network corresponding to the device identification information notified from the image acquisition unit 211 from the neural networks stored in the NN storage unit 231 (step S202). Next, the NN calculation unit 213 calculates a judgment value from the gesture image information and user identification information stored in the image storage unit 232 using the neural network selected by the selection unit 212 based on the device identification information, and the operation identification unit 214 stores judgment value information indicating the calculated judgment value in the operation identification information storage unit 233 in association with the operation identification information acquired from the air conditioners 1A and 1B (step S203).

Next, the image acquisition unit 211 determines whether or not gesture image information, user information, and device identification information have been acquired from the air conditioners 1A and 1B (step S204). If the image acquisition unit 211 determines that gesture image information, user information, and device identification information have not been acquired from the air conditioners 1A and 1B (step S204: No), the processing of step S214 described below is executed. On the other hand, it is assumed that the image acquisition unit 211 has determined that gesture image information, user information, and device identification information have been acquired from the air conditioners 1A and 1B (step S204: Yes). In this case, the selection unit 212 selects a neural network corresponding to the device identification information notified from the image acquisition unit 211 from among the neural networks stored in the NN storage unit 231 (step S205). Next, the NN calculation unit 213 calculates a judgment value from the gesture image information and the user identification information stored in the image storage unit 232 using the neural network selected by the selection unit 212 based on the device identification information (step S206). Next, the operation identification unit 214 judges whether or not the operation identification information has been successfully identified based on the calculated judgment value (step S207). Here, if the operation identification unit 214 judges that the identification of the operation identification information has failed (step S207: No), the selection unit 212 judges whether or not a selectable neural network exists among the neural networks corresponding to the other air conditioners 1A and 1B different from the selected neural network (step S208). Here, the "selectable neural network" refers to a neural network corresponding to another air conditioner of the same type as the air conditioners 1A and 1B corresponding to the neural network that failed to identify the operation identification information, and corresponds to a neural network with no history selected by the selection unit 212 after the identification of the operation device identification information failed in step S207. Then, when the selection unit 212 determines that there is no selectable neural network (step S208: No), the process of identifying the operation identification information ends, and the process of step S214 described below is executed. On the other hand, when the selection unit 212 determines that there is a selectable neural network (step S208: Yes), it newly selects a neural network corresponding to the other air conditioners 1A, 1B that is different from the previously selected neural network (step S209). Thereafter, the NN calculation unit 213 uses the newly selected neural network to newly calculate a judgment value from the gesture image information and user information acquired from the air conditioners 1A, 1B (step S210). Next, the process of step S207 is executed again.

On the other hand, if the operation identification unit 214 determines in step S207 that the operation identification information has been successfully identified based on the calculated determination value (step S207: Yes), the operation identification information transmission unit 215 transmits the identified operation identification information to the air conditioners 1A and 1B (step S211). After that, the NN update unit 216 and the NN calculation unit 213 execute a weighting coefficient determination process (step S212).

Here, the weighting coefficient determination process executed by the cloud server 2 according to the present embodiment will be described in detail with reference to FIG. 15. First, the NN calculation unit 213 executes a calculation for repeating a convolution operation using a plurality of convolution filters to which the weighting coefficients selected by the selection unit 212 are set and a pooling process for a preset number of times for the gesture image information stored in the image storage unit 131 (step S301). As a result, the NN calculation unit 213 calculates a feature value corresponding to the gesture image information stored in the image storage unit 131. Next, the NN calculation unit 213 calculates a judgment value by executing a calculation of a part corresponding to the fully connected layer and the judgment layer to which the weighting coefficients selected by the selection unit 212 are set for the calculated feature value (step S302). Next, the NN update unit 216 calculates the error between the calculated judgment value and the median value of the range of judgment values indicated by the judgment value information stored in the operation identification information storage unit 233 (step S303). The NN update unit 216 then determines new weight coefficients for the neural network using backpropagation based on the calculated error (step S304).

Returning to FIG. 14, next, the NN update unit 216 updates the weight coefficient information of the neural network selected by the selection unit 212 stored in the NN storage unit 231 with the weight coefficient information indicating the weight coefficient determined by the weight coefficient determination process (step S213). Next, the NN update unit 216 determines whether or not a preset NN update time has arrived based on the time information measured by the timer unit 205 (step S214). If the NN update unit 216 determines that the NN update time has not yet arrived (step S214: No), the process of step S201 is executed again. On the other hand, if the NN update unit 216 determines that the NN update time has arrived (step S214: Yes), as shown in FIG. 16, the NN update unit 216 compares the amount of calculation in the neural networks of the air conditioners 1A and 1B stored in the NN storage unit 231 (step S215). After that, the NN update unit 216 identifies the neural network with the least amount of calculation (step S216). Next, the NN update unit 216 updates all of the neural networks corresponding to the air conditioners 1A and 1B to the identified neural network (step S217). Specifically, the NN update unit 216 updates all of the weighting coefficient information of the neural networks corresponding to the air conditioners 1A and 1B to the weighting coefficient information of the identified neural network.

Next, the operation identification information acquisition unit 217 determines whether or not the operation identification information and the device identification information have been acquired from the air conditioners 1A and 1B (step S218). Here, if the operation identification information acquisition unit 217 determines that the operation identification information and the device identification information have not been acquired from the air conditioners 1A and 1B (step S218: No), the processing of the above-mentioned step S201 is executed. On the other hand, it is assumed that the operation identification information acquisition unit 217 has determined that the operation identification information and the device identification information have been acquired from the air conditioners 1A and 1B (step S218: Yes). In this case, the selection unit 212 selects all of the neural networks stored in the NN storage unit 231 (step S219). Thereafter, the NN update unit 216 and the NN calculation unit 213 execute the weight coefficient determination processing for all of the neural networks (step S220). Next, the NN update unit 216 updates the weight coefficient information of the corresponding neural network stored in the NN storage unit 231 with the weight coefficient information indicating the weight coefficient of each neural network determined by the weight coefficient determination process (step S221). Then, the process of step S201 is executed again.

As described above, in the control system according to the present embodiment, the NN update unit 216 updates the weight coefficient information of the neural network selected by the selection unit 212 based on the gesture image information and the operation identification information corresponding to the judgment value calculated from the gesture image information, each time the image acquisition unit 111 acquires gesture image information. In addition, the NN update unit 216 identifies the neural network with the least amount of calculation among the neural networks corresponding to the air conditioners 1A and 1B stored in the NN storage unit 231 each time a preset NN update time arrives, and updates all of the weight coefficient information of the neural networks corresponding to the air conditioners 1A and 1B to the weight coefficient information of the identified neural network. As a result, the weight coefficients of the neural networks corresponding to the air conditioners 1A and 1B are periodically updated to the weight coefficient of the neural network with the least amount of calculation, i.e., the most efficient calculation, so that it is possible to shorten the time required to identify the operation content corresponding to the gesture made by the user and improve the accuracy of identifying the operation content.

By the way, with the development of deep learning technology, neural networks are being used for voice recognition, image recognition, etc., and are also being used in devices equipped with pattern recognition functions, automatic estimation functions, etc. In addition, with the recent emergence of CPUs dedicated to neural network calculations, it has become possible to mount the calculation function of the neural network on the edge side, i.e., on the device, and it has become possible to build a system that executes neural network calculations on the device. In addition, in the past, it was common for users to operate devices by pressing operation buttons on an operation device connected to the device, but in recent years, there has been an increase in devices that can be operated by voice input and gesture operation by users using the above-mentioned pattern recognition function and automatic estimation function. The control system according to this embodiment is this type of system, in which the air conditioners 1A and 1B transmit gesture image information obtained by photographing the user with the imaging device 181 mounted thereon to the cloud server 2, and the cloud server 2 calculates a judgment value from the gesture image information using a neural network, and specifies the operation identification information based on the calculated judgment value. Systems are being provided in which the identified operation identification information is transmitted to the air conditioners 1A, 1B to operate the air conditioners 1A, 1B. In such control systems, there is a demand for improving the learning effect of the neural network used by the cloud server 2 to improve the accuracy of identifying the operation identification information. In this embodiment, the control system employs a so-called neural network battle method in which the performance of the neural networks corresponding to the air conditioners 1A, 1B, which have the same gesture operation content, is compared, specifically, the amount of calculation therein, and the one with the least amount of calculation and the highest efficiency in calculating the judgment value is identified, and the other neural network is updated to it, thereby improving the learning effect of the neural networks.

In addition, when the operation identification unit 214 fails to identify the operation identification information associated with the judgment value calculated by the NN calculation unit 213, the selection unit 212 according to this embodiment newly selects a neural network corresponding to the other air conditioners 1A and 1B. Then, it is assumed that the operation identification unit 214 succeeds in identifying the operation identification information associated with the judgment value newly calculated from the gesture image information using the neural network newly selected by the NN calculation unit 213. In this case, the NN update unit 216 updates the weighting coefficient information of the neural network previously selected by the selection unit 212 based on the gesture image information and the operation identification information corresponding to the newly calculated judgment value. This allows the neural network to learn, that is, the weighting coefficient information to be updated, using the gesture image information for which the operation identification unit 214 failed to identify the operation identification information as teacher data, thereby promoting the optimization of the weighting coefficients of the neural network.

Furthermore, when the operation receiving unit 113 of the air conditioners 1A, 1B receives the above-mentioned correction operation, the NN update unit 216 according to this embodiment updates all of the weight coefficient information of the neural network of each of the air conditioners 1A, 1B based on the gesture image information and the operation identification information after correction by the correction operation. This makes it possible to train the neural network using the gesture image information that requires correction of the correspondence with the operation identification information as teacher data, that is, to update the weight coefficient information, thereby facilitating the optimization of the weight coefficients of the neural network.

Furthermore, when the operation reception unit 113 receives the above-mentioned new registration operation, the selection unit 212 according to this embodiment selects one neural network from the neural networks for the air conditioners 1A and 1B already stored in the NN memory unit 231. Then, the NN update unit 216 stores the selected one neural network in the NN memory unit 231 as the neural network for the newly registered air conditioner. This makes it possible to quickly optimize the weighting coefficient of the neural network for the newly registered air conditioner, thereby improving the accuracy of identifying operation identification information based on the user's gesture image information from the beginning of use of the air conditioner.

Furthermore, in this embodiment, the image correction unit 112 of the air conditioners 1A, 1B corrects the gesture image information acquired by the image acquisition unit 111 based on the imaging conditions of the imaging device 181. Then, the NN calculation unit 213 of the cloud server 2 calculates a judgment value corresponding to the corrected gesture image information from the gesture image information corrected by the image correction unit 112, using the neural network selected by the selection unit 212. This can improve the accuracy of the operation identification unit 214 in identifying the operation identification information.

Furthermore, when the operation reception unit 113 of the air conditioners 1A, 1B receives the above-mentioned gesture registration operation, the NN update unit 216 according to this embodiment updates the neural network based on the image information corresponding to at least one type of gesture by the user acquired by the image acquisition unit 111 and the operation identification information that identifies the content of the operation accepted by the operation reception unit 113. This allows the user to freely change the correspondence between the gesture and the content of the operation, thereby improving user convenience.

Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments. For example, the NN calculation unit 213 may use a neural network selected by the selection unit 212 based on the device identification information to calculate a judgment value only from the gesture image information stored in the image storage unit 232. In this case, there is no need to identify the user in the air conditioners 1A and 1B, which has the advantage of reducing the processing load in the air conditioners 1A and 1B.

In the embodiment, an example has been described in which all calculations of the convolutional layer, pooling layer, fully connected layer, and judgment layer are performed in the cloud server 2. However, this is not limited to the above, and for example, only the calculations of the convolutional layer and pooling layer may be performed in the cloud server 2, and the calculations of the fully connected layer and judgment layer may be performed in the air conditioners 1A and 1B. In this case, the NN update unit 216 of the cloud server 2 may, for example, identify the neural network with the least amount of calculations in at least one of the convolutional layer and pooling layer from among the neural networks corresponding to the air conditioners 1A and 1B stored in the NN storage unit 231 each time the NN update time arrives.

This configuration can promote optimization of the weight coefficients of the neural network even if the cloud server 2 is configured to perform only the calculations of the convolution layer and pooling layer of the neural network.

In the embodiment, the imaging device 181 may include a thermal image sensor. In this case, the user information storage unit 132 may store heat dissipation area information indicating the position of the area of the user's surface where heat dissipation is high, which is calculated from the temperature distribution of the user's surface shown in the thermal image of the user obtained by the imaging device 181, in association with the user identification information. The user identification unit 117 may identify the area of the user's surface where heat dissipation is high from the temperature distribution of the user's surface shown in the thermal image included in the image information acquired by the image acquisition unit 111, and identify the user of the air conditioner 1A, 1B based on the heat dissipation area information stored in the user information storage unit 132 and the position of the identified area of high heat dissipation. In the embodiment, the imaging device 181 may be, for example, a camera mounted on a mobile terminal or the like. In this case, a well-known image processing technique may be adopted as a method for extracting the image of the user from the image captured by the imaging device 181.

In the embodiment, an example has been described in which in the processing of steps S212 and S213, the teacher data is set to the median value of the range of judgment values indicated by the judgment value information stored in the operation identification information storage unit 233, and the neural network is made to perform so-called supervised learning. However, this is not limited to this, and so-called unsupervised learning may be adopted, for example, in which the NN calculation unit 213 clusters judgment values calculated in the past and automatically generates teacher data for the judgment values.

In this embodiment, air conditioners 1A and 1B may be manufactured by different manufacturers.

In the embodiment, an example has been described in which the neural network used by the NN calculation unit 213 is a convolutional neural network, but this is not limiting, and the neural network used by the NN calculation unit 213 may be a neural network of another structure, such as a recurrent neural network.

The various functions of the air conditioners 1A, 1B and cloud server 2 according to the present disclosure may be realized by software, firmware, or a combination of software and firmware. In this case, the software or firmware may be written as a program, and the program may be stored and distributed on a computer-readable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc), or an MO (Magneto-Optical Disc), and the program may be read and installed on a computer to configure a computer capable of realizing each of the above-mentioned functions. In addition, when each function is realized by sharing the work between an OS (Operating System) and an application, or by cooperation between an OS and an application, only the parts other than the OS may be stored on the recording medium.

Furthermore, each program can be superimposed on a carrier wave and distributed over a network. For example, the program can be posted on a bulletin board (BBS, Bulletin Board System) on the network and distributed over the network. These programs can then be started and run under the control of the OS in the same way as other application programs, thereby enabling the above-mentioned processing to be performed.

This disclosure is suitable for a control system capable of controlling an air conditioner in response to gestures made by a user.

1A, 1B Air conditioner, 2 Cloud server, 6A, 6B Operation device, 81 Data circuit termination device, 82 Router, 100 Control unit, 101, 201 CPU, 102, 202 Main memory unit, 103, 203 Auxiliary memory unit, 104 Neuro engine, 105, 205 Timer unit, 106 Local communication unit, 107 Wireless module, 108 Imaging interface, 109, 209 Bus, 111, 211 Image acquisition unit, 112 Image correction unit, 113 Operation reception unit, 114 Device control unit, 115, 215 Operation identification information transmission unit, 116 Operation mode setting unit, 117 User identification unit, 118 Image transmission unit, 119 Device setting unit, 120 Registration request unit, 131, 232 Image storage unit, 132 User information storage unit, 133 Operation mode storage unit, 134 device setting storage unit, 135 operation information storage unit, 181 imaging device, 212 selection unit, 213 NN calculation unit, 214 operation identification unit, 216 NN update unit, 217 operation identification information acquisition unit, 231 NN storage unit, 233 operation identification information storage unit, H building, L10 input layer, L20 hidden layer, L30 output layer, NW1 wide area network, NW2 local network

Claims (11)

a neural network storage unit that stores a neural network for identifying one type of operation content corresponding to a gesture image of the same type from gesture image information showing a gesture of the same type of user corresponding to each of a plurality of devices;
an imaging device provided in each of the plurality of devices and configured to image a user of each of the plurality of devices;
an image acquisition unit that acquires, from the imaging device, gesture image information indicating a gesture of the user captured by the imaging device;
a selection unit that selects, from the neural networks stored in the neural network storage unit, a neural network corresponding to an apparatus provided with an imaging device that has generated the gesture image information acquired by the image acquisition unit;
a neural network calculation unit that calculates a judgment value corresponding to the gesture image information acquired by the image acquisition unit, using the neural network selected by the selection unit;
an operation identification unit that identifies operation identification information that identifies an operation on the device associated with the determination value calculated by the neural network calculation unit;
a device control unit that controls a device provided with an imaging device that generates the gesture image information, based on the operation identification information identified by the operation identification unit;
a neural network update unit that updates the neural network selected by the selection unit based on the gesture image information and operation identification information corresponding to a determination value calculated from the image information every time the image acquisition unit acquires the gesture image information, and that identifies a neural network with the least amount of calculation from among the neural networks corresponding to each of the plurality of devices stored in the neural network storage unit every time a preset neural network update time arrives, and updates all of the neural networks corresponding to each of the plurality of devices to the identified neural network.
Control system. the selection unit, when the operation identification unit fails to identify the operation identification information for the device associated with the determination value, selects a new neural network corresponding to another device;
the neural network calculation unit recalculates a judgment value corresponding to the gesture image information from the gesture image information by using a newly selected neural network;
The operation identification unit identifies operation identification information associated with the determination value recalculated by the neural network calculation unit,
the neural network update unit updates a neural network corresponding to an apparatus provided with an imaging device that has generated the gesture image information acquired by the image acquisition unit, based on the gesture image information and the operation identification information corresponding to the recalculated determination value.
The control system of claim 1 . an operation receiving unit that receives the content of the operation performed by the user on the operation device,
when the operation receiving unit receives a correction operation for correcting an operation content corresponding to the gesture image acquired by the image acquiring unit, the neural network updating unit updates all of the neural networks of each of the plurality of devices based on the gesture image information and the operation identification information after the correction operation.
A control system according to claim 1 or 2. The system further includes an operation receiving unit that receives a new registration operation for the user to newly register a neural network corresponding to a new device to the operation device,
when the operation reception unit receives the new registration operation, the selection unit selects at least one neural network from among the neural networks for the multiple devices already stored in the neural network storage unit;
the neural network update unit stores the at least one neural network selected by the selection unit in the neural network storage unit as a neural network for each of the at least one device to be newly registered;
A control system according to any one of claims 1 to 3. the neural network is a convolutional neural network;
the neural network update unit, every time a time comes to update the neural network, identifies a neural network having the smallest amount of calculation in at least one of a convolution layer and a pooling layer from among the neural networks corresponding to each of the plurality of devices stored in the neural network storage unit, and updates all of the neural networks corresponding to each of the plurality of devices with the identified neural network;
A control system according to any one of claims 1 to 4. an image correction unit that corrects the gesture image information acquired by the image acquisition unit based on an image capture condition including at least one of a performance of the image capture device, an installation position of the image capture device, an ambient brightness of the image capture device, and a background image of a gesture image of a user captured by the image capture device;
the neural network calculation unit calculates a judgment value corresponding to the gesture image information after the correction by the image correction unit, using the neural network selected by the selection unit, from the gesture image information after the correction by the image correction unit;
A control system according to any one of claims 1 to 5. an operation receiving unit that receives the content of the operation performed by the user on the operation device,
When the operation reception unit receives a gesture registration operation for registering image information corresponding to a new gesture, the neural network update unit updates the neural network based on image information corresponding to at least one type of gesture of the user acquired by the image acquisition unit and operation identification information identifying the content of the operation accepted by the operation reception unit.
A control system according to any one of claims 1 to 6. the imaging device includes a thermal image sensor;
A user identification unit that identifies a user based on a thermal image of the user captured by the imaging device,
The neural network calculation unit calculates a judgment value corresponding to the gesture image information from the gesture image information acquired by the image acquisition unit and user identification information for identifying the user identified by the user identification unit, using the neural network selected by the selection unit.
A control system according to any one of claims 1 to 7. a neural network storage unit that stores a neural network for identifying one type of operation content corresponding to a gesture image of the same type from gesture image information showing a gesture of the same type of user corresponding to each of a plurality of devices;
an imaging device provided in each of the plurality of devices and configured to image a user of each of the plurality of devices;
an image acquisition unit that acquires, from the imaging device, gesture image information indicating a gesture of the user captured by the imaging device;
a selection unit that selects, from the neural networks stored in the neural network storage unit, a neural network corresponding to an apparatus provided with an imaging device that has generated the gesture image information acquired by the image acquisition unit;
a neural network calculation unit that calculates a judgment value corresponding to the gesture image information acquired by the image acquisition unit, using the neural network selected by the selection unit;
an operation identification unit that identifies operation identification information that identifies an operation on the device associated with the determination value calculated by the neural network calculation unit;
a device control unit that controls a device provided with an imaging device that generates the gesture image information for identifying the operation identification information based on the operation identification information identified by the operation identification unit;
a neural network update unit that updates the neural network selected by the selection unit based on the gesture image information and operation identification information corresponding to a determination value calculated from the image information every time the image acquisition unit acquires the gesture image information, and that identifies a neural network with the least amount of calculation from among the neural networks corresponding to each of the plurality of devices stored in the neural network storage unit every time a preset neural network update time arrives, and updates all of the neural networks corresponding to each of the plurality of devices to the identified neural network.
Learning system. a step in which the control system acquires, from an imaging device provided in each of a plurality of devices and configured to image a user of each of the plurality of devices, gesture image information indicating a gesture of the user captured by the imaging device;
a step in which the control system selects, from among neural networks stored in a neural network storage unit that stores neural networks for identifying one type of operation content corresponding to a gesture image of the same type of user from gesture image information showing the same type of gesture of the user, the neural network corresponding to each of the plurality of devices, a neural network corresponding to a device provided with an imaging device that generated the acquired gesture image information;
A step of the control system calculating a judgment value corresponding to the acquired gesture image information by using the selected neural network from the gesture image information;
A step of the control system specifying operation identification information for identifying an operation on the device associated with the calculated determination value;
a step of controlling an apparatus provided with an imaging device that generates the gesture image information, based on the identified operation identification information by the control system;
updating the selected neural network based on the gesture image information and operation identification information corresponding to a determination value calculated from the image information, every time the control system acquires the gesture image information;
and a step of the control system identifying a neural network with the least amount of calculation from among the neural networks corresponding to the plurality of devices stored in the neural network storage unit, and updating all of the neural networks corresponding to the plurality of devices to the identified neural network, each time a preset neural network update time arrives.
Device control method. Computer,
a neural network storage unit that stores a neural network for identifying one type of operation content corresponding to a gesture image of the same type from gesture image information showing a gesture of the same type of user corresponding to each of a plurality of devices;
an imaging device provided in each of the plurality of devices and configured to image a user of each of the plurality of devices;
an image acquisition unit that acquires, from the imaging device, gesture image information indicating a gesture of the user captured by the imaging device;
a selection unit that selects, from the neural networks stored in the neural network storage unit, a neural network corresponding to an apparatus provided with an imaging device that has generated the gesture image information acquired by the image acquisition unit;
a neural network calculation unit that calculates a judgment value corresponding to the gesture image information acquired by the image acquisition unit, using the neural network selected by the selection unit;
an operation identification unit that identifies operation identification information that identifies an operation on the device associated with the determination value calculated by the neural network calculation unit;
a device control unit that controls a device provided with an imaging device that generates the gesture image information, based on the operation identification information identified by the operation identification unit;
a neural network update unit that updates the neural network selected by the selection unit based on the gesture image information and operation identification information corresponding to a determination value calculated from the image information, every time the image acquisition unit acquires the gesture image information, and that identifies a neural network requiring the least amount of calculation from among the neural networks corresponding to each of the plurality of devices stored in the neural network storage unit, and updates all of the neural networks corresponding to each of the plurality of devices to the identified neural network, every time a preset neural network update time arrives;
A program to function as a

Images