JP6764105B2 - Seat information processing system, seat information acquisition device and seat information acquisition program - Google Patents

Description

The present invention relates to a seat information processing system, a seat information acquisition device, and a seat information acquisition program, and processes, for example, seat information in a facility that involves seating in a seat (seat) in a restaurant such as a restaurant or cafe or a library. Applicable to the system.

Conventionally, there is known a technique of recognizing the availability of seats in a restaurant or the like, generating and providing availability information (for example, providing it on the Internet).

As a conventional technique for providing free information, for example, there is a technique of Patent Document 1.

Patent Document 1 is a technique for providing information indicating a place where a user is absent as free information in a place where the user can use the facility. In the technique described in Patent Document 1, the position information of each seat is registered in advance in a restaurant or the like, and a person (customer) at each seat position is used by a motion sensor (a motion sensor using a temperature detection sensor). ) Is recognized, and vacant information is generated by vacant seats that cannot be recognized by people (customers).

Japanese Unexamined Patent Publication No. 2016-177775

However, in the conventional techniques such as the description technique of Patent Document 1, it is necessary to input information regarding the details of the layout of the seats of the restaurant on the premise of acquiring the vacant information, so that the details of the layout are changed every time the layout is changed. There is a problem that it is not convenient because it requires input of information.

Further, in the technique described in Patent Document 1, there is a problem that the introduction cost is high because a dedicated sensor (human sensor) is required to recognize the seating status of each seat.

In view of the above problems, a seat information processing system, a seat information acquisition device, and a seat information acquisition program that realize highly convenient processing of seat information at low cost are desired.

The first invention is based on (1) an imaging means for imaging the inside of a facility in which one or more seats are arranged, and (2) a frame for each time series imaged by the imaging means, at least in the facility. It is provided with a seat information acquisition means for recognizing the number of in-use seats used in the facility and acquiring seat information indicating the usage status of the seats in the facility based on the recognized number of in-use seats. (3) The seat information The acquisition means (3-1) for each frame , converts the pixel value for each pixel into a pixel vector represented by a vector value composed of RGB information and acquires it, and further converts each acquired pixel vector into a unit vector. Obtain the dot product value between the pixel vector processing means for acquiring the normalized normalized pixel vector and the frame (3-2) adjacent frames in which the time series is adjacent to obtain the dot product value of the normalized pixel vector corresponding to each pixel. Means, (3-3) Foreground area recognition means for recognizing the foreground area in each frame based on the inter-frame inner product value between frames acquired by the inner product value acquisition means, and (3-4) Foreground area recognition. Based on the foreground area recognized by the means, the person object recognition process for recognizing the person object reflected in each frame is performed, and based on the result of the person object recognition process, the number of seats in use in the facility is used. It is characterized by having a recognition means for recognizing.

The second aspect of the present invention is the seat information acquisition device, based on (1) frames for each time series in which one or a plurality of seats are arranged in a facility, and at least the in-use seats used in the facility. It has a seat information acquisition means that recognizes the number and acquires seat information indicating the usage status of the seats in the facility based on the recognized number of seats in use, and (2) the seat information acquisition means is (3- 1) For each frame, the pixel value for each pixel is converted into a pixel vector represented by a vector value composed of RGB information and acquired, and each acquired pixel vector is normalized to a unit vector. (3-2) An inner product value acquisition means for acquiring the inter-frame inner product value of the normalized pixel vector corresponding to each pixel between frames adjacent to each other in the time series, and (3-3). ) Foreground area recognition means for recognizing the foreground area in each frame based on the inter-frame inner product value between frames acquired by the inner product value acquisition means, and (3-4) foreground area recognized by the foreground area recognition means. Based on this, a person object recognition process for recognizing the person object reflected in each frame is performed, and based on the result of the person object recognition process, a recognition means for recognizing the number of seats in use in the facility is provided. It is characterized by having.

The third seat information acquisition program of the present invention uses a computer at least in the facility based on (1) time-series frames in which one or a plurality of seats are arranged in the facility. It is made to function as a seat information acquisition means for recognizing the number of seats in use and acquiring seat information indicating the usage status of the seats in the facility based on the recognized number of seats in use. (2) The seat information acquisition means (2-1) For each frame, the pixel value for each pixel is converted into a pixel vector represented by a vector value composed of RGB information and acquired, and each acquired pixel vector is normalized to a unit vector. A pixel vector processing means for acquiring a converted pixel vector, and (2-2) an inner product value acquiring means for acquiring an inter-frame inner product value of a normalized pixel vector corresponding to each pixel between frames adjacent to each other in a time series. 2-3) The foreground area recognition means for recognizing the foreground area in each frame based on the inter-frame inner product value between the frames acquired by the inner product value acquisition means, and (2-4) the foreground area recognition means recognized. Based on the foreground area, the person object recognition process for recognizing the person object reflected in each frame is performed, and based on the result of the person object recognition process, the recognition of the number of seats in use in the facility is recognized. It is characterized by having means.

According to the present invention, highly convenient processing of seat information can be realized at low cost.

It is a block diagram which showed the whole structure of the seat information processing system which concerns on embodiment. It is a block diagram which showed the functional structure of the local computer which concerns on embodiment. It is a block diagram which showed the functional configuration of the DB server which concerns on embodiment. It is explanatory drawing which showed the structural example of the store information DB which concerns on embodiment. It is a block diagram which showed the functional configuration of the Web application server which concerns on embodiment. It is a block diagram which showed the functional configuration of the client terminal which concerns on embodiment. It is a top view of the inside of the store which concerns on embodiment. It is a figure (perspective view) when the inside of the store which concerns on embodiment is seen from a camera. This is an example (No. 1) of an image captured by the camera according to the embodiment. This is an example (No. 2) of an image captured by the camera according to the embodiment. It is explanatory drawing which showed the example of the foreground area image acquired by the object recognition part which concerns on embodiment. It is explanatory drawing which showed the example of the circumscribing rectangle which the object recognition part which concerns on embodiment cut out from an image. It is a flowchart which showed the flow of the moving body recognition processing by the object recognition part which concerns on embodiment. It is explanatory drawing which showed the configuration example of the store selection screen (Web screen) provided by the Web application server which concerns on embodiment. It is explanatory drawing which showed the configuration example of the store information provision screen (Web screen) provided by the Web application server which concerns on embodiment.

(A) Main Embodiments Hereinafter, one embodiment of the seat information processing system, the seat information acquisition device, and the seat information acquisition program according to the present invention will be described in detail with reference to the drawings.

(A-1) Configuration of the Embodiment FIG. 1 is a block diagram showing an overall configuration of the seat information processing system 1 of the embodiment.

First, the overall outline of the seat information processing system 1 will be described.

The seat information processing system 1 is a system that processes information related to seats of one or a plurality of facilities such as restaurants (hereinafter, referred to as "seat information").

In this embodiment, the seat information processing system 1 will be described as a system that processes seat information of N stores (N is an arbitrary integer) (first store to Nth store).

In the seat information processing system 1, each store (first to Nth stores) processes a camera 10 (10-1 to 10-N) for photographing the inside of the store, seat information in the store, and the like. A local computer 20 (20-1 to 20-N) is arranged.

Further, in the seat information processing system 1, information including seat information of each store is held from the local computer 20 (20-1 to 20-N) of each store, and the client terminal 50 possessed by the recipient of the seat information It has a Web application server 30 as a seat information providing device to be provided. Further, in the seat information processing system 1, a DB server 40 is arranged as a means for managing various data in the Web application server 30. The number of devices (computers) constituting the seat information providing device of the present invention and the connection configuration between the devices are not limited. In this embodiment, as shown in FIG. 1, the Web application server 30 and the DB server 40 will be described as being installed in the data center.

As described above, the client terminal 50 is a terminal possessed by the recipient of the store seat information, and connects to the Web application server 30 according to the operation of the information provider to acquire the seat information of each store. Performs the process of displaying.

It is assumed that each device in the seat information processing system 1 can communicate with each other. In the example of this embodiment, each device in the seat information processing system 1 will be described as having a configuration in which they can be connected to each other via the Internet.

Next, the configuration of each device constituting the seat information processing system 1 will be described.

First, the configurations of the cameras 10 (10-1 to 10-N) and the local computers 20 (20-1 to 20-N) installed in each store will be described. The detailed configuration (for example, hardware configuration) of the devices installed in each store may be different, but functionally, they can be shown as the same configuration as shown in FIG.

The camera 10 installed in each store is a camera that captures an image in a predetermined area inside the store (in the area including a seat (for example, a seat such as a chair) for which seat information is processed). As the camera 10, various digital cameras can be applied. The camera 10 supplies an image of the inside of the store to the local computer 20 in the same store.

The local computer 20 is a device that processes seat information in the installed store. Specifically, the local computer 20 acquires seat information in the store based on the image supplied from the camera 10, and supplies the acquired seat information to the Web application server 30.

In this embodiment, the seat information acquired by the local computer 20 includes at least seats that are not used and are vacant (hereinafter, referred to as “processed seats”) among the seats to be processed (hereinafter referred to as “processed seats”) at the store where the installation is installed. , "Vacant seats") (hereinafter referred to as "vacant seats") is assumed to contain recognizable information.

Specifically, in the example of this embodiment, the seat information acquired by the local computer 20 includes the number of seats to be processed at the installed store (hereinafter, also referred to as “number of installed seats”) and the seats to be processed. Contains information on the number of seats (hereinafter, also referred to as "in-use seats") that a person (for example, a customer of a store) is in use (for example, in a seat) (hereinafter, also referred to as "in-use seats"). It is assumed that there is. The local computer 20 may acquire the number of vacant seats itself as seat information. Further, when the information on the number of installed seats in each installed store is held on the seat information providing device (Web application server 30 and DB server 40), the local computer 20 simply uses only the number of seats in use as seat information. You may try to get it.

The internal configuration of the local computer 20 can be functionally shown as shown in FIG. The local computer 20 is constructed by, for example, installing a program (including the "seat information acquisition program" according to the embodiment) on a computer having a memory and a processor (for example, a PC, a single board computer, a microcomputer, etc.). You may.

As shown in FIG. 2, the local computer 20 has a video acquisition unit 21, an object recognition unit 22, a seat information processing unit 23, and a data recording unit 24.

The data recording unit 24 is a data recording means for recording various data used in each process in the local computer 20.

The image acquisition unit 21 acquires an image (frame) captured by the camera 10 every predetermined period (hereinafter, this period is referred to as “T1”) and supplies the image (frame) to the object recognition unit 22. The image acquisition unit 21 may control the camera 10 for each period T1 to take an image, or from a series of images (moving images) continuously captured and supplied by the camera 10 at intervals of the period T1. The image may be acquired. The interval of the period T1 is not limited, but may be, for example, 0.2 to 1 second.

The object recognition unit 22 analyzes the images for each time series (images for each period T1) supplied from the image acquisition unit 21, and captures the human being reflected in the installation store (in the area captured by the camera 10). It is detected as an object (hereinafter, also referred to as "human object"), and a user (hereinafter, referred to as "seat user") who is further using (for example, sitting) a seat to be processed is detected from the detected human object. Perform the processing to be performed. In this embodiment, the "seat" does not necessarily have to be equipped with a seat (chair), and is a place where people can eat and drink while standing (the facility can be used) (so-called "standing seat"). It may be.

The specific procedure of the object recognition process by the object recognition unit 22 will be described later.

The seat information processing unit 23 uses the recognition result of the object recognition unit 22 to perform a process of acquiring seat information in the installed store. In this embodiment, the seat information processing unit 23 acquires at least information on the number of installed seats and the number of active seats in the installed store as seat information.

The seat information processing unit 23 of this embodiment reads and acquires data on the number of installed seats from the data set in advance in the data recording unit 24. The method of accepting data input of the number of installed seats in the data recording unit 24 in the local computer 20 is not limited. For example, the local computer 20 may accept data input of the number of installed seats by an input means (for example, a device such as a display, a keyboard, a mouse, etc.) (not shown), or accept data input from an external device by communication. You may.

The seat information processing unit 23 acquires the number of seats in use based on the number of seat users currently recognized by the object recognition unit 22. That is. The seat information processing unit 23 performs a process of acquiring the number of seat users currently recognized by the object recognition unit 22 as the number of seats in use.

Then, the seat information processing unit 23 acquires information including the acquired number of installed seats and the number of seats in use as seat information, and transmits the information to the Web application server 30.

The timing at which the seat information processing unit 23 transmits the seat information to the Web application server 30 is not limited. For example, the seat information processing unit 23 may transmit the seat information (latest acquired seat information) at the timing requested by the Web application server 30, or may send the seat information at predetermined intervals. It may be transmitted, or it may be transmitted every time the seat information changes.

Next, the internal configuration of each device (Web application server 30, DB server 40) constituting the seat information providing device will be described.

FIG. 3 is a block diagram showing the internal configuration of the DB server 40.

The DB server 40 is a device that provides various data required for processing of the Web application server 30. Specifically, the DB server 40 holds a database of information about each store (hereinafter, referred to as "store information DB"), and performs data processing (for example, updating or reading data) in response to a request from the Web application server 30. Processing) is performed.

The DB server 40 has a DB processing unit 41 and a data recording unit 42.

The internal configuration of the DB server 40 can be functionally shown as shown in FIG. The DB server 40 can be constructed using, for example, a computer having a memory and a processor (for example, a PC, a workstation, etc.).

As shown in FIG. 3, the DB server 40 has a DB processing unit 41 and a data recording unit 42.

The data recording unit 42 is a data recording means for recording various data used in each process in the DB server 40. As shown in FIG. 3, it is assumed that at least the store information DB 421 is recorded in the data recording unit 42.

The DB processing unit 41 processes data in the database (store information DB 421) recorded in the data recording unit 42. The DB processing unit 41 can be constructed by using a DBMS (Database Management System) such as ORACLE (registered trademark) or MySQL (registered trademark), for example. Therefore, the store information DB 421 recorded in the data recording unit 42 needs to have a data structure corresponding to that in the DB processing unit 41.

Next, the data structure of the store information DB 421 will be described with reference to FIG.

FIG. 4 is an explanatory diagram showing the data structure of the store information DB 421. In FIG. 4, the data structure of the store information DB 421 is shown in a table format. In FIG. 4, information on one store is illustrated in one line.

As shown in FIG. 4, the store ID, store name, number of installed seats, number of seats in use, access information, and store information are registered in the store information DB 421 for each store.

The item of "store ID" is an item that functions as a key for identifying each store in the store information DB 421. As shown in FIG. 4, in this embodiment, it is assumed that IDs 1 to N are assigned to the first store to the Nth store, respectively.

The item of "store name" is an item for recording the name of the store. In FIG. 4, the store name of the first store (store ID = 1) is "AAA", the store name of the second store (store ID = 2) is "BBB", and the store name of the third store (store ID = 3). The store name of the store is "CCC", ..., The store name of the Nth store (store ID = N) is "XXX cafeteria".

The information of "the number of installed seats" and "the number of seats in use" is an item for recording the seat information of each store.

The access information is information for accessing the local computer 20 of each store. Examples of the access information include an IP address for accessing the local computer 20, a host name, an FQDN (Fully Qualified Domain Name), and the like. In FIG. 4, the access information for accessing the local computers 20-1 to 20-N of each store is shown as host1 to hostN, respectively.

The item of "store information" is an item representing information of each store (for example, information presented to the information recipient who possesses the client terminal 50). FIG. 4 shows an example of recording the address, business hours, and homepage URL information as the store information of each store. In the store information DB 421, the number and combinations of information items to be recorded as store information are not limited.

The store information DB 421 is configured as described above. Then, the DB processing unit 41 performs data processing (for example, data update / reading processing) of the store information DB 421 configured as described above in response to the request (command) of the Web application server 30.

FIG. 5 is a block diagram showing the internal configuration of the Web application server 30.

The Web application server 30 has a Web service unit 31, a server application 32, and a data recording unit 33.

The internal configuration of the Web application server 30 can be functionally shown as shown in FIG. The Web application server 30 can be constructed using, for example, a computer having a memory and a processor (for example, a PC, a workstation, etc.).

The data recording unit 33 is a data recording means for recording various data used in each process in the Web application server 30.

The Web service unit 31 provides the client terminal 50 with information including seat information of each store as content (for example, a Web page). The Web service unit 31 provides the client terminal 50 with content according to the control of the server application 32.

As the Web service unit 31, for example, middleware of various Web servers can be applied. In this case, the server application 32 is an application (server-side program) that performs a content generation process or the like to be provided in response to a request from the outside (for example, the client terminal 50) on the middleware as the Web service unit 31. ..

The server application 32 acquires the seat information of each store (in this embodiment, the information including the number of installed seats and the number of seats in use) from the local computer 20 of each store, and the latest acquired seat information is the store of the DB server 40. Register in the information DB 421.

Further, the server application 32 receives information (data related to some or all stores) of the store information DB 421 of the DB server 40 in response to a request (operation received by the client terminal 50) received from the client terminal 50 by the Web service unit 31. ) To get. Then, the server application 32 generates a content (Web screen) using the acquired information of the store information DB 421, and performs a process of causing the Web service unit 31 to provide the request source client terminal 50.

A configuration example of the content (Web screen) provided by the Web application server 30 will be described later.

Next, the internal configuration of the client terminal 50 will be described.

FIG. 6 is an explanatory diagram showing an example of the internal configuration of the client terminal 50.

The client terminal 50 can be constructed by using various terminal devices (computers; hardware) such as a smart phone, a PC, and a tablet PC. FIG. 6 shows an example in which a smart phone is applied as the hardware of the client terminal 50.

As shown in FIG. 6, the client terminal 50 has a touch panel display 51 and a client application 52.

The touch panel display 51 is a device (a device that has a user interface function) that receives information output to the information recipient who owns the client terminal 50 and input from the information provider (for example, input using a GUI screen). Is. In this embodiment, an example of constructing the client terminal 50 using a smart phone is shown, but the client terminal 50 bears a user interface (input means from the information recipient and output means to the information provider). The type of device is not limited. For example, when constructing the client terminal 50 using a PC, a display, a mouse, and a keyboard may be used as the user interface.

The client application 52 accepts an operation from the information recipient using the touch panel display 51, transmits a request according to the operation of the information recipient to the Web application server 30, and provides content supplied from the Web application server 30. A process of displaying (information including seat information of one or a plurality of stores) on the touch panel display 51 is performed.

The client application 52 may be constructed using, for example, various Web browsers. In this embodiment, it is assumed that the client application 52 is set to a predetermined URL (URL for providing seat information of each store) on the Web application server 30 as a connection destination (connection destination URL) in the Web browser. explain.

The client application 52 may be constructed as a dedicated application without using a Web browser.

(A-2) Operation of the Embodiment Next, the operation of the seat information processing system of the embodiment having the above configuration (seat information processing method according to the embodiment) will be described.

First, the installation environment of the camera 10 in each store will be described.

FIG. 7 is an example of a plan view of the first store (store ID = 1) in the store.

Hereinafter, an example of the installation environment of the camera 10 will be described by taking the inside of the first store as an example.

As shown in FIG. 7, in the first store, a square counter table R1, eight chairs C (C1 to C8) arranged side by side on one side of the counter table R1, and a countertop R2 are arranged. Have been placed.

Then, in the first store, it is assumed that these eight chairs C1 to C8 are the seats to be processed.

Then, as shown in FIG. 7, a camera 10-1 is arranged at a corner (edge) in the first store.

FIG. 8 is a view (perspective view) of the inside of the first store (store ID = 1) when viewed from the side of the camera 10-1 installed in the store.

As described above, since the chairs C1 to C8 are the seats to be processed in the first store, the camera 10-1 is set to the position, orientation (posture), and zoom setting so that all the chairs C1 to C8 are within the field of view to be imaged. Need to be installed. In FIG. 8, the frame of the field of view A imaged by the camera 10-1 is illustrated by a alternate long and short dash line. As shown in FIG. 8, all the chairs C1 to C8 are contained in the field of view A.

Therefore, when there is no person in the first store, the image captured by the camera 10-1 is as shown in FIG. In the image shown in FIG. 9, all the chairs C1 to C8 are shown. Actually, when the user of each chair C is seated, the user's face is located slightly above the chair, so that the area where the user's face is actually located is included in the field of view A. It is desirable to install the camera 10-1 so that it fits.

Next, the process of acquiring the seat information in the installed store by the local computer 20-1 installed in the first store will be described.

As described above, the seat information processing unit 23 acquires at least information on the number of installed seats and the number of seats in use in the installed store as seat information. As described above, the seat information processing unit 23 of this embodiment reads and acquires the data of the number of installed seats from the data set in advance in the data recording unit 24. Further, the seat information processing unit 23 performs a process of acquiring the number of seat users currently recognized by the object recognition unit 22 as the number of seats in use. Then, the seat information processing unit 23 acquires information including the acquired number of installed seats and the number of seats in use as seat information, and transmits the information to the Web application server 30.

Next, the operations of the video acquisition unit 21 and the object recognition unit 22 will be described.

As described above, in this embodiment, the image acquisition unit 21 acquires the image captured by the camera 10 and supplies it to the object recognition unit 22 every predetermined period T1. The camera 10-1 installed in the first store sends a frame (image) of the area of the field of view A including the chairs C1 to C8 to the image acquisition unit 21 of the local computer 20-1 at least every period T1. Supply.

The object recognition unit 22 performs object recognition processing on frames (images) for each time series supplied from the video acquisition unit 21.

The object recognition unit 22 of this embodiment first analyzes a frame (image) for each period T1 and performs a process of recognizing a human object (hereinafter, referred to as a “human object recognition process”). That is, the object recognition unit 22 performs a process of recognizing a human object (human object recognition process) from a series of frames (images) captured by the camera 10.

Various methods can be applied to the processing method for recognizing a human object in the object recognition unit 22. Here, the object recognition unit 22 will be described as using the human object detection process using the so-called “background subtraction method”.

The background subtraction method is a difference obtained by comparing an image to be detected (observation target) (hereinafter referred to as "observation image") and an image acquired in advance (hereinafter referred to as "background image"). Based on this, it is a process of extracting an object (moving object) that does not exist in the background image.

Here, the object recognition unit 22 includes a frame of an observation image to be detected by a human object (hereinafter, referred to as an “observation frame”) and a frame of a background image whose time series precedes the observation image (hereinafter, “background frame”). Based on the difference from (referred to as "moving object"), a process of detecting a moving object (hereinafter referred to as "moving object") (hereinafter referred to as "moving object detection process") is performed, and the result of the moving object detection process is used. The person object shall be detected.

The motion detection process by the object recognition unit 22 will be described below.

[Overview of pre-processing of each frame]
First, the object recognition unit 22 performs preprocessing for improving the detection accuracy for each frame for which the motion detection process is performed. The concept that is the premise of the preprocessing by the object recognition unit 22 will be described below.

In general, in the moving object detection process by the background subtraction method using the RGB color space, it is not possible to distinguish between "detection of change in light" and "detection of change due to movement of a person" when preprocessing is not performed. For example, when a person passes near the lighting, the brightness around it fluctuates slightly. The shadowed part becomes dark as people pass by. After that, after the person passes by, the shadowed part becomes bright. In the moving object detection process, when the RGB color space is used, such a change is also detected as a moving object, but since it is not actually a moving object, the change in light is noise in the moving object detection. Therefore, in order to reduce the detection of noise in the moving object detection process, it is necessary to distinguish between the change in the amount of light and the change in the object. A change in an object is roughly a change in color.

Therefore, in the object recognition unit 22 of this embodiment, when the moving object detection process is performed, the brightness is eliminated by normalizing the RGB three-dimensional vector of each pixel of the color image. Then, the object recognition unit 22 calculates the inner product of the normalization vector of each pixel of the latest image and the normalization vector of each pixel of the past image, and detects only the color change to detect the object. Improve the accuracy of detection. Details will be described later.

[Determining color and brightness]
Next, when the object recognition unit 22 performs the motion detection process, the determination of the color and the brightness for each pixel constituting each frame will be described.

The color of an object is determined by the wavelength of the light reflected by the object. The distribution of wavelengths of light reflected by an object is fixed. For example, apples reflect a lot of light with long wavelengths (red, etc.), but absorb most of the other wavelengths.

When a human or a camera recognizes light of a long wavelength, it is judged as red. Briefly, color is determined by the ratio of wavelengths of reflected light. The color of an object that reflects less green and blue light than red light is judged to be red, the color of an object that reflects blue light to the same extent as red light, and the color of an object that reflects less green light is purple. Is determined. On the other hand, the brightness (brightness) is determined by the weighted average of RGB. To put it briefly, the amount of reflected light determines the brightness. Generally, the calculation formula for determining the brightness (brightness) can be expressed by the following formula (1), for example.
Brightness = 0.3R + 0.6G + 0.1B ... (1)

[Inner product of RGB normalized vectors]
Next, the normalization process of each pixel constituting each frame will be described.

If "color type" is determined by the ratio of RGB and "brightness" is determined by the weighted average of RGB, it is considered that the direction of the three data vectors, RGB, is the color type and the length of the vector is brightness. be able to. That is, when only the brightness changes, it is considered that only the length of the vector changes. By normalizing the RGB vector, it is possible to ignore the change in brightness and detect only the change in color.

If the direction of the RGB normalization vector determines the type of color, the color change can be detected by examining the dot product value of the two normalization vectors. If there is no change in color, the direction of the vector does not change at all. Therefore, the dot product value of the vector should be quite close to "1". However, if there is a change in color, the dot product value approaches the direction from 1 to 0. That is, the closer the inner product value is to 0, the greater the change in color. Therefore, if the inner product of the RGB normalization vectors is taken between frames, it is possible to discriminate the color change according to the magnitude of the inner product.

[Motion detection processing based on RGB normalization vector]
When detecting a moving object in a moving image (multiple consecutive frames (still images)), it is detected by binarizing the RGB or RGB brightness difference of the same-position pixels of two frames adjacent to each other in time series. Is common. Further, instead of two still images whose time series are adjacent to each other, one image may be an image acquired from a camera and one image may be an average of several past images, and the difference between them may be taken.

When a pixel having one RGB information is expressed as a vector, it can be expressed as the following equation (2). In the following, as shown in equation (2), a vector indicating one pixel composed of RGB information will be referred to as a “pixel vector”.

Here, if the pixels having RGB information are arranged in i rows and j columns (both i and j are integers of 1 or more) are expressed as a vector matrix A (hereinafter referred to as "RGB image matrix"), the following ( It can be shown as Eq. 3). In this case, it can be said that the RGB image matrix A represents each pixel constituting the i-by-j frame in the form of a pixel vector.

Here, a pixel having one luminance information (luminance = s) is expressed as a scalar. Then, when the pixels having the luminance information are arranged in i rows and j columns (both i and j are integers of 1 or more) as a scalar matrix S (hereinafter referred to as "luminance image matrix"), the following (4) ) Can be shown as an equation. In this case, it can be said that the luminance image matrix S represents each pixel constituting the frame of i-th row and j-column by the luminance s.

In the following, time (time series) is expressed as t _n . It is assumed that n represents a time (time series), and the smaller the value, the earlier time (time series). It is assumed that n is represented by an integer of 0 or more.

Here, the difference between the RGB image matrix based on an arbitrary time t ₁ frame and the RGB image matrix based on a time t ₀ (time adjacent to the time t _{1 in} the previous time series) time t ₀ (time series is When the difference on the pixel vector of each pixel between adjacent frames) is A _diff , A _diff can be expressed as the following equation (5). Further, the difference between the luminance image matrix based on the frame at time t _{1 and} the luminance image matrix based on the frame at time t ₀ (difference in luminance of each pixel between frames adjacent in time series) S _diff is as follows. It can be expressed as in equation 6). Also, the difference S _diff between the luminance image matrix and time t ₁ and the time-series luminance image matrix based on the frame time adjacent t ₀ is based on the frame time t ₁ is less, it is expressed as (8) it can.

Further, when the difference between the RGB image matrix based on the frame of an arbitrary time t _{n + 1} (n is an integer of 1 or more) and the average of the RGB image matrix based on the past n frames is A _diff , the A _diff is as follows. It can be expressed as the equation (7) of.

Further, when the difference between the luminance image matrix based on the frame of arbitrary time t _{n + 1} (n is an integer of 1 or more) and the average of the luminance image matrix based on the past n frames is S _diff , the S _diff is as follows. It can be expressed as the equation (8) of.

ｓ ＝ ０．３Ｒ ＋ ０．６Ｇ ＋ ０．１Ｂ …（１１） Here, (6) and Formula (8), by binarizing by performing threshold processing based on the comparison with the threshold value th1 for luminance of each pixel (a scalar quantity), the observation frame (time t ₁ or time The foreground region, which is a region where a moving object exists, and the background region, which is a region where a moving object does not exist, can be detected on the t _{n + 1} frame). For example, as in the following equations (9) and (10), a pixel whose luminance exceeds the threshold th1 is a foreground region, and when the luminance is a threshold th1 or less, it is a background region. You may set "0" indicating that. Further, also in the equations (5) and (7), after converting each pixel vector constituting the RGB image matrix into brightness (scalar amount) using the following equation (11), the same equation (9), Threshold processing can also be performed according to the equation (10).

s = 0.3R + 0.6G + 0.1B ... (11)

However, as described above, the method of detecting the foreground region from the observation frame based only on the brightness has a problem that the change in brightness and the change in color cannot be distinguished. When the light changes on the image, the ratio of RGB does not change significantly, but the average value of RGB changes (that is, the brightness changes). That is, as shown in the above equations (5) to (8), if the method of detecting the foreground region from the observed image based only on the brightness, the change in brightness due to the change in light also becomes the foreground region.

Therefore, in order to focus only on the color change, the object recognition unit 22 of this embodiment binarizes the inner product value of the RGB normalized vector of the same position pixel of the observation frame and the background frame to obtain the foreground from the observation frame. Performs processing to detect the area. In other words, when an object is actually moving, the object that appears in the same position pixels of adjacent frames in the time series should change on the moving image. Then, as described above, since the RGB ratio of the wavelength of the reflected light differs depending on the object, the color changes.

Therefore, the object recognition unit 22 of this embodiment can detect a foreground region that is resistant to noise by using the inner product value of the RGB normalized vector between frames (between the observation frame and the background frame).

Specifically, the object recognition unit 22 can perform the foreground region detection process based on the inner product value of the RGB normalized vector by performing the following calculation process. The object recognition unit 22 acquires the RGB image matrix A for each frame, and normalizes (normalizes the pixel vector) to all the pixels (pixel vectors) of the acquired RGB image matrix A. Converts to data with the brightness component excluded.

The object recognition unit 22 obtains the normalized RGB image matrix by normalizing the RGB image matrix A (converting the pixel vector of each pixel into a unit vector) by, for example, the calculation shown in the following equation (12). can do. In equation (12), the normalized matrix and the normalized vector (normalized pixel vector) are expressed in bold. In the following, the normalized RGB image matrix will be referred to as a "normalized RGB image matrix".

The object recognition unit 22 acquires a normalized RGB image matrix for each frame, and inner product value (hereinafter, normalized pixel vector) of each pixel vector (normalized pixel vector) of the normalized RGB image matrix between frames whose time series is adjacent to each other (hereinafter, By acquiring the “inter-frame vector inner product value”) and performing threshold processing (binarization) on the acquired inter-frame vector inner product value, a process of separating the observed frame into a foreground region and a background region is performed. This inter-frame vector dot product value indicates the degree of color change in the corresponding pixel, which is an abstraction of the change in luminance.

The object recognition unit 22 uses, for example, the following equation (13) to form a normalized RGB image matrix based on an arbitrary time t _n frame and a time t _{n + 1} frame (time t _n and time series are adjacent to each other). It is possible to obtain the inter-frame vector dot product value of each pixel with the normalized RGB image matrix based on the matching frame). In the equation (13), the matrix of the inter-frame vector inner product values of i-by-j columns is shown as "I". In the following, this matrix I will be referred to as an "inner product value matrix".

Then, the object recognition unit 22 performs threshold processing based on comparison with the threshold th2 to binarize each inter-frame vector inner product value (inter-frame vector inner product value corresponding to each pixel) constituting the inner product value matrix I. By doing so, it is possible to detect the foreground region, which is the region where the moving object exists, and the background region, which is the region where the moving object does not exist, on the observation frame. For example, a pixel whose inter-frame vector inner product value exceeds the threshold value th2 is set to “1” indicating that it is a foreground region, and when the inter-frame vector inner product value is equal to or less than the threshold value th2, it is set to “0” indicating that it is a background region. You may set it. The specific value set in the threshold value th2 is not limited. For example, the object recognition unit 22 may be set in advance with the optimum th2 obtained by an experiment or the like, or may be configured as a parameter that can be adjusted by the user's operation.

As described above, the object recognition unit 22 normalizes the RGB image matrix A for each frame to acquire the normalized RGB image matrix, and further, the inner product based on the normalized RGB image matrix between the frames whose time series is adjacent to each other. Get the value matrix I. Then, the object recognition unit 22 can detect the foreground region in each frame by binarizing the inter-frame vector inner product value constituting the inner product value matrix I. That is, the object recognition unit 22 can acquire an image in which the foreground area is 1 and the background area is 0 (hereinafter, referred to as “foreground area image”).

Here, as an example, it is assumed that the frame (image) captured by the camera 10-1 is an image as shown in FIG. In the frame (image) shown in FIG. 10, the seat user is seated in the chairs C1, C5, and C6. That is, in the frame shown in FIG. 10, it is assumed that the moving objects are only the seat users seated in the chairs C1, C5, and C6.

Then, the object recognition unit 22 acquires a normalized RGB image matrix for a series of frames captured by the camera 10-1, and further, an inner product value matrix based on the normalized RGB image matrix between frames whose time series is adjacent to each other. I can be obtained. Then, the object recognition unit 22 can obtain an image in which the foreground region and the background region are distinguished as shown in FIG. 11 by binarizing the inner product value matrix I between the frames adjacent to each other in the time series. .. In FIG. 11, the foreground region is shown as black and the background region is shown as white. As can be seen by comparing FIGS. 10 and 11, in FIG. 11, the area in which the seat user seated in the chairs C1, C5, and C6 is shown is the foreground area (the area in which the moving object is shown), and the other areas are used as the background. It can be seen that it is detected as an area. In this way, the object recognition unit 22 can detect the area in which the moving object is captured by acquiring the inner product value matrix I between the frames adjacent to each other in the time series.

Then, as shown in FIG. 12, the object recognition unit 22 cuts out and manages an image (hereinafter, referred to as “extrinsic rectangular image”) of a rectangular (extrinsic rectangular) region including the region of each recognized moving object.

The object recognition unit 22 may recognize that each of the detected moving objects is a human object, or may perform further recognition processing to detect the human object from the detected moving object. The method for further narrowing down human objects from the detected moving objects in the object recognition unit 22 is not limited, and various methods can be applied.

Here, it is assumed that the object recognition unit 22 recognizes that each detected moving object is a human object. Then, when the frame as shown in FIG. 10 is acquired, the object recognition unit 22 recognizes the seat users seated in the chairs C1, C5, and C6 as the human objects OB101, OB102, and OB103, respectively.

Further, the object recognition unit 22 may recognize the seat user from the detected person object. The specific process of recognizing the seat user from the person object detected by the object recognition unit 22 is not limited. Here, the object recognition unit 22 will be described as recognizing all the recognized person objects as the person objects of the seat users.

[Overall flow of motion detection process]
Next, the flow of the entire motion detection process performed by the object recognition unit 22 will be described with reference to the flowchart of FIG.

FIG. 13 is a flowchart showing an example of an operation in which when the object recognition unit 22 acquires a frame (image data) of time t _n , the dynamic detection process is performed on the frame.

When the frame of time t _n is acquired, the object recognition unit 22 holds an RGB image matrix in which the RGB information of each pixel constituting the acquired frame is used as a pixel vector (S101).

Next, the object recognition unit 22 normalizes each pixel vector of the acquired RGB image matrix to acquire the normalized RGB image matrix (S102).

Next, the object recognition unit 22 normalizes the pixel vector corresponding to each pixel of the acquired RGB image matrix, and acquires the normalized RGB image matrix (S103).

Next, the object recognition unit 22 has normalized the RGB image matrix based on the frame of time t _n and the normalized RGB image matrix based on the frame of time t _n-1 (the frame in which the time t _n and the time series are adjacent to each other). The inter-frame vector inner product value (inter-frame vector inner product value) of each pixel with and from the RGB image matrix is calculated and acquired as an inner product value matrix (S104).

Next, the object recognition unit 22 acquires the foreground region image in the frame of time t _n by binarizing the inter-frame vector inner product value constituting the inner product value matrix (S105).

As described above, the object recognition unit 22 acquires a foreground area image as shown in FIG. 11 for each frame, and performs recognition processing of the human object and the seat user based on the acquired foreground area image.

Next, the operation of updating the seat information in the Web application server 30 and the DB server 40 will be described.

Here, it is assumed that information other than the seat information (number of installed seats, number of seats in use) of each store is registered as initial master data in the store information DB 421 of the DB server 40.

When the server application 32 of the Web application server 30 receives seat information (number of installed seats and number of seats in use) from the local computer 20 of each store, the information of the DB server 40 (store information DB 421) is based on the seat information. (Seat information related to the received store (number of installed seats and number of seats in use)) is updated. As described above, the timing at which the seat information processing unit 23 transmits the seat information to the Web application server 30 is not limited.

As described above, the DB server 40 (store information DB 421) is always kept in the latest state (the state where the latest seat information of each store is held) by the seat information received from the local computer 20 of each store. become.

Next, the operation of providing seat information on the Web application server 30 and the DB server 40 will be described.

As described above, the Web application server 30 provides the content including the seat information held by the DB server 40 in response to the request of the client terminal 50. The content (web screen data) provided from the Web application server 30 to the client terminal 50 will be displayed on the touch panel display 51 of the client terminal 50.

For example, when the Web application server 30 (Web service unit 31) receives access (for example, access by a protocol such as HTTP) from the client terminal 50 at a predetermined URL (URL of the homepage for providing seat information). , The content (web screen data) of the Web screen (hereinafter, referred to as "store selection screen") capable of accepting the selection of the store is generated and provided to the client terminal 50.

FIG. 14 is an explanatory diagram showing a configuration example of the store selection screen.

The store selection screen (Web screen) of FIG. 14 displays a button (link) to which the store name of each store is added, and the user who owns the client terminal 50 selects (clicks) a store that provides content including seat information. ; Touch) is accepted. On the store selection screen shown in FIG. 14, the button B101 for accepting the selection of the first store (store name: "AAA cafe") and the button B101 for accepting the selection of the second store (store name: "BBB restaurant") are accepted. The state in which the button B102, the button B102 for accepting the selection of the third store (store name: “CCC coffee”), and the like are displayed is shown.

At this time, the server application 32 acquires the store name for each store from the DB server 40 (store information DB 421), and causes the Web service unit 31 to generate a store selection screen (Web screen) as shown in FIG. Do. The specific configuration of the store selection screen is not limited as long as it is a Web screen on which any store can be selected.

Here, on the store selection screen shown in FIG. 14, it is assumed that the button B101 for accepting the selection of the first store (store name: “AAA cafe”) is selected by the user who owns the client terminal 50.

When the first store (store name: "AAA cafe") is selected on the store selection screen, the server application 32 uses the DB server 40 (store information DB421) to display the seat information (number of installed seats and seats) of the first store. Acquire the number of seats in use) and store information (address, business hours, homepage URL). Then, the server application 32 receives the information (information including the seat information and the store information) acquired by the Web service unit 31 based on the acquired seat information and the store information of the first store (store name: "AAA cafe"). A process is performed in which a Web screen for presentation (hereinafter, also referred to as a “store information providing screen”) is generated and provided to the connection source client terminal 50.

FIG. 15 is an explanatory diagram showing a configuration example of the store information providing screen.

In the store information providing screen shown in FIG. 15, a field F101 for displaying seat information and a field F102 for displaying information regarding the first store (information including the store name and store information) are arranged.

On the store information providing screen shown in FIG. 15, the number of vacant seats and the number of seats (the number of seats in use) are displayed as seat information in the field F101. On the store information providing screen, only the number of vacant seats may be displayed as seat information, or the number of installed seats and the number of seats (number of seats in use) may be displayed.

(A-3) Effect of Embodiment According to this embodiment, the following effects can be achieved.

In the seat information processing system 1 of this embodiment, the number of vacant seats in each store can be recognized only by registering only the information on the number of installed seats as the seat information in each store. As a result, in the seat information processing system 1 of this embodiment, since it is not necessary to input the position information of the seats of each store, the introduction work and the layout change of the seats are facilitated, and the convenience is improved as compared with the conventional case. are doing.

Further, in the seat information processing system 1 of this embodiment, the number of seats in use (number of seated users) can be recognized only by installing a camera 10 in each store and performing image processing. As a result, in the seat information processing system 1 of this embodiment, it is not necessary to attach a sensor for each seat, so that the introduction cost can be reduced. In other words, the seat information processing system 1 of this embodiment does not recognize the presence or absence of a user for each seat, but is reflected in the image captured by the camera 10 by the person object tracking process and the seat user object detection process. Since only the number of seated users is counted, the number of seated users can be recognized with a smaller amount of processing as compared with the case where the image recognition processing is performed for each seat.

Further, in the seat information processing system 1 of this embodiment, the object recognition unit 22 normalizes the RGB image matrix for each frame and acquires the inner product value matrix I between the frames in which the time series are adjacent to each other. Then, the object recognition unit 22 detects the foreground area image in each frame by binarizing the inter-frame vector inner product value constituting the inner product value matrix I, and performs human object detection processing based on the foreground area image. ing. As a result, in the seat information processing system 1 of this embodiment, it is possible to ignore the change in brightness and perform the moving object detection process (detection process in the foreground region) based only on the change in color. That is, in the seat information processing system 1 of this embodiment, more accurate motion recognition is performed by performing motion detection processing (foreground region detection processing) based on the inner product value matrix I (inner product value corresponding to each pixel). Processing (human object recognition processing) can be performed.

(B) Other Embodiments The present invention is not limited to each of the above embodiments, and modified embodiments as illustrated below can also be mentioned.

(B-1) In the above embodiment, the processing performed by each local computer 20 may be processed on the server side (Web application server 30 or a server attached to the Web application server 30).

(B-2) In the above embodiment, the moving object detection process (foreground area detection process) based on the inner product value matrix I (inner product value corresponding to each pixel) is performed, and the human object recognition process in the seat information processing system 1 is performed. However, the target moving object to be detected by the moving object detection process based on the inner product value matrix I is not limited to humans, and may be applied to the processing for detecting non-human moving objects such as animals and automobiles. That is natural.

1 ... Seat information processing system, 10,10-1 to 10-N ... Camera, 20, 20-1 to 20-N ... Local computer, 21 ... Video acquisition unit, 22 ... Object recognition unit, 23 ... Seat information processing unit , 24 ... Data recording unit, 30 ... Web application server, 31 ... Web service department, 32 ... Server application, 33 ... Data recording unit, 40 ... DB server, 41 ... DB processing unit, 42 ... Data recording unit, 421 ... Store Information DB, 50 ... Client terminal, 51 ... Touch panel display, 52 ... Client application.

Claims (4)

An imaging means that images the inside of a facility where one or more seats are arranged, and
At least the number of in-use seats used in the facility is recognized based on the time-series frames imaged by the imaging means, and the usage status of the seats in the facility is determined based on the recognized number of in-use seats. Equipped with a seat information acquisition means to acquire the indicated seat information
The seat information acquisition means
For each frame, the pixel value for each pixel is converted into a pixel vector represented by a vector value composed of RGB information and acquired, and each acquired pixel vector is normalized to a unit vector to obtain a normalized pixel vector. Pixel vector processing means and
An inner product value acquisition means for acquiring the inter-frame inner product value of the normalized pixel vector corresponding to each pixel between frames adjacent to each other in the time series.
Foreground area recognition means that recognizes the foreground area in each frame based on the inter-frame inner product value between frames acquired by the inner product value acquisition means,
Based on the foreground area recognized by the foreground area recognition means, the person object recognition process for recognizing the person object reflected in each frame is performed, and the person object recognition process is used in the facility based on the result of the person object recognition process. A seat information processing system characterized by having a recognition means for recognizing the number of seats in use. The foreground region recognition means according to claim 1, wherein the foreground region recognition means recognizes the foreground region in each frame based on the result of binarizing the inter-frame inner product value between each frame based on a predetermined threshold value. Seat information processing system. At least the number of in-use seats used in the facility is recognized based on the time-series frames obtained by imaging the inside of the facility in which one or more seats are arranged, and the above-mentioned is based on the recognized number of in-use seats. It has a seat information acquisition means for acquiring seat information indicating the usage status of seats in the facility.
The seat information acquisition means
For each frame, the pixel value for each pixel is converted into a pixel vector represented by a vector value composed of RGB information and acquired, and each acquired pixel vector is normalized to a unit vector to obtain a normalized pixel vector. Pixel vector processing means and
An inner product value acquisition means for acquiring the inter-frame inner product value of the normalized pixel vector corresponding to each pixel between frames adjacent to each other in the time series.
Foreground area recognition means that recognizes the foreground area in each frame based on the inter-frame inner product value between frames acquired by the inner product value acquisition means,
Based on the foreground area recognized by the foreground area recognition means, the person object recognition process for recognizing the person object reflected in each frame is performed, and the person object recognition process is used in the facility based on the result of the person object recognition process. A seat information acquisition device characterized by having a recognition means for recognizing the number of seats in use. Computer,
At least the number of in-use seats used in the facility is recognized based on the time-series frames obtained by imaging the inside of the facility in which one or more seats are arranged, and the above-mentioned is based on the recognized number of in-use seats. It functions as a seat information acquisition means to acquire seat information indicating the usage status of seats in the facility.
The seat information acquisition means
For each frame, the pixel value for each pixel is converted into a pixel vector represented by a vector value composed of RGB information and acquired, and each acquired pixel vector is normalized to a unit vector to obtain a normalized pixel vector. Pixel vector processing means and
An inner product value acquisition means for acquiring the inter-frame inner product value of the normalized pixel vector corresponding to each pixel between frames adjacent to each other in the time series.
Foreground area recognition means that recognizes the foreground area in each frame based on the inter-frame inner product value between frames acquired by the inner product value acquisition means,
Based on the foreground area recognized by the foreground area recognition means, the person object recognition process for recognizing the person object reflected in each frame is performed, and the person object recognition process is used in the facility based on the result of the person object recognition process. A seat information acquisition program characterized by having a recognition means for recognizing the number of seats in use.

Images