JP6653675B2 - Article management system and article management method by image analysis - Google Patents

Description

  The present invention relates to an article management system and method for managing articles at home based on image analysis.

  A technique described in Patent Literature 1 is known as a system for managing consumables and replacement parts using images. Patent Literature 1 discloses that “a camera capable of photographing a stock area for storing a consumable based on identification information for identifying the detected consumable in an image forming system including an image forming apparatus that communicates with the camera is registered. Here, if it is determined that the consumable is registered, a request to acquire the stock amount of the consumable is sent to the camera together with the identification information for identifying the consumable. Acquiring the current stock amount of the consumable item specified by the identification information, and displaying the acquired current stock amount of the consumable item on the display means in association with the stock area. "

JP-A-2005-200839

  Since various articles exist in a house, it is difficult for a user to sequentially manage the state of each article. In addition, it is difficult for the user to efficiently manage the articles existing in the house due to the purchase, exchange, disposal, sale, etc. of the articles, which change in a fluid manner.

  An object of the present invention is to provide a system and a method for efficiently and comprehensively managing articles.

A typical example of the invention disclosed in the present application is as follows. That is, an article management system including an image acquisition device that acquires an image and a computer, wherein the computer has a processor, a memory connected to the processor, and a network interface connected to the processor, wherein the memory is Storing an article list that manages articles present in the space where the image acquisition device is installed, wherein the article list is a field that stores the name of the article, information indicating the use state of the article, and the number of times the article has been used. Comprising a record configured, the image acquisition device acquires the image data of the space, transmits the image data to the computer, the computer, by analyzing the image data, articles present in the space, identifying a use state of the article, and calculates the number of times of use of the article based on the results of the analysis of the image data, Updating the article list based on number of uses of serial article, on the basis of the article list, said generates display data for displaying the use status of the article in the image data, and outputs the display data, the article list , A target article whose use count is smaller than a threshold is specified, proposal information for suggesting a disposal method of the target article is generated, and the proposal information is output .

  According to the present invention, articles can be managed comprehensively by updating the article list based on the image data, and the state of the articles can be efficiently managed based on the display data. Problems, configurations, and effects other than those described above will be apparent from the following description of the embodiments.

FIG. 1 is a diagram illustrating an example of a configuration of an article management system according to a first embodiment. FIG. 2 is a diagram illustrating an example of an image acquisition device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image acquisition device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image acquisition device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image acquisition device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image acquisition device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image acquisition device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image acquisition device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image acquisition device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image acquisition device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image acquisition device according to the first embodiment. FIG. 2 is a diagram illustrating an example of an image acquisition device according to the first embodiment. FIG. 3 illustrates an example of a configuration of a computer according to the first embodiment. FIG. 3 illustrates an example of a configuration of a management computer according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of image analysis information according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of an article list according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of an article list according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of an article list according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of an article list according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of an article list according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of an article list according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of an article list according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of an article list according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of an article list according to the first embodiment. FIG. 6 is a diagram illustrating an example of a data structure of an excluded article list according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of location information according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of person information according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of calendar information according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of house information according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of needs information according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of maintenance information according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of replacement item information according to the first embodiment. FIG. 4 is a diagram illustrating an example of a data structure of event item information according to the first embodiment. 6 is a flowchart illustrating an example of an article management process executed by the computer according to the first embodiment. 5 is a flowchart illustrating an example of a display process of image data performed by the article management module according to the first embodiment. FIG. 4 is a diagram illustrating a display example of image data according to the first embodiment. 5 is a flowchart illustrating an example of a process performed by the article management module according to the first embodiment to notify the location of an article. 6 is a flowchart illustrating an example of a process for suggesting movement or disposal of an article, which is executed by the article management module according to the first embodiment. FIG. 7 is a diagram illustrating an example of an operation GUI for displaying an item list according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. 5 is a flowchart illustrating an example of a process for proposing maintenance of an article performed by the article management module according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. 6 is a flowchart illustrating an example of a process for proposing replacement of an article performed by the article management module according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. 9 is a flowchart illustrating an example of a process of presenting a missing item of an event performed by the item management module according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. FIG. 6 is a diagram illustrating an example of an operation state of the operation GUI according to the first embodiment. 9 is a flowchart illustrating an example of a need confirmation process performed by the management computer according to the first embodiment. FIG. 5 is a diagram illustrating an example of an operation GUI according to the first embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of the configuration of the article management system according to the first embodiment.

  The article management system includes a plurality of houses 100 and a management computer 110. The house 100 includes a computer 120 and a plurality of image acquisition devices 130. The management computer 110 and the computers 120 included in each house 100 are connected to each other via an external network 105. The computer 120 and the plurality of image acquisition apparatuses 130 are connected to each other via a home network 150.

  The image acquisition device 130 acquires image data and transmits the acquired image data to the computer 120. The image acquisition device 130 may be, for example, a camera provided in a lighting device (such as a light bulb and a ceiling light), an air conditioner, and a robot device. An example of the image acquisition device 130 will be described with reference to FIG.

  The computer 120 manages articles owned by the user of the house 100. The specific configuration of the computer 120 will be described with reference to FIG.

  The management computer 110 controls the entire system. More specifically, the management computer 110 realizes cooperation with various services based on the state of the article in each house 100, the article management policy (intention), the service use policy (intention), and the like. Examples of the service include an article sharing service, an article trading service, a maintenance service, a delivery service, and a calendar service. The specific configuration of the management computer 110 will be described with reference to FIG.

  2A to 2K are diagrams illustrating an example of the image acquisition device 130 according to the first embodiment. Here, a ceiling hanging light with a camera will be described as an example.

  The image acquisition device 130 shown in FIG. 2 is a ceiling hanging lamp having a stereo camera 201, a sensor 202, and a voice microphone 203, and using a light emitting diode as a light source. 2A and 2B are reference perspective views of a ceiling hanging light. FIG. 2C is a front view of the ceiling hanging light. FIG. 2D is a right side view of the ceiling hanging light. FIG. 2E is a left side view of the ceiling hanging light. FIG. 2F is a plan view of the ceiling hanging light. FIG. 2G is a bottom view of the ceiling hanging light. FIG. 2H is a rear view of the ceiling hanging light. FIG. 2I is a reference plan view showing a translucent portion of the ceiling hanging lamp. The hatched portions have translucency and are turned on using light emitting diodes as light sources. FIG. 2J is an AA cross-sectional view of the ceiling hanging light with the internal structure omitted. FIG. 2K is a reference perspective view showing a use state of the ceiling hanging light.

  The head of the ceiling hanging light, which is the image acquisition device 130, has a stereo camera 201, a sensor 202, and an audio microphone 203. When the sensor 202 detects movement, the stereo camera 201 and the audio microphone 203 operate.

  FIG. 3 is a diagram illustrating an example of a configuration of the computer 120 according to the first embodiment.

  The computer 120 has a processor 301, a memory 302, a network interface 303, and a storage device 304. Each hardware is connected to each other via a bus or the like.

  The processor 301 executes a program stored in the memory 302. The processor 301 operates as a module that realizes a specific function by executing processing according to a program. In the following description, when processing is described using a module as a subject, it indicates that the processor 301 is executing a program that realizes the module.

  The memory 302 stores a program to be executed by the processor 301. Further, the memory 302 includes a work area used by the program. The memory 302 according to the present embodiment stores a program that implements the article management module 311.

  The article management module 311 analyzes the image data, and specifies the articles existing in the room corresponding to the image data and the state of the articles. Further, the article management module 311 receives an input regarding an intention to manage an article and an intention to use a service.

  The article management module 311 manages articles according to the state of the article, the intention to manage the article, and the intention to use the service. The article management module 311 provides a service to the user by cooperating with the management computer 110.

  The storage device 304 stores data permanently. The storage device 304 may be, for example, a hard disk drive (HDD), a solid state drive (SSD), and a nonvolatile memory. The storage device 304 of the present embodiment stores image analysis information 321, an item list 322, an excluded item list 323, location information 324, person information 325, and calendar information 326. Although not shown, the storage device 304 stores a history of image data in which image data and date and time are associated.

  The image analysis information 321 stores an analysis result of the image data acquired from the image acquisition device 130. The specific data structure of the image analysis information 321 will be described with reference to FIG.

  The article list 322 is information for managing articles. In this embodiment, the state of the article, the intention to manage the article, and the intention to use the service are managed using the article list 322. The specific data structure of the article list 322 will be described with reference to FIGS. 6A to 6I.

  The excluded article list 323 is information for managing articles to be excluded from management targets. The articles registered in the excluded article list 323 are not managed using the article list 322. The specific data structure of the excluded article list 323 will be described with reference to FIG.

  The place information 324 is information for managing places where articles can be installed or stored. It is assumed that the location information 324 is set in advance. In addition, it can be updated as needed in accordance with the renovation of the house 100. A specific data structure of the location information 324 will be described with reference to FIG.

  The personal information 325 is information for managing image data of a resident of the house 100 and a visitor visiting the house 100. The specific data structure of the person information 325 will be described with reference to FIG.

  The calendar information 326 is information for managing an event held at the house 100. A specific data structure of the calendar information 326 will be described with reference to FIG.

  FIG. 4 is a diagram illustrating an example of a configuration of the management computer according to the first embodiment.

  The management computer 110 has a processor 401, a memory 402, a network interface 403, and a storage device 404. Each hardware is connected to each other via a bus or the like.

  The processor 401, the memory 402, the network interface 403, and the storage device 404 are the same as the processor 301, the memory 302, the network interface 303, and the storage device 304, and a description thereof will be omitted.

  The memory 402 stores a program that implements the overall control module 411. The overall control module 411 controls the entire system.

  The storage device 404 stores house information 421, needs information 422, maintenance information 423, replacement article information 424, event article information 425, and template image information 426.

  The house information 421 is information for managing the house 100 included in the article management system. A specific data structure of the house information 421 will be described with reference to FIG.

  The needs information 422 is information for managing the needs of the user for the article. A specific data structure of the needs information 422 will be described with reference to FIG.

  The maintenance information 423 is information for managing an article maintenance service. The data structure of the maintenance information 423 will be described with reference to FIG.

  The exchange article information 424 is information for specifying an exchange article. The data structure of the replacement article information 424 will be described with reference to FIG.

  The event article information 425 is information for managing articles necessary for the event. The specific data structure of the event article information 425 will be described with reference to FIG.

  The template image information 426 is information for managing a template image for specifying an article. The template image information includes a record including a template image representing an article, feature amounts of the template image, and attribute information of the article. One record corresponds to one article. One record may include a plurality of feature amounts calculated based on template images captured from various angles. It should be noted that the attribute information includes various kinds of information about the article, such as the type of the article, the article name, the size, the weight, and the use state.

  Note that as an algorithm for calculating a feature amount from an image, SIFT (Scale Independent Feature Transform), SURF (Speed Up Robust Features), and the like can be considered. Note that the present invention is not limited to the algorithm for calculating the feature amount.

  Since the template image information 426 stores template images of various articles, the management computer 110 holds the template images. Note that the computer 120 may hold the template image information 426.

  Next, information held by the computer 120 will be described.

  FIG. 5 is a diagram illustrating an example of a data structure of the image analysis information 321 according to the first embodiment.

  The image analysis information 321 includes a record including a date and time 501, a device ID 502, an article type 503, an article name 504, coordinates 505, a user 506, and a use state 507. One record corresponds to the article specified from the image analysis.

  The date and time 501 stores the date and time when the image data was acquired. The device ID 502 stores identification information of the image acquisition device 130 that has acquired the image data.

  The article type 503 stores the type of the specified article. The article name 504 stores the specific name of the article, such as the model number and product name of the specified article. The coordinates 505 store the coordinates indicating the position of the article in the room where the image data is acquired. The user 506 stores identification information of a person who has used the specified article. The use state 507 stores information indicating the use state of the article. In the present embodiment, either “in use” or “unused” is stored.

  In the case of an article having no distinction between used and unused, the user 506 and the use state 507 are blank.

  6A to 6I are diagrams illustrating an example of the data structure of the article list 322 according to the first embodiment.

  The article list 322 includes a record including an article ID 601, an article 602, an owner 603, a state 604, a storage 605, a maintenance setting 606, a lending setting 607, a sale setting 608, and a distribution state 609.

  The article ID 601 stores identification information for uniquely identifying a record in the article list 322.

  The article 602 is a group of fields for storing details of the article. As shown in FIG. 6B, the article 602 includes an article type 621, an article name 622, a size 623, and a weight 624.

  The item type 621 and the item name 622 are the same fields as the item type 503 and the item name 504. The size 623 stores the size of the article. The weight 624 stores the weight of the article.

  The owner 603 is a field group that stores details of the owner of the article. As shown in FIG. 6C, the owner 603 includes an owner ID 625 and a house ID 626.

  The owner ID 625 stores identification information of a user who is the owner of the article. The house ID 626 stores identification information of the house 100 to which the owner belongs. In this embodiment, the owner 603 includes the house ID 626 because there is a case where an article existing in another house 100 is rented. Note that the house ID 626 of the article in the house 100 is blank. In the house ID 626, a value set in house information 421 described later is stored.

  The article management module 311 receives identification information of the owner of the article in his / her home 100 via a terminal operated by the user. The article management module 311 acquires a record for a certain period from the image analysis information 321 and, based on the analysis result of the record, sets the user who has used the most or the user who first used the article as the owner. May be set. When renting an article from another house 100, information on the other house 100 is set in the owner 603.

  The state 604 is a field group for storing details of the use state of the article. As shown in FIG. 6D, the state 604 includes a use period 627, a total use time 628, a use start time 629, a use state 630, a user 631, and the number of times 632.

  The use period 627 stores the use start date and use end date of the article. The total use time 628 stores the total value of the use time of the article. The use start time 629 stores the date and time when the use start of the article is detected. The use state 630 is the same as the use state 507. The user 631 is the same as the user 506. The number of times 632 stores the number of times the article has been used. It should be noted that blanks are provided for articles whose use state does not change.

  The storage 605 is a group of fields for storing details of storage positions of articles. As shown in FIG. 6E, the storage 605 includes a storage location 633 and coordinates 634.

  The storage location 633 stores identification information of a location where an article is stored. The coordinates 634 store coordinates indicating the position of the article at a location corresponding to the storage location 633.

  The maintenance setting 606 is a group of fields for storing details of the intention to use the maintenance service. As shown in FIG. 6F, the maintenance settings 606 include a maintenance service 635, an intention 636, a scheduled date 637, and a maintenance state 638.

  The maintenance service 635 stores a value indicating whether or not the article maintenance service exists. In the present embodiment, the maintenance service 635 stores either “present” or “absent”. The intention 636 stores a value indicating whether to use the maintenance service. In this embodiment, the intention 636 stores one of “0” indicating that the maintenance service is not used and “1” indicating that the maintenance service is used. The scheduled date 637 stores the date and time when the maintenance service is used. The maintenance status 638 stores a value indicating the usage status of the maintenance service. The maintenance state 638 stores one of “before use” and “under use”.

  The lending setting 607 is a group of fields for storing details of the intention to use the sharing service. As shown in FIG. 6G, the lending setting 607 includes an intention 639, a borrower 640, a lending period 641, and a number of needs 642.

  The intention 639 stores a value indicating whether to lend an item. In the present embodiment, one of “1” indicating that the article is to be lent and “0” indicating that the article is not to be lent is stored. The borrower 640 stores the identification information of the house 100 from which the article was borrowed. Note that the borrower 640 may include identification information of the user. The lending period 641 stores a period for lending the article to another person. The number of needs 642 stores the number of users who want to borrow articles.

  The number of needs 642 may store identification information of the house 100 to which the user who wants to borrow the article belongs. In this case, the number of identification information matches the number of users.

  A value is set to the intention 639 via the terminal operated by the user. Values are set for the borrower 640 and the lending period 641 based on the usage state of the sharing service. The value of the number of needs 642 is set by the processing shown in FIG.

  The sale setting 608 is a group of fields for storing details of the intention to use the trading service. As shown in FIG. 6H, the sale setting 608 includes an intention 643, a desired sale price 644, a sale state 645, a sale price 646, and a number of needs 647.

  The intention 643 stores a value indicating whether to sell the article. In this embodiment, the intention 643 stores either “1” indicating that the article is to be sold or “0” indicating that the article is not to be sold. The desired sale price 644 stores a desired sale price of the article. The sale status 645 stores a value indicating the transaction status of the article in the trading service. For example, if another person is considering purchasing an item, "under consideration" is stored. The sale price 646 stores the sale price of the article when the sales contract is concluded. The number of needs 647 stores the number of users who want to purchase an article.

  The number of needs 647 may store identification information of the house 100 to which the user who wants to purchase the article belongs. In this case, the number of identification information matches the number of users.

  A value is set to the intention 643 and the desired sale price 644 via the terminal operated by the user. Values are set in the sale status 645 and the sale price 646 based on the service use status. The number of needs 647 is set by the processing shown in FIG.

  The physical distribution status 609 is a group of fields for storing details of the use status of the article delivery service. As shown in FIG. 6I, the physical distribution status 609 includes a collection destination 648, a scheduled collection date 649, a collection status 650, a delivery destination 651, a scheduled delivery date 652, and a delivery status 653.

  The collection destination 648 stores the name of an article delivery service or the name of a company that provides the delivery service. The scheduled collection date 649 stores the scheduled collection date of the article. The collection status 650 stores a value indicating the status of the collection of articles. The delivery destination 651 stores the address or name of the delivery destination of the article. The scheduled delivery date 652 stores the scheduled delivery date of the article. The delivery status 653 stores a value indicating the status of the delivery of the article.

  Based on the use state of the delivery service, a value is set in the physical distribution state 609, and the value of the physical distribution state 609 is updated.

  FIG. 7 is a diagram illustrating an example of a data structure of the excluded article list 323 according to the first embodiment.

  The excluded article list 323 includes a record including an ID 701, an article type 702, and an article name 703. One record corresponds to one article.

  The ID 701 stores identification information for uniquely identifying a record of the excluded article list 323. The article type 702 and the article name 703 are the same as the article type 503 and the article name 504.

  The user operates the terminal to add a record to the excluded article list 323, and updates or deletes a record included in the excluded article list 323.

  FIG. 8 is a diagram illustrating an example of a data structure of the location information 324 according to the first embodiment.

  The location information 324 includes a record including a location ID 801, a location type 802, a location 803, and a device list 804. One record corresponds to one place.

  The location ID 801 stores identification information for uniquely identifying a storage location. The place ID 801 is also used as identification information of a record of the place information 324. The location type 802 stores a value indicating the location type. The place type 802 stores “room” indicating a normal room, “warehouse” indicating an external storage place such as a warehouse, and the like. The location 803 stores information indicating the position of the room. In the case of a warehouse remote from the house 100, the location 803 stores the address of the warehouse or the URL of a company that provides a service that lends the warehouse. The device list 804 stores identification information of the image acquisition device 130 installed in the storage location corresponding to the location ID 801. The device list 804 of the record corresponding to the storage location where the image acquisition device 130 is not installed is blank.

  The user operates the terminal to add a record to the location information 324, and updates or deletes a record included in the location information 324. The location ID 801 is automatically assigned by the article management module 311 when a record is added.

  FIG. 9 is a diagram illustrating an example of a data structure of the person information 325 according to the first embodiment.

  The person information 325 includes a record including a person ID 901, a name 902, and image data 903. One record corresponds to one user.

  The person ID 901 stores identification information for uniquely identifying a user. The person ID 901 is also used as identification information of a record of the person information 325. The name 902 stores the name of the user. The image data 903 stores user image data. Note that a plurality of image data may be registered for one user. The image data 903 may be an address of a storage area for storing the image data instead of the image data.

  The user operates the terminal to add a record to the person information 325 and update or delete a record included in the person information 325. Note that the person ID 901 is automatically given by the article management module 311 when a record is added.

  FIG. 10 is a diagram illustrating an example of a data structure of the calendar information 326 according to the first embodiment.

  The calendar information 326 includes a record including a date and time 1001 and an event name 1002. One record corresponds to one event.

  The date and time 1001 stores the date and time of the event. The event name 1002 stores the name of the event to be performed.

  The user operates the terminal to add a record to the calendar information 326 and to update or delete a record included in the calendar information 326.

  Note that the computer 120 may generate or update the calendar information 326 by cooperating with an online calendar service. When the calendar information can be obtained from the image data, the calculator 120 may generate or update the calendar information 326 based on the information.

  Next, information held by the management computer 110 will be described.

  FIG. 11 is a diagram illustrating an example of a data structure of the house information 421 according to the first embodiment.

  The house information 421 includes a record including a house ID 1101 and an address 1102. One record corresponds to one house 100.

  The house ID 1101 stores identification information for uniquely identifying the house 100. The house ID 1101 is also used as identification information of a record of the house information 421. The address 1102 stores the address of the house 100.

  When the management computer 110 receives an access from the computer 120 of the house 100 for the first time, the management computer 110 generates a house ID. The management computer 110 adds a record to the house information 421 and sets a house ID in the house ID 1101 of the added record. The management computer 110 requests the computer 120 to input an address. The user of each house 100 inputs an address to the computer 120 via the terminal.

  FIG. 12 is a diagram illustrating an example of a data structure of the needs information 422 according to the first embodiment.

  The needs information 422 includes a record including a house ID 1201, a needs type 1202, an article type 1203, an article name 1204, and a desired purchase price 1205.

  The house ID 1201 is the same as the house ID 1101. The need type 1202 stores a value indicating the type of need for the article. In the present embodiment, either “purchase” indicating purchase of an article or “borrow” indicating borrowing of an article is stored. The article type 1203 and the article name 1204 are the same as the article type 503 and the article name 504. The desired purchase price 1205 stores the desired purchase price of the article.

  The user operates the terminal to register needs for articles. The information input to the terminal is transmitted to the management computer 110 via the computer 120. The management computer 110 updates the needs information 422 based on the received information.

  FIG. 13 is a diagram illustrating an example of a data structure of the maintenance information 423 according to the first embodiment.

  The maintenance information 423 includes a record including an ID 1301, an article type 1302, an article name 1303, a manual 1304, a maintenance service 1305, and a replacement article 1306. One record corresponds to the article to be maintained.

  The ID 1301 stores identification information for uniquely identifying a record of the maintenance information 423. The article type 1302 and the article name 1303 are the same as the article type 503 and the article name 504.

  The manual 1304 stores a URL or the like for acquiring a manual of an article. The maintenance service 1305 stores information on the presence or absence of the maintenance service and contact information when applying for the maintenance service. The exchange article 1306 is a component of the article to be maintained, and stores information on exchangeable articles.

  The maintenance information 423 may be manually generated in advance, or a manual or the like may be obtained from a company that provides the product, and may be automatically generated based on an analysis result of the manual.

  FIG. 14 is a diagram illustrating an example of a data structure of the replacement item information 424 according to the first embodiment.

  The replacement article information 424 includes a record including an ID 1401, an article type 1402, a size 1403, the number of users 1404, and an article name 1405. One record corresponds to one article.

  The ID 1401 stores identification information for uniquely identifying a record of the replacement article information 424. The article type 1402 is the same as the article type 503. The size 1403 is the same as the size 623. The number of occupants 1404 stores the recommended number of occupants of articles. The article name 1405 is the same as the article name 504.

  FIG. 15 is a diagram illustrating an example of a data structure of the event item information 425 according to the first embodiment.

  The event article information 425 includes a record including an event ID 1501, an event name 1502, an event period 1503, and an article list 1504. One record corresponds to one event.

  The event ID 1501 stores identification information for uniquely identifying an event. The event ID 1501 is also used as identification information of a record of the event item information 425. The event name 1502 stores the name of the event. The event period 1503 stores an event holding period. The article list 1504 stores information on articles used for the event. The article list 1504 stores the type and number of articles. Note that the article list 1504 may store an article name.

  It is assumed that the event article information 425 is set in advance. The event article information 425 can be updated as appropriate.

  Next, processing performed in the article management system will be described.

(1) Method of Updating Article List 322 FIG. 16 is a flowchart illustrating an example of an article management process performed by the computer 120 according to the first embodiment.

  When acquiring image data from the image acquisition device 130, the computer 120 starts the processing described below. Note that the image data includes the identification information of the image acquisition device 130.

  The article management module 311 analyzes the image data and specifies an article included in the image data (Step S101). Specifically, the following processing is executed.

  The article management module 311 calculates a feature amount of the received image data. In this embodiment, the SIFT feature amount is calculated. The article management module 311 acquires the template image information 426 from the management computer 110.

  When the size of the template image information 426 is large, a predetermined number of record groups are acquired. In this case, the article management module 311 acquires the record group again after completing the processing for the acquired record group.

  The article management module 311 selects one template image (record) from the template image information 426. The article management module 311 compares the feature amount of the image data with the feature amount of the selected template image, and determines whether or not there is an article corresponding to the selected template image in the received image data.

  If there is no article corresponding to the selected template image in the received image data, the article management module 311 selects a new template image and performs the same processing.

  If there is an article corresponding to the selected template image in the received image data, the article management module 311 acquires the attribute information given to the selected template image, and temporarily stores the acquired attribute information in the memory. . The article management module 311 selects a new template image and performs the same processing.

  If the processing has been completed for all template images stored in the template image information 426, the article management module 311 proceeds to the next processing. The above is the description of the process in step S101.

  The article management module 311 starts loop processing of the specified article (step S102). Here, the article management module 311 selects a target article. Specifically, the article management module 311 acquires one piece of attribute information from the attribute information stored in the memory.

  The article management module 311 refers to the article list 322 and determines whether or not the target article has been registered in the article list 322 (step S103).

  Specifically, the article management module 311 determines whether or not the article type 621 and the article name 622 of the article list 322 match the article type and the article name included in the attribute information. If the above condition is satisfied, the article management module 311 determines that the target article has been registered in the article list 322.

  If it is determined that the target article has not been registered, the article management module 311 adds a record to the image analysis information 321 (Step S104). Specifically, the following processing is executed.

  The article management module 311 adds a record to the image analysis information 321, sets the date and time 501 of the added record to the date and time included in the image data, and sets the device ID 502 to the identification information of the image acquisition device 130.

  The article management module 311 sets the article type and the article name included in the attribute information in the article type 503 and the article name 504 of the added record. Further, the article management module 311 sets the coordinates at which the article exists at the coordinates 505 of the added record. Here, the position of the center of gravity of the article is set as coordinates. Note that the present invention is not limited to the standard for specifying the coordinates of the article.

  Further, the article management module 311 determines whether or not the compared image data is image data indicating an article in use based on the attribute information.

  If the compared image data is not image data indicating an article in use, the article management module 311 sets “unused” in the use state 507 of the added record. At this time, the user 506 is left blank.

  If the compared image data is image data indicating an article in use, the article management module 311 uses the image data registered in the personal information 325 to specify a person existing near the target article. Since the specifying method is the same as the method for specifying the article, the description is omitted. The article management module 311 sets the identification information of the identified person as the user 506 of the added record, and sets “in use” in the use state 507 of the record.

  When the distance between the article and the person is larger than the threshold, the article management module 311 may not set the identification information of the person to the user.

  Note that the following processing is executed for the use state 507 of an article that does not change state, such as furniture such as a sofa. If the distance between the article and the person is smaller than the threshold, the article management module 311 sets “in use” to the use state 507 of the record, and if the distance between the article and the person is greater than or equal to the threshold, the record Is set to “unused” in the use state 507 of the “. In addition, the article management module 311 sets “in use” to the use state 507 when the article moves from the default position to another position, and sets “not used” to the use state 507 when the article returns to the default position. "Use" is set.

  Next, the article management module 311 adds a record to the article list 322 (step S105). Thereafter, the article management module 311 proceeds to step S110. Specifically, the following processing is executed.

  The article management module 311 adds a record to the article list 322 and sets identification information in the article ID 601 of the added record. The article management module 311 sets the article type, the article name, the size, and the weight included in the attribute information in the article type 621, the article name 622, the size 623, and the weight 624 of the added record.

  The article management module 311 sets the date and time included in the received image data as the use start time in the use period 627 of the added record.

  Further, the article management module 311 determines whether or not the compared image data is image data indicating an article in use based on the attribute information.

  If the compared image data is not image data indicating an article in use, the article management module 311 sets “unused” in the use state 630.

  If the compared image data is image data indicating an article in use, the article management module 311 sets “in use” in the use state 630.

  Further, the article management module 311 refers to the device list 804 of the location information 324, searches for a record including the identification information of the image acquisition device 130 included in the received image data, and acquires the location ID 801 of the searched record. . The article management module 311 sets the acquired location ID 801 in the storage location 633 of the added record. Further, the article management module 311 sets the coordinates at which the article exists in the coordinates 634 of the added record. The values set for the coordinates 505 and 634 are the same.

  If the attribute information includes information on the presence or absence of the maintenance service, the article management module 311 sets a value in the maintenance service 635 of the added record based on the information. The above is the description of the processing in step S105.

  If it is determined in step S103 that the target article has been registered in the article list 322, the article management module 311 refers to the excluded article list 323 to determine whether the target article is an article to be excluded. (Step S106).

  Specifically, the article management module 311 determines whether there is a record in which the article type 702 and the article name 703 match the article type and the article name included in the acquired attribute information.

  If it is determined that the target article is an article to be excluded, the article management module 311 proceeds to step S110.

  When it is determined that the target article is not an exclusion target article, the article management module 311 registers a record in the image analysis information 321 (step S107). The processing in step S107 is the same processing as the processing in step S104.

  Next, the article management module 311 determines whether the state of the target article has changed (step S108). Specifically, the following (process 1) and (process 2) are executed.

  (Process 1) The article management module 311 acquires a value indicating the use state of the target article from the attribute information. Further, the article management module 311 refers to the article list 322 and acquires the use state 630 of the record corresponding to the target article.

  The article management module 311 compares the obtained two values to determine whether the use state has changed before and after the analysis of the image data. Specifically, it is determined whether the status has changed from “in use” to “unused” or “unused” to “in use”.

  (Process 2) The article management module 311 refers to the device list 804 of the location information 324, searches for a record including the identification information of the image acquisition device 130 included in the received image data, and specifies the location ID 801 of the searched record. get. Further, the article management module 311 obtains coordinates at which the target article exists. Further, the article management module 311 refers to the article list 322 and acquires the storage location 633 and the coordinates 634 of the record corresponding to the target article.

  The article management module 311 compares the acquired values to determine whether at least one of the storage location and the coordinates has changed before and after the analysis of the image data.

  If at least one of (Process 1) and (Process 2) described above is satisfied, the article management module 311 determines that the state of the target article has changed.

  If it is determined that the state of the target article has not changed, the article management module 311 proceeds to step S110.

  When it is determined that the state of the target article has changed, the article management module 311 updates the record of the article list 322 (Step S109).

  When (Process 1) is satisfied, the following (Process 3), (Process 4), and (Process 5) are executed.

  (Process 3) The article management module 311 sets the current use state of the target article in the use state 630 of the record corresponding to the target article.

  (Process 4) When the use state 630 is updated from “unused” to “used”, the article management module 311 sets the date and time included in the image data in the use start time 629 of the record. At this time, the article management module 311 specifies a person existing near the target article using the image data registered in the person information 325. The article management module 311 sets the identification information of the specified person in the user 631 of the record. When the same identification information is already stored, the article management module 311 does not need to set the identification information of the person. The article management module 311 also updates the number of times 632. Specifically, 1 is added to the number of times 632. Note that the value of the number of times 632 may be initialized after a certain period has elapsed. In this case, the number of times 632 stores the number of times of use per unit time, that is, a value indicating the frequency of use.

  (Process 5) When the use state 630 is updated from “use” to “unused”, the article management module 311 calculates the use time using the use start time 629 and the date and time included in the image data, and The use time is added to the total use time 628. Further, the article management module 311 initializes the use start time 629 of the record.

  If (Process 2) is satisfied, the article management module 311 sets the current storage location and coordinates in the storage location 633 and coordinates 634 of the record corresponding to the target article.

  In step S110, the article management module 311 determines whether the processing has been completed for all the specified articles (step S110).

  If it is determined that the processing has not been completed for all the specified articles, the article management module 311 returns to step S102 and performs the same processing.

  If it is determined that the processing has been completed for all the specified articles, the article management module 311 ends the article management processing. As described above, by generating the article list 322, automatic and comprehensive article management can be realized.

(2) Display Method of Image Data of Room FIG. 17 is a flowchart illustrating an example of a display process of image data performed by the article management module 311 according to the first embodiment. FIG. 18 is a diagram illustrating a display example of the image data according to the first embodiment.

  The article management module 311 starts processing described below when receiving a room display request from the user. The request includes room identification information. Instead of the room identification information, the identification information of the image acquisition device 130 may be included. In the following description, a room for displaying image data is also referred to as a target room.

  The article management module 311 acquires the image data of the target room (Step S201).

  Next, the article management module 311 refers to the article list 322 and specifies an article existing in the target room (Step S202).

  Specifically, the article management module 311 searches for a record in which the storage location 633 matches the identification information of the target room.

  Next, the article management module 311 acquires information indicating the owner and use state of the specified article (step S203).

  Specifically, the article management module 311 acquires the owner 603, the state 604, and the like from the record retrieved in step S202, and acquires the value of the coordinates 634.

  Next, the article management module 311 generates display data for displaying the owner 603, the state 604, and the like in the acquired image data (step S204). At this time, the article management module 311 generates display data based on the coordinates 634 so that the owner 603 and the state 604 overlap the image of the article.

  Next, the article management module 311 outputs the display data to the user (Step S205), and ends the processing. For example, on a terminal operated by the user, image data as shown in FIG. 18 is displayed based on the display data.

(3) Method of Notifying of Location of Article FIG. 19 is a flowchart illustrating an example of processing performed by the article management module 311 of the first embodiment to notify the location of an article.

  When receiving the inquiry about the location of the article including the name of the article from the user, the article management module 311 starts the processing described below. Note that the above inquiry may be a character input or a voice input.

  The article management module 311 refers to the article list 322 and searches for a record corresponding to the inquired article (step S301).

  Specifically, the article management module 311 searches for a record in which the article name 622 matches the name of the article.

  The article management module 311 outputs information indicating the location of the article to the user based on the record storage 605 (step S302), and ends the processing.

  For example, the article management module 311 acquires the storage location 633, and further refers to the location information 324 to search for a record that matches the value acquired by the location ID 801. The article management module 311 outputs the location 803 of the found record as information indicating the location of the article. Note that the location information of the article may be output as characters or may be output as audio.

  Further, the article management module 311 obtains image data of the room where the article exists based on the storage location 633, and outputs display data indicating the position of the article on the image data based on the coordinates 634. Is also good.

(4) Method of Proposing Movement or Disposal of Article FIG. 20 is a flowchart illustrating an example of a process of proposing movement or disposal of an article performed by the article management module 311 according to the first embodiment. FIG. 21 is a diagram illustrating an example of an operation GUI for displaying the article list 322 according to the first embodiment. 22A to 22C are diagrams illustrating an example of the operation state of the operation GUI according to the first embodiment.

  The article management module 311 starts the processing described below periodically or when receiving an instruction from the user.

  The article management module 311 starts an article loop process (step S401). Here, the article management module 311 refers to the article list 322 and selects one target article (record). In this embodiment, the records are selected in order from the upper record.

  Next, the article management module 311 calculates the number of times the target article has been used (step S402). For example, the following calculation method can be considered.

  The article management module 311 calculates the number of times 632 in the article list 322 as the number of times the target article has been used.

  As another calculation method, the following method can be considered. The article management module 311 reads, from the image analysis information 321, a record in which the article type 503 and the article name 504 match the article type 621 and the article name 622 of the record corresponding to the target article. At this time, the article management module 311 acquires a record for a certain period. For example, a record one hour before the current time is acquired. The article management module 311 calculates the number of times the target article has been used, based on the acquired use state 507 of the record. For example, the article management module 311 calculates the number of times the use state 507 has changed from “unused” to “in use” as the target article use count.

  Next, the article management module 311 determines whether the number of times the target article has been used is lower than a threshold (step S403). In this embodiment, it is assumed that a threshold value of the number of times of use is set for each type of article. The information on the threshold may be held by the computer 120 or may be held by the management computer 110.

  If it is determined that the number of times the target article has been used is equal to or greater than the threshold, the article management module 311 proceeds to step S409.

  When it is determined that the number of times of using the target article is smaller than the threshold, the article management module 311 inquires of the management computer 110 about the needs of the target article (step S404). The inquiry includes the article type 621 and the article name 622.

  The article management module 311 shifts to a wait state until a response is received from the management computer 110. When receiving the inquiry, the management computer 110 executes the following processing.

  The management computer 110 refers to the needs information 422 and searches for a record in which the article type 1203 and the article name 1204 match the article type 621 and the article name 622 included in the inquiry. When there is a record that satisfies the above condition, the management computer 110 transmits a response including the searched record to the computer 120. If there is no record that satisfies the above condition, the management computer 110 sends a response to notify the error to the computer 120.

  If the article management module 311 receives a response from the management computer 110, the article management module 311 determines whether there is a need for the target article (step S405). Specifically, the following processing is executed.

  The article management module 311 determines whether the response includes a record. If no record is included in the response, the article management module 311 determines that there is no need for the target article. If the response includes the record, the article management module 311 determines that there is a need for the target article.

  If there is a need for the target product, the product management module 311 sets the number of records whose needs type 1202 is “borrowed” to the number of needs 642 and sets the number of records whose needs type 1202 is “purchase” to the number of needs. Set to 647. The above is the description of the processing in step S405.

  If it is determined that there is a need for the target article, the article management module 311 outputs target article disposal proposal information including information on the need (step S406). Thereafter, the article management module 311 proceeds to step S410.

  If it is determined that there is no need for the target article, the article management module 311 refers to the location information 324 to search for a candidate storage location (step S407).

  Specifically, the article management module 311 searches for a record whose place type 802 is “warehouse”.

  The article management module 311 outputs target article movement proposal information including information on candidate storage locations (step S408). Thereafter, the article management module 311 proceeds to step S409.

  In step S409, the article management module 311 determines whether the processing has been completed for all the articles registered in the article list 322 (step S409).

  If it is determined that the processing has not been completed for all the articles registered in the article list 322, the article management module 311 returns to step S401 and executes the same processing.

  When it is determined that the processing has been completed for all the articles registered in the article list 322, the article management module 311 ends the processing.

  On the terminal operated by the user, an operation GUI 2100 of the article list 322 is displayed in a format as shown in FIG. It should be noted that the Living room displays the items present in the house 100, and the Warehouse displays the items present in the warehouse. By using the storage 605 and the location information 324, the article management module 311 can generate display data for displaying articles separately for the Living room and the Warehouse.

  In the operation GUI 2100, the icon of “SELL” or “SHARE” is highlighted on the thumbnail of the article for which the determination result in step S405 is YES based on the disposal proposal information. The icon of “SELL” indicates that there is a need for “purchase”. The “SHARE” icon indicates that there is a “borrow” need.

  When the user presses the thumbnail of the article on which the “SELL” icon is displayed, the operation GUI 2100 switches to the screen shown in FIG. 22A. In the operation GUI shown in FIG. 22A, the SELL button is highlighted.

  When the user presses the SELL button, the article management module 311 refers to the article list 322 and sets “1” in the intention 643 of the record corresponding to the selected article. Further, the article management module 311 transmits an inquiry including the article type 621 and the article name 622 of the record corresponding to the selected article to the management computer 110.

  When receiving the inquiry, the management computer 110 acquires the sale price of the selected article from the server that provides the article trading service based on the article type 621 and the article name 622. The selling price of an article similar to the selected article may be used. The management computer 110 transmits a response including the sale price to the article management module 311.

  The article management module 311 sets a value for the desired sale price 644 of the record corresponding to the selected article based on the response from the management computer 110, and displays a screen as shown in FIG. 22B. In addition, the user can appropriately correct the displayed price. When the user presses the SELL button on the screen illustrated in FIG. 22B, the article management module 311 changes the sale state 645 of the record corresponding to the selected article to “Selling”. The article management module 311 may be connected to a server that performs an article trading service via the management computer 110, and may register information on the selected article.

  When the sale status 645 is changed to “under sale”, the display of the thumbnail of the article on the operation GUI 2100 may be changed.

  When the user presses the thumbnail of the article on which the “SHARE” icon is displayed, the operation GUI 2100 switches to the screen shown in FIG. 22A. At this time, the SHARE button is highlighted instead of the SELL button.

  When the user presses the SHARE button, the article management module 311 refers to the article list 322 and sets “1” in the intention 639 of the record corresponding to the selected article. Further, the article management module 311 transmits a delivery procedure request including the address of the warehouse and the article type 621 and the size 623 of the record corresponding to the selected article to the management computer 110. The address of the warehouse can be obtained from the location information 324.

  When receiving the delivery procedure request, the management computer 110 acquires, from the house information 421, the address of the house 100 that has transmitted the request. The management computer 110 transmits the address of the house 100, the address of the warehouse, the item type 621, and the size 623 to a server that provides a delivery service. As a result, a delivery procedure can be performed. The management computer 110 transmits a response from the server to the article management module 311.

  The article management module 311 updates the distribution status 609 of the record corresponding to the selected article based on the response. For example, the address of the house 100 is set as the collection destination 648, the address of the warehouse is set as the delivery destination 651, and arbitrary values are set as the scheduled collection date 649 and the scheduled delivery date 652. After the delivery of the article is completed, the article management module 311 updates the storage 605 of the record corresponding to the selected article.

  As illustrated in FIG. 22C, the article management module 311 may display as many icons as the number of users who need the article based on the number of needs 642 of the record corresponding to the selected article. At this time, the article management module 311 may display the details of the user who needs the article in cooperation with the management computer 110.

  In the operation GUI 2100, the icon of “SEND” is highlighted on the thumbnail of the article for which the determination result in step S405 is NO. The “SEND” icon indicates an article for which external storage of the article is recommended.

  When the user presses the thumbnail of the article on which the “SEND” icon is displayed, the operation GUI 2100 switches to the screen illustrated in FIG. 22A. At this time, the SEND button is highlighted instead of the SELL button.

  When the user presses the SEND button, the article management module 311 transmits a delivery procedure request including the warehouse address and the article type 621 and size 623 of the record corresponding to the selected article to the management computer 110.

  When receiving the delivery procedure request, the management computer 110 acquires, from the house information 421, the address of the house 100 that has transmitted the request. The management computer 110 transmits the address of the house 100, the address of the warehouse, the item type 621, and the size 623 to a server that provides a delivery service. As a result, a delivery procedure can be performed. The management computer 110 transmits a response from the server to the article management module 311.

  The article management module 311 updates the distribution status 609 of the record corresponding to the selected article based on the response. For example, the address of the house 100 is set as the collection destination 648, the address of the warehouse is set as the delivery destination 651, and arbitrary values are set as the scheduled collection date 649 and the scheduled delivery date 652. After the delivery of the article is completed, the article management module 311 updates the storage 605 of the record corresponding to the selected article.

  As described above, by managing the use state and the like of the articles using the article list 322, it is possible to determine the articles to be in the house 100 and the articles to be moved or shared to the warehouse. This can prevent unnecessary articles from increasing in the house 100.

(5) Method of Proposing Maintenance of Articles FIG. 23 is a flowchart illustrating an example of processing performed by the article management module 311 of the first embodiment to propose maintenance of articles. 24A and 24B are diagrams illustrating an example of an operation state of the operation GUI according to the first embodiment.

  The article management module 311 starts the processing described below periodically or when receiving an instruction from the user.

  The article management module 311 starts an article loop process (step S501). The processing in step S501 is the same as the processing in step S401.

  Next, the article management module 311 determines whether the total use time of the target article is greater than a threshold (Step S502).

  Specifically, the article management module 311 determines whether the total use time 628 of the record corresponding to the target article is larger than a threshold. In the present embodiment, it is assumed that a threshold value of the total use time is set for each type of article. The information on the threshold may be held by the computer 120 or may be held by the management computer 110.

  If it is determined that the total use time of the target article is equal to or less than the threshold, the article management module 311 proceeds to step S508.

  When it is determined that the total use time of the target article is larger than the threshold, the article management module 311 inquires of the management computer 110 about the maintenance information of the target article (step S503). The inquiry includes the article type 621 and the article name 622 of the record corresponding to the target article.

  The article management module 311 shifts to a wait state until a response is received from the management computer 110. When receiving the inquiry, the management computer 110 executes the following processing.

  The management computer 110 refers to the maintenance information 423 and searches for a record in which the article type 1302 and the article name 1303 match the article type 621 and the article name 622 included in the inquiry. The management computer 110 transmits a response including the searched record to the computer 120.

  When receiving a response from the management computer 110, the article management module 311 determines whether a maintenance service for the target article exists (step S504).

  Specifically, the article management module 311 determines whether there is a maintenance service for the target article based on the maintenance service 1305 of the record included in the response. At this time, the article management module 311 sets the maintenance service 1305 of the record included in the response as the maintenance service 635.

  If it is determined that there is no maintenance service for the target article, the article management module 311 proceeds to step S506.

  If it is determined that there is a maintenance service for the target article, the article management module 311 outputs maintenance proposal information related to the maintenance service (Step S505). Thereafter, the article management module 311 proceeds to step S506. The maintenance proposal information includes a manual 1304 of a record included in the response and a maintenance service 1305.

  After the determination result of step S504 is NO or the process of step S505 is completed, the article management module 311 determines whether or not a replacement article exists in the target article (step S506).

  Specifically, the article management module 311 determines whether there is an exchange article for the target article based on the exchange article 1306 of the record included in the response.

  When it is determined that there is no exchange article in the target article, the article management module 311 proceeds to step S508.

  If it is determined that a replacement article is present in the target article, the article management module 311 outputs replacement article replacement proposal information (step S507). Thereafter, the article management module 311 proceeds to step S508. The exchange proposal information includes the exchange item 1306 of the record included in the response.

  In step S507, the article management module 311 determines whether or not the processing has been completed for all the articles registered in the article list 322 (step S508).

  If it is determined that the processing has not been completed for all the articles registered in the article list 322, the article management module 311 returns to step S501 and performs the same processing.

  When it is determined that the processing has been completed for all the articles registered in the article list 322, the article management module 311 ends the processing.

  When “none” is set in the maintenance service 635 of the target article record, the article management module 311 may omit the processing of step S503, step S504, and step S505, and may proceed to step S506.

  In the operation GUI 2100, a maintenance icon is highlighted on the thumbnail of the article for which at least one of the determination results in step S504 and step S506 is YES.

  When the user presses the thumbnail of the article on which the maintenance icon is displayed, the operation GUI 2100 switches to the screen shown in FIG. 22A. At this time, the MAINTENANCE button is highlighted instead of the SELL button. Further, when the user presses the MAINTENANCE button, a menu as shown in FIG. 24A is developed. In the present embodiment, three menus, MANUAL, STORES, and MAINTENANCE SERVICE are displayed.

  When the user presses MANUAL, the article management module 311 acquires a manual from an external server or the like based on the manual 1304 included in the maintenance proposal information, and displays the acquired manual on the operation GUI 2100.

  When the user presses STORES, the article management module 311 displays a part of the exchange article as a thumbnail as shown in FIG. 24B based on the exchange article 1306 included in the exchange proposal information. In the case where the article management module 311 manages the history of the exchanged article, the exchanged article that is normally purchased and the new article or the offer article of the external service are displayed so as to be understood. When the user presses the MORE button, the screen is switched to a screen listing all products.

  When the user presses MAINTENANCE SERVICE, the article management module 311 acquires maintenance service information from an external server or the like based on the maintenance service 1305 included in the maintenance proposal information, and displays the acquired maintenance service information on the operation GUI 2100. .

(6) Method of Proposing Exchange of Articles FIG. 25 is a flowchart illustrating an example of a process of suggesting exchange of articles performed by the article management module 311 according to the first embodiment. 26A and 26B are diagrams illustrating an example of the operation state of the operation GUI according to the first embodiment.

  The article management module 311 starts the processing described below periodically or when receiving an instruction from the user.

  The article management module 311 starts an article loop process (step S601). The processing in step S601 is the same as the processing in step S401.

  Next, the article management module 311 determines whether the total use time of the target article is greater than a threshold (Step S602). The processing in step S602 is the same as the processing in step S502. Note that the threshold used for the determination may be the same as the threshold used in step S502, or may be different.

  If it is determined that the total use time of the target article is equal to or less than the threshold, the article management module 311 proceeds to step S606.

  If it is determined that the total use time of the target article is greater than the threshold, the article management module 311 calculates the number of people using the target article (Step S603).

  Specifically, the article management module 311 refers to the article list 322 and counts the number of pieces of identification information stored in the user 631 of the record corresponding to the target article.

  Next, the article management module 311 inquires of the management computer 110 about an exchange article (step S604). The inquiry includes the article type 621 and the article name 622 of the record corresponding to the target article, and the number of users.

  The article management module 311 shifts to a wait state until a response is received from the management computer 110. When receiving the inquiry, the management computer 110 executes the following processing.

  The management computer 110 searches for a record in which the article type 1402 matches the article type 621 included in the inquiry. Further, the management computer 110 specifies a record in which the number of employees 1404 is equal to or less than the number of employees included in the inquiry. The management computer 110 transmits a response including the specified record size 1403 and the article name 1405 to the computer 120.

  When receiving a response from the management computer 110, the article management module 311 outputs replacement article replacement proposal information (step S605). Thereafter, the article management module 311 proceeds to step S606. The replacement proposal information includes the size 1403 and the article name 1405 included in the response.

  In step S606, the article management module 311 determines whether the processing has been completed for all the articles registered in the article list 322 (step S606).

  If it is determined that the processing has not been completed for all the articles registered in the article list 322, the article management module 311 returns to step S601 and performs the same processing.

  When it is determined that the processing has been completed for all the articles registered in the article list 322, the article management module 311 ends the processing.

  In the operation GUI 2100, the icon of “REPLACE” is highlighted on the thumbnail of the article for which the determination result in step S602 is YES.

  When the user presses the thumbnail of the article on which the “REPLACE” icon is displayed, the operation GUI 2100 switches to the screen shown in FIG. 22A. At this time, the REPLACE button is highlighted instead of the SELL button.

  When the user presses the REPLACE button, the article management module 311 transmits the replacement proposal information related to the selected article to the management computer 110.

  The management computer 110 transmits the replacement proposal information to the server that provides the article trading service, and acquires the sale information including the user who is exhibiting the target article and the sale price. Further, the management computer 110 transmits the replacement proposal information to the server that provides the sales service of the article, and acquires the sales information including the selling price of the target article. The management computer 110 transmits the sale information and the sales information to the article management module 311.

  The article management module 311 displays the screen shown in FIG. 26A based on the sale information and the sale information. The user selects the purchase destination of the target article by pressing the thumbnail. For example, when the upper thumbnail in FIG. 26A is pressed, the screen is switched to the screen in FIG. 26B. When the PURCHASE button is pressed on this screen, a sales transaction is established.

  In this way, by exchanging items using the item list 322, only necessary and useful items can be placed in the house 100.

(7) Method of Managing Articles According to Event FIG. 27 is a flowchart illustrating an example of processing performed by the article management module 311 according to the first embodiment to present a missing article of an event. 28A to 28E are diagrams illustrating an example of the operation state of the operation GUI according to the first embodiment.

  The article management module 311 starts the processing described below periodically or when receiving an instruction from the user.

  The article management module 311 starts loop processing of the record of the calendar information 326 (step S701). Here, the article management module 311 refers to the calendar information 326 and selects one target event (record). In this embodiment, the records are selected in order from the upper record.

  The article management module 311 determines whether it is necessary to prepare articles related to the target event (Step S702).

  Specifically, the article management module 311 determines whether the period between the date and time 1001 of the target record and the current date and time is smaller than a predetermined threshold. For example, it is determined whether it is two weeks or less. If the period is smaller than the predetermined threshold, the item management module 311 determines that it is necessary to prepare an item for the target event.

  If it is determined that there is no need to prepare articles related to the target event, the article management module 311 proceeds to step S708.

  If it is determined that it is necessary to prepare articles related to the target event, the article management module 311 inquires of the management computer 110 about the articles necessary for the target event (step S703). The inquiry includes the event name 1002.

  The article management module 311 shifts to a wait state until a response is received from the management computer 110. When receiving the inquiry, the management computer 110 executes the following processing.

  The management computer 110 refers to the event item information 425 and searches for a record in which the event name 1502 matches the event name 1002 included in the inquiry. The management computer 110 transmits a response including the article list 1504 of the searched record to the computer 120.

  When receiving a response from the management computer 110, the article management module 311 specifies a missing article (step S704). Specifically, the following processing is executed.

  The article management module 311 generates an empty missing article list. The item management module 311 selects one item type from the item list 1504 included in the response, and counts the number of records that match the item type selected by the item type 621 among the items stored in the house 100. I do.

  The article management module 311 determines whether the number of records is smaller than the number specified in the article list 1504. If the above-described condition is satisfied, the article management module 311 determines that articles of the article type are insufficient. At this time, the article management module 311 adds a record including the type of the article and the number of additions to the missing article list.

  The article management module 311 performs the same processing for all the article types included in the article list 1504. The above is the description of the processing in step S704.

  Next, the article management module 311 refers to the missing article list and the article list 322 to identify articles that can be moved from the warehouse (step S705). Specifically, the following processing is executed.

  The article management module 311 selects one article type from the missing article list. The article management module 311 refers to the article list 322 and searches the articles stored in the warehouse for a record that matches the article type selected by the article type 621. At this time, the article management module 311 subtracts the number of records retrieved from the number of records added in the missing article list.

  The article management module 311 adds the searched record to the moving article list. The article management module 311 performs the same process for all article types included in the missing article list. The above is the description of the processing in step S705.

  The article management module 311 outputs the specified article as a moving article list (Step S706). Further, the article management module 311 outputs a missing article list (step S707). Thereafter, the article management module 311 proceeds to step S708.

  In step S708, the article management module 311 determines whether or not processing has been completed for all events registered in the calendar information 326 (step S708).

  If it is determined that the processing has not been completed for all the events registered in the calendar information 326, the article management module 311 returns to step S701 and performs the same processing.

  If it is determined that the processing has been completed for all the events registered in the calendar information 326, the article management module 311 ends the processing.

  The article management module 311 highlights, based on the moving article list, thumbnails of articles that need to be moved among the articles displayed on the Warehouse side.

  When the user presses the thumbnail of the article, the screen switches to a screen as shown in FIG. 28A. When the user presses the MOVE button, a menu for selecting a destination is displayed as shown in FIG. 28B. When the user presses the My HOME button, the item management module 311 transmits a delivery procedure request including the item type 621 and the size 623 to the management computer 110. The processing in this case is the same as when the SHARE button is pressed.

  The article management module 311 displays the thumbnails of the missing articles on the upper side of the Warehouse as shown in FIG. 28C based on the missing article list.

  When the user presses the thumbnail of the missing item, the item management module 311 displays a menu of purchasing needs or borrowing needs.

  When the purchase needs are selected, the screen is switched to the screen of FIG. 28D. When the BUY button is pressed on this screen, the article management module 311 transmits a purchase request registration request including the identification information of the house 100, the type of the article, the type of the need, and the desired purchase price to the management computer 110. .

  When the borrowing needs are selected, the screen is switched to the screen of FIG. 28E. When the RENT button is pressed on this screen, the article management module 311 transmits a borrowing need registration request including the identification information of the house 100, the type of the article, the type of the need, and the like to the management computer 110.

(8) Intention and Needs Matching Process FIG. 29 is a flowchart illustrating an example of a needs confirmation process executed by the management computer 110 according to the first embodiment.

  The management computer 110 starts processing described below periodically or when receiving an instruction from the user.

  The management computer 110 acquires a value relating to the intention to dispose of the article from each house 100 (step S801).

  Specifically, the management computer 110 transmits a request to acquire the lending setting 607 and the selling setting 608 of each record of the article list 322 to the computer 120.

  When receiving the request, the article management module 311 refers to the intention 639 and the intention 643 of each record in the article list 322 and searches for a record in which at least one of the intention 639 and the intention 643 is set to “1”. . The article management module 311 transmits a response including the article type 621, the article name 622, the intention 639, the intention 643, and the desired sale price 644 of the searched record to the management computer 110 together with the identification information of the house 100.

  Next, the management computer 110 generates an intention list in the memory 402 (Step S802).

  More specifically, a record including house identification information, article type 621, article name 622, intention 639, intention 643, and desired sale price 644 is added to the intention list.

  Next, the management computer 110 starts loop processing of the record of the intention list (step S803). Here, the management computer 110 selects one target record from the intention list.

  Next, the management computer 110 determines whether the lending intention of the article corresponding to the target record is valid (step S804).

  Specifically, the management computer 110 determines whether or not the intention 639 of the target record is “1”. When the intention 639 is “1”, the management computer 110 determines that the lending intention of the article is valid.

  If it is determined that the lending intention of the article corresponding to the target record is not valid, the management computer 110 proceeds to step S808.

  When it is determined that the lending intention of the article corresponding to the target record is valid, the management computer 110 specifies a borrowing need (step S805).

  Specifically, the management computer 110 refers to the needs information 422 and, from the records in which the needs type 1202 is “borrowed”, sets the product type 1203 and the product name 1204 to the product type 621 and product name 622 of the target record. Search for matching records. The management computer 110 counts the number of searched records.

  Next, the management computer 110 transmits a lending intention update request to the house 100 holding the article corresponding to the target record (step S806). Note that the rental intention update request includes the type of the article, the name of the article, the number of searched records, and the identification information of the house 100 having the borrowing needs.

  When receiving the rental intention update request, the article management module 311 searches for a record in which the article type 621 and the article name 622 match the article type and the article name included in the request. The article management module 311 sets the number of records included in the lending intention update request to the number of needs 642 of the retrieved records. In addition, the article management module 311 outputs a notification including the article name and the identification information of the house 100 having the borrowing needs.

  Next, the management computer 110 transmits the borrowing proposal information to the house 100 having the borrowing needs (Step S807). The borrowing proposal information includes the identification information of the house 100 holding the article, the article type, and the article name. Thereafter, the management computer 110 proceeds to Step S808.

  When receiving the borrowing proposal information, the article management module 311 outputs the information.

  If the decision result in the step S804 is NO, or after the process in the step S807 is executed, the management computer 110 decides whether or not the intention to sell the article corresponding to the target record is valid (step S808). .

  Specifically, the management computer 110 determines whether or not the intention 643 of the target record is “1”. When the intention 643 is “1”, the management computer 110 determines that the intention to sell the article is valid.

  If it is determined that the intention to sell the article corresponding to the target record is not valid, the management computer 110 proceeds to step S812.

  When it is determined that the intention to sell the article corresponding to the target record is valid, the management computer 110 specifies a purchase need (step S809).

  Specifically, the management computer 110 refers to the needs information 422 and, from the records in which the needs type 1202 is “buy”, sets the product type 1203 and the product name 1204 to the product type 621 and product name 622 of the target record. Search for matching records. The management computer 110 counts the number of searched records.

  Next, the management computer 110 transmits a sale intention update request to the house 100 holding the article corresponding to the target record (step S810). The sale intention update request includes the type of the article, the name of the article, the number of searched records, the desired purchase price, and the identification information of the house 100 having the purchase needs.

  When receiving the sale intention update request, the article management module 311 searches for a record in which the article type 621 and the article name 622 match the article type and the article name included in the request. The article management module 311 sets the number of records included in the sale intention update request to the number of needs 647 of the retrieved records. In addition, the article management module 311 outputs a notification including an article name, a desired purchase price, and identification information of the house 100 having a borrowing need.

  Next, the management computer 110 transmits the purchase proposal information to the house 100 having the purchase needs (step S811). The purchase proposal information includes the identification information of the house 100 holding the article, the article type, the article name, and the selling price. Thereafter, the management computer 110 proceeds to Step S812.

  In step S812, the management computer 110 determines whether the processing has been completed for all records in the intention list (step S812).

  If it is determined that the processing has not been completed for all the records in the intention list, the management computer 110 returns to step S803 and executes the same processing.

  When it is determined that the processing has been completed for all records in the intention list, the management computer 110 ends the processing. At this time, the management computer 110 deletes the intention list from the memory 402.

  FIG. 30 is a diagram illustrating an example of the operation GUI according to the first embodiment.

  As shown in FIG. 30, the article management module 311 displays a menu screen for setting the management intention of the article at an arbitrary timing. For example, when receiving an input regarding the intention to sell, the article management module 311 updates the intention 643 of the record corresponding to the selected article.

  Note that the present invention is not limited to the above-described embodiment, and includes various modifications. Further, for example, in the above-described embodiment, the configuration has been described in detail for easy understanding of the present invention, and the present invention is not necessarily limited to the configuration including all the described configurations. Further, a part of the configuration of each embodiment can be added, deleted, or replaced with another configuration.

  In addition, each of the above-described configurations, functions, processing units, processing means, and the like may be partially or entirely realized by hardware, for example, by designing an integrated circuit. The present invention can also be realized by software program codes for realizing the functions of the embodiments. In this case, a storage medium storing the program code is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the program code itself and the storage medium storing the program code constitute the present invention. As a storage medium for supplying such a program code, for example, a flexible disk, a CD-ROM, a DVD-ROM, a hard disk, an SSD (Solid State Drive), an optical disk, a magneto-optical disk, a CD-R, a magnetic tape, A non-volatile memory card, ROM, or the like is used.

  Further, the program code for realizing the functions described in the present embodiment can be implemented by a wide range of programs or script languages such as assembler, C / C ++, perl, Shell, PHP, and Java (registered trademark).

  Further, by distributing the program code of the software for realizing the functions of the embodiment via a network, the program code is stored in a storage means such as a hard disk or a memory of a computer or a storage medium such as a CD-RW or a CD-R. Alternatively, a processor included in a computer may read and execute a program code stored in the storage unit or the storage medium.

  In the above-described embodiment, the control lines and the information lines are considered to be necessary for the description, and do not necessarily indicate all the control lines and the information lines on the product. All components may be interconnected.

100 house 105 external network 110 management computer 120 computer 130 image acquisition device 150 home network 301, 401 processor 302, 402 memory 303, 403 network interface 304, 404 storage device 311 item management module 321 image analysis information 322 item list 323 excluded item list 324 Location information 325 Person information 326 Calendar information 411 Overall control module 421 House information 422 Needs information 423 Maintenance information 424 Replacement article information 425 Event article information 426 Template image information 2100 Operation GUI

Claims (10)

An article management system including an image acquisition device and a computer for acquiring an image,
The computer has a processor, a memory connected to the processor, and a network interface connected to the processor,
The memory stores an article list that manages articles existing in the space where the image acquisition device is installed,
The article list includes a record including a name of the article, information indicating the use state of the article, and a field for storing the number of times the article has been used ,
The image acquisition device,
Acquiring image data of the space,
Transmitting the image data to the computer,
The calculator is:
By the analysis of the image data, to identify the article present in the space and the use state of the article,
Calculating the number of uses of the article based on the result of the analysis of the image data ,
Updating the article list based on the number of uses of the article ,
Based on the article list, it generates display data for displaying the use status of the article during the image data, and outputs the display data,
An article management system, characterized by referring to the article list, identifying a target article whose number of uses is smaller than a threshold, generating proposal information for suggesting a disposal method of the target article, and outputting the proposal information .   The article management system according to claim 1,
  The article management system, wherein the proposal information includes information for proposing at least one of lending the target article, selling the target article, and changing a storage location of the target article.   An article management system including an image acquisition device and a computer for acquiring an image,
  The computer has a processor, a memory connected to the processor, and a network interface connected to the processor,
  The memory stores an article list that manages articles existing in the space where the image acquisition device is installed,
  The article list includes a record including a name of the article, information indicating a use state of the article, and a field for storing a use time of the article,
  The image acquisition device,
  Acquiring image data of the space,
  Transmitting the image data to the computer,
  The calculator is:
  By the analysis of the image data, to identify the article present in the space and the use state of the article,
  Calculating the use time of the article based on the result of the analysis of the image data,
  Updating the article list based on the usage time of the article,
  Based on the article list, generate display data for displaying the use state of the article in the image data, and output the display data,
  Referencing the article list, specifying a target article whose use time is greater than a threshold, generating proposal information for proposing replacement of parts used for the target article or use of a maintenance service, and outputting the proposal information. An article management system characterized by the following.   An article management system including an image acquisition device and a computer for acquiring an image,
  The computer has a processor, a memory connected to the processor, and a network interface connected to the processor,
  The memory stores an article list that manages articles existing in the space where the image acquisition device is installed,
  The article list includes a record including a name of the article, information indicating a use state of the article, and a field for storing a use time of the article,
  The image acquisition device,
  Acquiring image data of the space,
  Transmitting the image data to the computer,
  The calculator is:
  By the analysis of the image data, to identify the article present in the space and the use state of the article,
  Calculating the use time of the article based on the result of the analysis of the image data,
  Updating the article list based on the usage time of the article,
  Based on the article list, generate display data for displaying the use state of the article in the image data, and output the display data,
  An article management system characterized by referring to the article list, identifying a target article whose use time is longer than a threshold, generating proposal information for proposing replacement of the target article, and outputting the proposal information.   An article management system including an image acquisition device and a computer for acquiring an image,
  Holds event item information that manages items necessary for the event,
  The computer has a processor, a memory connected to the processor, and a network interface connected to the processor,
  The memory stores an article list that manages articles existing in the space where the image acquisition device is installed,
  The article list includes a record including a field for storing information indicating a name of the article and a use state of the article,
  The image acquisition device,
  Acquiring image data of the space,
  Transmitting the image data to the computer,
  The calculator is:
  By the analysis of the image data, to identify the article present in the space and the use state of the article,
  Update the article list based on the result of the analysis of the image data,
  Based on the article list, generate display data for displaying the use state of the article in the image data, and output the display data,
  With reference to the event item information, an item necessary for the first event is specified, and with reference to the item list, a list of items present in an external storage location among the items necessary for the first event. Generate a moving goods list that is
  With reference to the article list, among the articles required for the first event, generate a missing article list that is a list of articles that need to be obtained using an external service,
  An article management system for outputting the moving article list and the missing article list.   An article management method in an article management system including an image acquisition device and a computer for acquiring an image,
  The computer has a processor, a memory connected to the processor, and a network interface connected to the processor,
  The memory stores an article list that manages articles existing in the space where the image acquisition device is installed,
  The article list includes a name of the article, information indicating the use state of the article, and a record including a field for storing the number of times the article has been used,
  The management method of the article,
  A first step in which the image acquisition device acquires image data of the space;
  A second step in which the image acquisition device transmits the image data to the computer;
  A third step in which the computer analyzes the image data to identify an article present in the space and a use state of the article;
  A fourth step of the computer updating the article list based on a result of the analysis of the image data;
  A fifth step in which the calculator generates display data indicating a use state of the article in the image data based on the article list;
  A sixth step of the computer outputting the display data,
  The fourth step is:
  The computer calculates the number of uses of the article based on the result of the analysis of the image data,
  The computer updates the article list based on the number of uses of the article,
  The article management method further comprises:
  The computer refers to the article list, the number of times of use to identify a target article smaller than a threshold,
  The computer generates proposal information for proposing a disposal method of the target article,
  Outputting the proposal information by the computer.   It is a management method of the article of Claim 6, Comprising:
  The article management method, wherein the proposal information includes information for proposing at least one of lending the target article, selling the target article, and changing a storage location of the target article.   An article management method in an article management system including an image acquisition device and a computer for acquiring an image,
  The computer has a processor, a memory connected to the processor, and a network interface connected to the processor,
  The memory stores an article list that manages articles existing in the space where the image acquisition device is installed,
  The article list includes a record including a name of the article, information indicating a use state of the article, and a field for storing a use time of the article,
  The management method of the article,
  A first step in which the image acquisition device acquires image data of the space;
  A second step in which the image acquisition device transmits the image data to the computer;
  A third step in which the computer analyzes the image data to identify an article present in the space and a use state of the article;
  A fourth step of the computer updating the article list based on a result of the analysis of the image data;
  A fifth step in which the calculator generates display data indicating a use state of the article in the image data based on the article list;
  A sixth step of the computer outputting the display data,
  The fourth step is:
  The computer calculates a use time of the article based on a result of the analysis of the image data,
  The computer updating the article list based on the usage time of the article,
  The article management method further comprises:
  Wherein the calculator refers to the article list to identify a target article whose use time is greater than a threshold,
  A step in which the computer generates proposal information that proposes replacement of parts used for the target article or use of a maintenance service,
  Outputting the proposal information by the computer.   An article management method in an article management system including an image acquisition device and a computer for acquiring an image,
  The computer has a processor, a memory connected to the processor, and a network interface connected to the processor,
  The memory stores an article list that manages articles existing in the space where the image acquisition device is installed,
  The article list includes a record including a name of the article, information indicating a use state of the article, and a field for storing a use time of the article,
  The management method of the article,
  A first step in which the image acquisition device acquires image data of the space;
  A second step in which the image acquisition device transmits the image data to the computer;
  A third step in which the computer analyzes the image data to identify an article present in the space and a use state of the article;
  A fourth step of the computer updating the article list based on a result of the analysis of the image data;
  A fifth step in which the calculator generates display data indicating a use state of the article in the image data based on the article list;
  A sixth step of the computer outputting the display data,
  The fourth step is:
  The computer calculates a use time of the article based on a result of the analysis of the image data,
  The computer updating the article list based on the usage time of the article,
  The article management method further comprises:
  Wherein the calculator refers to the article list to identify a target article whose use time is greater than a threshold,
  The computer generates proposal information for proposing replacement of the target article,
  Outputting the proposal information by the computer.   An article management method in an article management system including an image acquisition device and a computer for acquiring an image,
  The article management system holds event article information for managing articles required for an event,
  The computer has a processor, a memory connected to the processor, and a network interface connected to the processor,
  The memory stores an article list that manages articles existing in the space where the image acquisition device is installed,
  The article list includes a record including fields for storing information indicating a name of the article and a use state of the article,
  The management method of the article,
  The image acquisition device acquires image data of the space,
  The image acquisition device, transmitting the image data to the computer,
  The computer, by analyzing the image data, to identify the article present in the space and the use state of the article,
  The computer updates the article list based on a result of the analysis of the image data,
  The computer, based on the article list, a step of generating display data indicating the use state of the article in the image data,
  The computer outputs the display data,
  A step in which the computer refers to the event item information to specify an item necessary for the first event;
  The computer refers to the item list, and among the items necessary for the first event, generating a moving item list that is a list of items present in an external storage location;
  A step in which the computer refers to the article list and generates a missing article list, which is a list of articles that need to be obtained by using an external service, among articles necessary for the first event;
  Outputting the moving article list and the missing article list by the computer.