JP7618138B2 - Flight System - Google Patents

Description

The present invention relates to a flight system equipped with an aircraft.

Conventionally, monitoring systems have been proposed for monitoring luggage stored in warehouses and the like. One example of such a monitoring system is disclosed in Patent Document 1. The luggage monitoring system disclosed in Patent Document 1 is a system that uses a drone that flies within a warehouse where pallets carrying luggage are stacked in multiple layers in a flat-layout warehouse.

The drone described above flies within the warehouse while using sensors to avoid collisions with obstacles within the warehouse, and acquires image information using a camera. The acquired image information can be transmitted to an external device to monitor the condition of the luggage within the warehouse. In particular, when the drone approaches tilted luggage or pallets, the camera mounted on the drone acquires image information, making it possible to detect the tilted luggage or pallets.

JP 2018-43815 A

If object recognition of luggage could be effectively utilized in facilities where luggage is stored on storage shelves, it would be useful for monitoring luggage, etc. The present invention aims to provide a flight system that makes it possible to effectively utilize object recognition of luggage in facilities where luggage is stored on storage shelves.

To achieve the above object, the flight system of the present invention is a flight system that includes an aircraft having an imaging means and is used in a facility where luggage is stored on storage shelves, and is configured such that the aircraft performs a flight operation to photograph the luggage while flying along a predetermined flight route, and performs object recognition of the luggage for each of the photographed images obtained by the flight operation and the luggage images, which are images of the luggage acquired in advance.

This configuration makes it possible to effectively utilize object recognition of luggage in facilities where luggage is stored on storage shelves. More specifically, the luggage image may be acquired as an image showing the luggage in a normal loading state. Note that the "loading state" refers to the state (position or orientation) of how the luggage is loaded on the storage shelf.

More specifically, the above configuration may be such that the luggage image is an image captured when the luggage is placed on the storage shelf. More specifically, the above configuration may be such that a difference between the placement state of the luggage in the captured image and the placement state of the luggage in the luggage image is recognized based on the result of the object recognition.

More specifically, the difference may be the difference between the orientation of the luggage in the captured image and the orientation of the luggage in the luggage image.

More specifically, the above configuration may be configured to notify a predetermined notification destination of the detection result when the difference exceeds a predetermined value.

More specifically, the above configuration may be such that the aircraft photographs the area below while flying along the flight route, and based on the information obtained from the photographs, detects any abnormal conditions related to the position or orientation of the luggage, including the luggage having fallen from the storage shelf, and creates a distribution map of the locations where the abnormal conditions have occurred based on the results of the abnormal condition detection process.

The monitoring system according to the present invention is useful for monitoring luggage in facilities where luggage is stored on storage shelves.

FIG. 1 is a diagram illustrating an example of a configuration of a logistics facility. FIG. 2 is a top view of the storage shelf and the article transport device. FIG. 2 is a front view of the article transport device. FIG. 4 is a diagram showing an example of the configuration of a support; FIG. 1 is a block diagram showing an example of a configuration of a monitoring system. FIG. 1 is an external view showing an example configuration of a drone. 4 is a flowchart showing the overall operation of the monitoring system. 13 is a flowchart showing specific operations in the second flight operation. 13 is a flowchart showing specific operations in the third flight operation. FIG. 2 is a schematic diagram illustrating an example of a first flight operation. FIG. 2 is a schematic diagram illustrating an example of a second flight operation. FIG. 11 is a schematic diagram illustrating an example of a third flight operation. FIG. 13 is an explanatory diagram regarding deviation of the orientation of luggage.

The following describes the present invention using the first and second embodiments as examples and with reference to the drawings.

(1) First embodiment 1. Overall configuration of a logistics facility First, a first embodiment of the present invention will be described. Fig. 1 shows an example of the configuration of a logistics facility installed in a warehouse or the like. Note that the monitoring system according to this embodiment will be described by taking an example in which the monitoring system is installed in a logistics facility 100 shown in Fig. 1.

The logistics facility 100 includes storage shelves 10, an inbound/outbound conveyor 20, running rails 30, transport carts 35, a picking conveyor 40, an automatic sorting device 50, a shipping waiting area 55, and a truck berth 60. Note that in FIG. 1, the roof and other components are omitted to make the layout of the logistics facility 100 easier to see. The logistics facility 100 also includes an item transport device 15 (see FIG. 2) that moves items into and out of the storage shelves 10 and the inbound/outbound conveyor 20. Note that the storage shelves 10 and the item transport device 15 will be described in detail later.

The logistics facility 100 further includes a management device that controls the operation of the item transport device 15, the inbound/outbound conveyor 20, the transport cart 35, etc. The management device controls the operation of each part so that the item transport device 15 unloads luggage from the storage shelf 10 and transports it to the shipping waiting area 55 via the inbound/outbound conveyor 20, the transport cart 35, the picking conveyor 40, and the automatic sorting device 50. The management device also controls the operation of each part so that the received luggage is stored in the storage shelf 10 by the item transport device 15 via the transport cart 35 and the inbound/outbound conveyor 20.

2. Storage Shelf and Item Transport Device Next, the storage shelf 10 and item transport device 15 will be described in more detail. Fig. 2 is a top view of the vicinity of the storage shelf 10. As shown in this figure, in the logistics facility 100, item transport devices 15A to 15C (hereinafter sometimes collectively referred to as "item transport device 15") are arranged close to the storage shelf 10. Fig. 3 is a front view of the storage shelf 10 and item transport device 15.

In the following description, the direction along the path of movement of the item transport device 15 is referred to as the "left-right direction X." Furthermore, of the directions perpendicular to the path of movement of the item transport device 15, the direction along the vertical direction is referred to as the "up-down direction Z," and the direction along the horizontal direction is referred to as the "front-rear direction Y." The up-down direction Z and the left-right direction X correspond to the up-down, left-right, and right directions on the paper surface of FIG. 3. The front-rear direction Y corresponds to the direction perpendicular to the paper surface of FIG. 3. Furthermore, the side of the item transport device 15 with respect to the storage shelf 10, i.e., the side in the direction in which the luggage is removed, is referred to as the "front side," and the opposite side is referred to as the "rear side."

The storage shelf 10 stores luggage M. As shown in FIG. 3, the storage shelf 10 has a plurality of support posts 102 and a plurality of shelf boards 104. The plurality of support posts 102 are erected in different positions in the left-right direction X along the up-down direction Z. The plurality of shelf boards 104 are arranged in different positions in the up-down direction Z along the left-right direction X. Each shelf board 104 is fixed across a pair of support posts 102 arranged separately in the left-right direction X. A storage section S for storing luggage M is formed as a space surrounded by a pair of support posts 102 adjacent to each other in the left-right direction X and a pair of shelf boards 104 adjacent to each other in the up-down direction Z. The plurality of storage sections S are arranged in a state of being aligned vertically and horizontally.

The luggage M stored on the storage shelf 10 is stored while placed on the support D. An example of the configuration of the support D is shown in FIG. 4. The support D has multiple tabletops Db and multiple legs Da. The tabletops Db extend in a direction perpendicular to the direction in which the legs Da extend, and are arranged on the legs Da in a direction perpendicular to the direction in which the legs Da are arranged. The support D is a so-called skid, which consists of a tabletop Db and legs Da and does not have a bottom panel. A typical example of a skid used as the support D is one made of wood.

As shown in FIG. 2, a pair of storage shelves 10 are provided facing each other in the front-rear direction Y, sandwiching one item transport device 15 that moves on one traveling rail 16. A number of units each consisting of one item transport device 15 and a pair of storage shelves 10 facing each other in the front-rear direction Y, sandwiching the item transport device 15, are arranged along the front-rear direction Y.

More specifically, as shown in FIG. 2, each unit is made up of item transport device 15A and storage shelves 10A and 10B sandwiching it in the front-to-rear direction Y, item transport device 15B and storage shelves 10C and 10D sandwiching it in the front-to-rear direction Y, and item transport device 15C and storage shelves 10E and 10F sandwiching it in the front-to-rear direction Y.

The item transport device 15 is configured as a stacker crane. As shown in FIG. 3, the item transport device 15 has a traveling cart 152, a mast 154, a lifting platform 156, and a transfer device 158.

The traveling cart 152 travels on traveling rails 16 installed on the floor surface along the left-right direction X. This allows the item transport device 15A to travel in the space between the front sides of the storage shelves 10A and 10B, the item transport device 15B to travel in the space between the front sides of the storage shelves 10C and 10D, and the item transport device 15C to travel in the space between the front sides of the storage shelves 10E and 10F.

The masts 154 are provided in pairs. The pair of masts 154 are erected in the up-down direction Z at both ends of the left-right direction X of the traveling carriage 152. The lifting platform 156 rises and falls along the masts 154. The transfer device 158 is placed on the lifting platform 156. The transfer device 158 is configured to be able to move freely in and out of the lifting platform 156. The transfer device 158 is configured with a sliding fork having a pair of slidable fork parts.

The item transport device 15 having the transfer device 158 performs unloading and scooping processes for the storage section S and the loading/unloading conveyor 20, and transfers the items to and from the storage section S and the loading/unloading conveyor 20. Here, the unloading process is a process of unloading the item M placed on the sliding fork, and the scooping process is a process of scooping up the item M and placing it on the sliding fork. In these unloading and scooping processes, the transfer device 158 supports the item M from below while loading and unloading the item M from the storage shelf 10. More specifically, the transfer device 158 transfers the item M while supporting it from below, with the sliding fork constituting the transfer device 158 inserted into the gap space Dc (see FIG. 4) between adjacent legs Da of the support body D.

For all luggage M placed on the storage shelf 10, an image (hereafter referred to as "luggage image α" for convenience) is obtained by photographing the luggage M so that the luggage M can be seen on the storage shelf 10, and the luggage image α is managed in association with the luggage M and the position of the storage shelf 10 on which it is placed. The luggage image α is obtained, for example, by photographing the storage shelf 10 from the side (front side) so that the luggage M is visible when the luggage is placed on the storage shelf 10, and is an image that shows the orientation in which the product M is placed.

As shown in FIG. 2, the inlet/outlet conveyors 20 are provided adjacent to the storage shelves 10 in the left-right direction X. The inlet/outlet conveyors 20 are provided in pairs so as to be adjacent to each of a pair of storage shelves 10. One end of each inlet/outlet conveyor 20 faces the traveling rail 16 that constitutes the movement path of the article transport device 15 in the front-rear direction Y. The other end of each inlet/outlet conveyor 20 faces the traveling rail 30 formed in a loop shape. A transport cart 35 runs on the traveling rail 30.

3. Configuration of the Monitoring System In the logistics facility 100, as the monitoring system according to this embodiment, a monitoring system that monitors the state of the packages M stored on the storage shelves 10 is provided. The configuration of the monitoring system will be described below.

Fig. 5 is a block diagram showing the configuration of a monitoring system 70 according to this embodiment. The monitoring system 70 includes a drone (an example of an air vehicle) 71, a management device 72, and a vibration sensor 73. Fig. 6 is an external view showing an example of the configuration of the drone 71 according to this embodiment. The drone 71 shown in Fig. 6 is a quadcopter equipped with four rotors (rotating wings). The drone 71 includes a frame 71A, a central portion 71B, and rotors 71C to 71F.

Frame 71A extends radially from a central position in all four directions, forming the diagonals of a square. Rotors 71C to 71F are positioned at each end of the diagonal frame 71A. The rotation shafts of rotors 71C to 71F are positioned at each vertex of the square.

The central portion 71B is disposed at the lower center of the frame 71A. A first distance sensor 714, a second distance sensor 715, and a camera 716 (an example of an imaging means) are provided in the central portion 71B. The first distance sensor 714 is a sensor that measures the distance to an object located in front of the central portion 71B. The second distance sensor 715 is a sensor that measures the distance to an object located below the central portion 71B. The first distance sensor 714 and the second distance sensor 715 are, for example, ultrasonic sensors. The camera 716 is capable of capturing images below and to the sides of the central portion 71B. The central portion 71B is also provided with a flight controller 711, a battery 718, and the like.

As shown in FIG. 5, the drone 71 has a flight controller 711, an ESC (Electronic Speed Controller) 712, a rotor motor 713, a first distance sensor 714, a second distance sensor 715, a camera 716, a communication unit 717, and a battery 718.

The flight controller 711 sends a control signal to the ESC 712 based on an instruction from the management device 72 received by the communication unit 717, and causes the ESC 712 to drive the rotor motor 713. In this way, the flight controller 711 controls the flight of the drone 71.

The flight controller 711 has a control unit 711A, a sensor group 711B, and a GPS receiver unit 711C. The control unit 711A is configured, for example, by a microcomputer. Information acquired by the sensor group 711B, the GPS receiver unit 711C, the first distance sensor 714, and the second distance sensor 715 is input to the control unit 711A. Based on the information acquired by the sensor group 711B and the GPS receiver unit 711C, the control unit 711A outputs an appropriate control signal to the ESC 712.

The control unit 711A has a memory unit 711A1. The memory unit 711A1 can store, for example, a program executable by the control unit 711A, flight route information described below, and images captured by the camera 716 described below. The control unit 711A also performs machine learning, and is capable of real-time object recognition of surrounding objects (such as the storage shelf 10, the item transport device 15, and baggage M) based on the images captured by the camera 716.

To achieve this object recognition, an algorithm such as YOLO (You Only Look Once) can be used. Note that "machine learning" is a method of autonomously finding laws and rules by repeatedly learning based on given information. In addition, the above machine learning may employ deep learning to smoothly process multidimensional data structures. This "deep learning" is machine learning that uses a multi-layered neural network (an information processing model that mimics the mechanism of the human nervous system).

The sensor group 711B includes, for example, a three-axis gyro sensor, a three-axis acceleration sensor, and a magnetic sensor. The three-axis gyro sensor detects the forward/backward tilt, left/right tilt, and rotational angular velocity to grasp the attitude and movement of the drone 71. The three-axis acceleration sensor detects acceleration in the forward/backward, left/right, and up/down directions. The magnetic sensor detects the orientation.

The GPS receiver 711C continuously receives GPS signals in order to utilize the Global Positioning System (GPS), which can also be used within the logistics facility 100. This allows the drone 71 to detect its own position information in real time using the GPS, and based on this detection result, it can fly correctly according to the flight route information. However, other means may be used as a means for flying the drone 71 according to the flight route information.

ESC712 passes a drive current to rotor motor 713 based on a control signal input from control unit 711A. Rotor motor 713 is provided at each tip of frame 71A shown in FIG. 6. That is, four rotor motors 713 are provided. Each rotating shaft of rotor motor 713 is directly connected to each of rotors 71C to 71F. One ESC712 corresponds to driving one rotor motor 713. That is, four ESCs 712 are provided to match the rotor motors 713.

The battery 718 may be, for example, a lithium polymer secondary battery. Of the four ESCs 712, one ESC 712 has the function of extracting power from the battery 718 and supplying it to the flight controller 711. The flight controller 711 then supplies power to the communication unit 717. In this way, the power supply unit can be omitted, making the aircraft lighter.

By controlling the rotation speed of each of the rotors 71C to 71F driven by the rotor motor 713, the drone 71 can perform flight actions such as ascending and descending, moving laterally in all directions, and rotating.

As shown in FIG. 5, the management device 72 includes a control unit 721, a communication unit 722, a storage unit 723, and a network interface 724. As described above, the management device 72 is not only used in the monitoring system 70, but also controls each device in the logistics facility 100, such as the item transport device 15.

The control unit 721 performs integrated control of the management device 72. The communication unit 722 communicates with the communication unit 717 in the drone 71 via wireless signals. The memory unit 723 is composed of, for example, a large-capacity HDD device or a semiconductor memory device. The memory unit 723 can store, for example, programs executable by the control unit 721, various information sent from the drone 71, a database of luggage M stored in the storage shelf 10, and the like. The network interface 724 is connected to a network NW such as the Internet.

The vibration sensor 73 is placed on or near the storage shelf 10 to detect vibrations. The detection information of the vibration sensor 73 is transmitted to the control unit 721. As a result, if an earthquake or other event occurs causing vibrations that exceed a specified value, the control unit 721 can detect this situation.

4. Abnormalities in luggage stored on the storage shelf If the storage shelf 10 is shaken by an earthquake or the like, there is a possibility that abnormalities will occur in the position of luggage stored on the storage shelf 10. Specifically, if the storage shelf 10 is shaken, luggage M stored in the storage section S of the storage shelf 10 may fall to the floor as shown in Fig. 3. If luggage M falls, it is necessary to check the damage status of the luggage M and return the undamaged luggage M to its original position.

On the other hand, even if the luggage M has not yet fallen, the shaking of the storage shelf 10 may cause the luggage M to shift beyond the allowable range from the correct position in the storage section S (the position where the luggage M is normally stored on the storage shelf 10), making it more likely for the luggage M to fall off the storage shelf 10. When luggage M is in such a state where it is about to fall (gets stuck), it becomes necessary to return the luggage to its original position in order to prevent it from falling.

The monitoring system 70 according to this embodiment performs the operation described below to detect the above-mentioned state in which the luggage M has fallen from the storage shelf 10 to the floor (hereinafter referred to as the "first state") and the state in which the luggage M has shifted beyond the allowable range in the storage shelf 10 (hereinafter referred to as the "second state") as abnormal states related to the position of the luggage M. Since the luggage M in the first state is on the floor and the luggage M in the second state protrudes outward from the side edge of the storage shelf 10, both can be confirmed by looking from above. The allowable range can be set arbitrarily depending on the importance of preventing the luggage M from falling. In FIG. 2, the luggage M in the first state is luggage M1 and the luggage M in the second state is luggage M2, and an example of an abnormal state related to the position of the luggage M on the storage shelf 10 is shown.

5. Operation of the Monitoring System When an earthquake or the like occurs, the monitoring system 70 according to this embodiment is capable of detecting an abnormal state (first state and second state) regarding the position of the luggage M. Hereinafter, the operation of such monitoring system 70 will be described with reference to the flowcharts shown in FIGS.

When the vibration sensor 73 detects vibrations exceeding a specified value due to the occurrence of an earthquake or the like, the operation of the flowchart in FIG. 7 is started. Note that the timing at which this operation is started is not limited to this form, and may be, for example, when a specified operation is performed by an administrator. When this operation is started, the first flight operation of step S10 is first performed. Note that, when the operation of the flowchart in FIG. 7 is started, the drone 71 is stopped at a specified takeoff and landing location.

More specifically, in step S10, the control unit 721 detects the position of the item transport device 15 and transmits a predetermined flight route that follows the front and back sides of the storage shelf 10 and avoids the position of the item transport device 15 to the drone 71 via the communication unit 722. Then, in the drone 71, the control unit 711A stores the predetermined flight route received via the communication unit 717 in the memory unit 711A1. Note that if the flight route is fixed, the route information may be stored in advance in the memory unit 711A1 so that the transmission is omitted.

The control unit 711A then drives and controls the rotor motor 713 to start the ascent of the drone 71, which is stopped at the takeoff and landing location. The control unit 711A then controls the drone 71 to fly along the above flight route at a predetermined altitude. The predetermined altitude is a position higher than the luggage M stored in the topmost storage section S of the storage shelf 10.

Figure 2 shows flight route FR as an example of a flight route. Note that drone 71 flies at a height between storage shelf 10 and ceiling CE, as shown by the dotted arrow in Figure 3, but the flight of drone 71 is obstructed because item transport device 15 is higher than this flight path. For this reason, flight route FR is a route that avoids item transport devices 15A-15C while following the front and back sides of storage shelves 10A-10F.

More specifically, it first enters the gap R1 between storage shelves 10A and 10B, advances along the front sides of storage shelves 10A and 10B, and when it reaches a position near item transport device 15A, it retreats and exits through gap R1. It then enters the gap R2 between storage shelves 10C and 10D, advances along the front sides of storage shelves 10C and 10D, and when it reaches a position near item transport device 15B, it retreats and exits through gap R2. It then enters the gap R3 between storage shelves 10E and 10F, advances along the front sides of storage shelves 10E and 10F, and when it reaches a position near item transport device 15C, it retreats and exits through gap R3.

Then, it advances along the back side of storage shelf 10F, enters gap R4 between storage shelves 10F and 10E, advances along the front side of storage shelves 10F and 10E, and advances to a position near item transport device 15C, then retreats and exits through gap R4. It then enters gap R5 between storage shelves 10D and 10C, advances along the front side of storage shelves 10D and 10C, advances to a position near item transport device 15B, then retreats and exits through gap R5. It then enters gap R6 between storage shelves 10B and 10A, advances along the front side of storage shelves 10B and 10A, advances to a position near item transport device 15A, then retreats and exits through gap R6. It then advances along the back side of storage shelf 10A, and the flight route ends.

In this way, when flying through each gap, the drone 71 advances to a position close to the item transport device and then retreats. This allows the aircraft to fly through the space without colliding with the item transport device. The above flight route covers the space on both the front and rear sides of multiple storage shelves 10. Note that this does not include areas where the rear sides of the storage shelves 10 face each other or areas where the item transport device 15 is located.

While flying along this flight route, the drone 71 continuously captures images of the floor side (below) with the camera 716. The images captured by the camera 716 are stored in the memory unit 711A1. The captured images thus obtained show the front and back sides of the storage shelf 10 along the flight route and the floor surface in the vicinity thereof, and if there is any luggage M that has become abnormal, the luggage M is also captured.

The control unit 711A is capable of performing object recognition of the luggage M in the first and second states from the captured images stored in the memory unit 711A1, thereby detecting abnormal areas where the first and second states have occurred (abnormal areas where an abnormality has occurred regarding the position of the luggage M). Note that it is also possible to distinguish between the first and second states here and detect the abnormal state, but in this embodiment, the first and second states are distinguished in the second flight operation described later.

As described above, in the first flight operation, the monitoring system 70 performs the first flight operation in which the drone 71 (aircraft) flies a predetermined route along the storage shelf 10 while taking pictures of the downward view, and detects abnormal conditions related to the position of the luggage M, including a state in which the luggage M has fallen from the storage shelf 10, based on the information obtained by the photography. The drone 71 detects abnormal areas as described above, and creates a distribution map of the positions where abnormal conditions have occurred by displaying the positions of each detected abnormal area on a map around the storage shelf 10. This distribution map is provided to the manager of the logistics facility 100 (hereinafter simply referred to as the "manager") and can be used for future responses, etc. For example, if the distribution map identifies a location that frequently becomes an abnormal area (i.e., a location where luggage is likely to be displaced), measures can be taken, such as inspecting the storage shelf 10 near that location for abnormalities.

As shown in FIG. 7, after the first flight operation (step S10) is completed, the operation of step S20 is started. However, if no abnormal baggage M is detected during the first flight operation, the administrator is notified of this and the operations after step S20 are omitted. Furthermore, if the purpose of the monitoring system 70 is achieved by obtaining the distribution map, etc., the operations after step S20 may be omitted.

In step S20, a second flight operation is performed. The second flight operation is performed along a route that overlaps with the flight route of the first flight operation. More specifically, the operation of the flowchart shown in FIG. 8 is performed.

The operation shown in FIG. 8 is performed for each of the routes of the gaps between the pair of front sides of the storage shelves 10 in the above-mentioned predetermined flight route, and for each route on the rear side of the storage shelves 10. In the example of FIG. 2, the operation is performed for each route of the gaps R1 to R6, and for each route on the rear side of the storage shelves 10A and 10F.

When the operation of the flowchart in FIG. 8 is started, first in step S201, when the control unit 711A detects that the drone 71 has moved into the first abnormal area, the control unit 711A descends the drone 71 until the luggage M is detected by the first distance sensor 714. Then, in step S202, the control unit 711A flies the drone 71 over the detected luggage M at a predetermined low speed while photographing the downward direction with the camera 716, thereby photographing the luggage M below. Here, the predetermined low speed is a flight speed slower than the flight speed in the first flight operation. By flying the drone 71 at a low speed, a clearer photographed image of the luggage M and its surroundings can be obtained. The photographed image is stored in the memory unit 711A1.

In the second flight operation, the drone 71 also moves in the vertical direction, so the control unit 711A can grasp the vertical position of each piece of luggage M that was found to be in an abnormal state in the first flight operation, and can easily distinguish and recognize whether the luggage M is in the first state or the second state. Note that instead of distinguishing between the first state and the second state in this manner, the control unit 711A may perform object recognition of the luggage M from the captured image stored in the memory unit 711A1 to make the distinction. The image of the luggage M captured in the second flight operation is clearer than that captured in the first flight operation, so the distinction can be made more accurately.

After step S202, in step S203, the control unit 711A determines whether there is a next abnormal area. If there is a next abnormal area (Y in step S203), the process proceeds to step S204, where the control unit 711A determines whether the drone 71 has moved to the next abnormal area. If the drone 71 has not moved to the next abnormal area, i.e., if it is located in a normal area (N in step S204), the process proceeds to step S205, where the control unit 711A flies the drone 71 at a predetermined high speed. Here, the predetermined high speed is a speed faster than the flight speed at the low speed in step S202.

Until the drone 71 moves to the next abnormal area, it will fly at high speed in step S205. Then, when the drone 71 moves to the next abnormal area (Y in step S204), it will proceed to step S206, where the control unit 711A will determine whether the luggage M has been detected based on the distance measurement results by the second distance sensor 715. When the luggage M has been detected (Y in step S206), it will proceed to step S202, where it will fly at a low speed and take pictures.

On the other hand, if luggage M is not detected (N in step S206), the process proceeds to step S207, where the control unit 711A determines whether the luggage M photographed immediately before is in the first state. If it is in the first state (Y in step S207), the control unit 711A determines that luggage M in the second state is present above the current position, and raises the drone 71 until luggage M is detected by the distance measurement results of the first distance sensor 714. After raising the drone 71, the process proceeds to step S202, where the control unit 711A photographs the luggage M (photographing below) while flying the drone 71 over the luggage M at a low speed.

That is, after descending to a position near the baggage M in the first state, the drone 71 ascends to detect the baggage M in the second state when it advances to an abnormal area where the baggage M in the first state is not detected. In other words, the drone 71 does not ascend until it reaches the abnormal area where the baggage M in the first state is not detected, and continues flying at the descended height. Note that the abnormal area (the area where either the first state or the second state has occurred) has already been detected by the first flight operation, and this detection result shows that the abnormal area where the baggage M in the first state is not detected is the area where the second state has occurred. Therefore, if the drone 71 ascends in the abnormal area where the baggage M in the first state is not detected, it can detect the baggage M in the second state. This makes it possible to reduce the movement distance of the drone 71, compared to, for example, ascending once to the altitude of the first flight operation and then descending to detect the baggage M in the second state.

Furthermore, if the luggage M photographed immediately before in step S207 is not in the first state (N in step S207), the control unit 711A, while descending the drone 71, determines whether the luggage M is detected by the second distance sensor 715 before the drone 71 descends to a predetermined minimum position corresponding to the luggage M in the first state. If the luggage M is detected (Y in step S208), the process proceeds to step S202, where the control unit 711A photographs the luggage M (photographs below) while flying the drone 71 at a low speed above the luggage M. In this case, the detected luggage M may be in either the first state or the second state.

On the other hand, if the second distance sensor 715 does not detect the luggage M before the drone 71 descends to the specified lowest position (N in step S208), the process proceeds to step S209, where the control unit 711A determines that the luggage M in the second state is located above, and raises the drone 71 until the luggage M is detected by the distance measurement result by the first distance sensor 714. After raising the drone 71, the process proceeds to step S202, where the control unit 711A photographs the luggage M (photographing below) while flying the drone 71 over the luggage M at a low speed.

Also, if there is no next abnormal area in step S203 (N in step S203), the process proceeds to step S211, where the control unit 711A causes the drone 71 to rise to a predetermined altitude for the first flight operation. Then, the operation of the flowchart shown in FIG. 8 is completed. For example, since there is no next abnormal area after the abnormal area in which the luggage M1 is located in the gap R1 in FIG. 2, the drone 71 rises to a predetermined altitude after photographing the luggage M1, and then retreats and exits to the outside through the gap R1.

Note that for gaps in which no abnormal area was detected during the first flight operation (e.g., gaps R3, R5, and R6 in FIG. 2), it has already been confirmed that there is no abnormality, so the drone 71 may not enter those gaps during the second flight operation. As described above, in the second flight operation, the drone 71 descends to a position near the luggage M so that the luggage M in the abnormal area can be photographed more clearly. However, instead of the drone 71 descending to a position near the luggage M, the camera 716 provided on the drone 71 may zoom in to photograph the lower part of the abnormal area with a higher resolution than in the first flight operation. In this way, it is possible to photograph the luggage M in the abnormal area more clearly without descending the drone 71.

As shown in FIG. 7, after the second flight operation in step S20, the process proceeds to step S30, where the third flight operation is performed. The third flight operation is performed on a route that overlaps with the flight route of the first flight operation. More specifically, the operation of the flowchart shown in FIG. 9 is performed here.

The operation shown in FIG. 9 is performed for each of the routes of the gaps between the pair of front sides of the storage shelves 10 in the above-mentioned predetermined flight route, and for each route on the rear side of the storage shelves 10. In the example of FIG. 2, the operation is performed for each route of the gaps R1 to R6, and for each route on the rear side of the storage shelves 10A and 10F.

Here, based on the image of the luggage M captured in step S202 of the second flight operation, the control unit 711A has previously recognized the abnormal area of the first state (the area where the first state occurred) among the abnormal areas detected in the first flight operation. When the operation of the flowchart in FIG. 9 is started, first in step S301, the control unit 711A lowers the drone 71 in the initial abnormal area of the first state while capturing an image of the front side of the storage shelf 10 (capturing the side) with the camera 716. This captures the storage state of the luggage M on the storage shelf 10 in the abnormal area of the first state. The captured image is stored in the memory unit 711A1.

Then, in step S302, the control unit 711A determines whether there is a next abnormal area of the first state, and if there is (N in step S302), proceeds to step S303. In step S303, the control unit 711A causes the drone 71 to rise to a predetermined altitude for the first flight operation in the next abnormal area of the first state, while photographing the front side of the storage shelf 10 (photographing the side) with the camera 716. This captures the storage state of the luggage M on the storage shelf 10 in the abnormal area of the first state. The captured image is stored in the memory unit 711A1.

Then, in step S304, the control unit 711A determines whether there is a next abnormal area of the first state, and if there is (N in step S304), proceeds to step S301.

Also, in step S302, if there is no next abnormal area in the first state (Y in step S302), the process proceeds to step 305, where the control unit 711A raises the drone 71 to a predetermined altitude for the first flight operation. Then, the operation of the flowchart shown in FIG. 9 is completed. Also, in step S304, if there is no next abnormal area in the first state (Y in step S304), the operation shown in FIG. 9 is completed as it is.

In this way, in the third flight operation, when the drone 71 reaches the abnormal area in the first state, it moves up and down while photographing the storage shelf 10 from the side. By photographing the front side of the storage shelf 10 in the abnormal area in the first state, it becomes possible to identify from which storage section S of the storage shelf 10 the luggage M in the first state has fallen.

Note that for gaps where no abnormal area of the first state was detected during the second flight operation (e.g., gaps R1 to R3, R5, and R6 in FIG. 2), it has already been confirmed that no abnormality of the first state has occurred, so drone 71 may be prevented from entering those gaps during the third flight operation.

6. Example of Flight Operation Next, an example of the first to third flight operations described above will be described with reference to Figures 10 to 12. Figures 10 to 12 show an example of the drone 71 flying a route on the front side or rear side of the storage shelf 10. In the example of Figures 10 to 12, luggage M11 and luggage M12 in a first state that have fallen to the floor surface are shown, and luggage M21 and luggage M22 in a second state that have been displaced on the storage shelf 10 are shown.

First, as shown in FIG. 10, the drone 71 performs a first flight operation at a predetermined altitude along flight route FR1 (step S10 in FIG. 7). When the first flight operation is completed, an abnormal area in which either the first state or the second state has occurred is detected. After that, the drone 71 performs a second flight operation along flight route FR2 shown in FIG. 11 (step S20 in FIG. 7).

Specifically, the drone 71 descends to the vicinity of the luggage M11 (step S201 in FIG. 8), and then photographs three pieces of luggage M21 (steps S202-S206 in FIG. 8). The drone 71 then moves to the next abnormal area (Y in step S204 in FIG. 8), but luggage M is not detected there (N in step S206 in FIG. 8), and because the luggage M photographed immediately before was in the first state (Y in step S207 in FIG. 8), the drone 71 rises and detects luggage M21 in the second state (step S210 in FIG. 8). This causes the drone 71 to photograph luggage M21 (step S202 in FIG. 8).

Then, drone 71 moves to the abnormal area where luggage M22 is located, but the luggage is not detected at the current height position (N in step S206 in FIG. 8), and the luggage M21 photographed immediately before is not in the first state (N in step S207 in FIG. 8). Therefore, drone 71 descends to detect luggage M22 (Y in step S208 in FIG. 8), and photographs two pieces of luggage M22 (steps S202 to S206 in FIG. 8).

The drone 71 then moves to the abnormal area where the luggage M12 is located, but the luggage is not detected at the current height position (N in step S206 in FIG. 8), and the luggage M22 photographed immediately before is not in the first state (N in step S207 in FIG. 8). Therefore, the drone 71 descends to detect the luggage M12 (Y in step S208 in FIG. 8), and photographs four pieces of luggage M12 (steps S202 to S206 in FIG. 8). The second flight operation proceeds as described above, and if there is no next abnormal area (N in step S203 in FIG. 8), the drone 71 ascends to the specified altitude (step S211 in FIG. 8).

After the second flight operation, the drone 71 performs a third flight operation on flight route FR3 shown in FIG. 12 (step S30 in FIG. 7). Specifically, the drone 71 photographs the storage shelf 10 with up and down movement at the positions of three pieces of luggage M11 in the first state (steps S301-S304 in FIG. 9), and then photographs the storage shelf 10 with up and down movement at the positions of four pieces of luggage M12 in the first state (steps S301-S304 in FIG. 9). As described above, the second flight operation proceeds, and if there is no abnormal area in the next first state (Y in step S302 in FIG. 9), the drone 71 ascends to a predetermined altitude (step S305 in FIG. 9).

7. Notification of various information and change of notification destination The monitoring system 70 is configured to appropriately notify the manager of various detection results, captured images, and other useful information obtained in each of the above-mentioned flight operations. This makes it easy for the manager to grasp the current situation and take appropriate measures even if an abnormality occurs in the luggage M due to an earthquake or the like.

The monitoring system 70 may identify the type of luggage M that is in an abnormal state, and change the notification destination depending on the results of the identification. Changing the notification destination is a concept that includes cases where notification destinations are added and cases where some notification destinations are excluded. Below, a specific example of changing the notification destination is described.

In this example, in the second flight operation described above, when photographing the luggage M in step S202, the identification information attached to the luggage M (e.g., a description of the contents or a barcode, etc.) is photographed. Then, the photographed image of the identification information linked to the information on the abnormal state of the luggage M (a first state and a second state are distinguished) is transmitted from the control unit 711A of the drone 71 to the control unit 721 of the management device 72.

The memory unit 723 stores in advance a database in which identification information and luggage type information (information indicating the type of luggage M) for each luggage M are stored. The control unit 721 recognizes the identification information from the captured image of the above-mentioned identification information, and searches the above-mentioned database for luggage type information corresponding to the identification information. If the condition that "the searched luggage type information indicates a specific type (e.g., glass products that are relatively easily broken by impact) and the luggage M is in the first state (a state in which it has fallen to the floor)" is satisfied, the control unit 721 not only notifies the manager of the occurrence of an abnormal state, but also notifies a specified cleaning company. Note that if the condition is not satisfied, the control unit 721 notifies the manager of the occurrence of an abnormal state, but does not notify the cleaning company.

In this way, when there is a high possibility that the luggage M has been damaged by falling, it is possible to quickly request a cleaning company to clean up the damaged luggage M. Note that the identification of the type of luggage M (identification of whether or not it is relatively susceptible to damage due to impact) is not limited to the above-mentioned method, and may be performed by object recognition from a photographed image showing the luggage M. Also, whether or not the luggage M is damaged may be directly identified by object recognition from a photographed image showing the luggage M.

8. Others As described above, the monitoring system 70 in this embodiment includes a drone 71 having an imaging means, and is used in a logistics facility 100 in which luggage M is placed on a storage shelf 10. In addition, in the monitoring system 70, the drone 71 flies a predetermined flight route along the storage shelf 10 while performing a first flight operation of photographing the luggage M below, and detects an abnormal state related to the position of the luggage M, including a second state in which the luggage M has fallen from the storage shelf 10, based on information obtained by the photographing. In this way, the monitoring system 70 is able to improve the detection function for an abnormal state of luggage by detecting the first state in which the luggage M has fallen from the storage shelf 10.

(2) Second embodiment Next, a second embodiment of the present invention will be described. The second embodiment is basically the same as the first embodiment, except for detecting a third state as an abnormal state related to the orientation of the luggage M, and points related thereto. In the following description, the emphasis will be placed on points that differ from the first embodiment, and a description of points in common with the first embodiment may be omitted.

When an earthquake or the like occurs and the storage shelf 10 shakes, the orientation of the luggage M placed on the storage shelf 10 may shift. Here, FIG. 13(a) shows a state in which the orientation of the luggage M is normal (initial state immediately after being placed on the storage shelf 10), and FIG. 13(b) shows a state in which the orientation of the luggage M shifts by an angle θ from the state in FIG. 13(a), as viewed from above. As shown in FIG. 13(a), if the orientation of the luggage M is normal, it is possible to insert the slide fork of the transfer device 158 into the gap space Dc of the support body D and lift the luggage M appropriately. However, if the angle θ shown in FIG. 13 becomes large (i.e., if the deviation in the orientation of the luggage M becomes large), it may become difficult to insert the slide fork into the gap space Dc or it may become impossible to lift the luggage M appropriately, which may cause problems in taking the luggage M in and out of the storage shelf 10. Therefore, it is important to detect such abnormalities and deal with them appropriately.

The monitoring system 70 according to the second embodiment is therefore configured to detect not only the first state (a state in which the luggage has fallen from the storage shelf 10 to the floor) and the second state (a state in which a positional deviation that exceeds the tolerance range of the storage shelf 10 has occurred), but also a direction deviation that exceeds a predetermined tolerance range of the storage shelf 10 (hereinafter referred to as the "third state") as an abnormal state related to the position or orientation of the luggage M. Note that the tolerance range (corresponding to the threshold value of the angle θ shown in FIG. 13) can be set arbitrarily, but in this embodiment it is set to 15° as an example.

In the second embodiment, the third state is detected by taking side images (taking images of the front side of the storage shelf 10) by the camera 716 in the third flight operation (step S30) described above. However, in the first embodiment, the image is taken only in the abnormal area of the first state, but in the second embodiment, the image is taken in all areas of the flight route. That is, while flying along the flight route described above, the drone 17 takes images of the front side of the storage shelf 10 while descending at the first vertical row position of the storage shelf 10, takes images of the front side of the storage shelf 10 while ascending at the next row position, takes images of the front side of the storage shelf 10 while descending at the next row position, and so on, moving in a serpentine manner in a side view to take images of the front side of the storage shelf 10 for all rows. This makes it possible to take images of the state of the luggage M as seen from the front side of the storage shelf 10 for all positions of the storage shelf 10 (all positions where luggage M can be placed).

The monitoring system 70 uses the captured image thus obtained (hereinafter, for convenience, referred to as "captured image β") to execute processing for detecting the third state for all positions on the storage shelf 10. More specifically, the monitoring system 70 recognizes the difference (angle θ1) between the orientation of the luggage M in the captured image β and the orientation of the luggage M in the luggage image α already described for all positions on the storage shelf 10. This detection is performed, for example, by performing object recognition of the luggage M for each of the captured image β and the luggage image α, and using the results of these object recognitions. Note that the luggage M in the normal orientation is shown in the luggage image α, and the luggage M in its current state is shown in the captured image β, so the recognized angle θ1 represents the magnitude of the deviation in the orientation of the luggage M.

The monitoring system 70 can therefore recognize any luggage M with an angle θ1 exceeding 15° (the above-mentioned tolerance range) as being in the third state, and detect the occurrence of the third abnormal state and its location. The detection results and information on the third state, such as captured images, are appropriately reported to the manager along with information on the first and second states. This allows the manager to easily take appropriate measures, such as correcting the orientation of luggage M that has fallen into the third state.

(3) Summary In addition to the above-described embodiment, the configuration of the present invention can be modified in various ways without departing from the spirit of the invention. In other words, the above-described embodiment should be considered to be illustrative in all respects and not restrictive, and the technical scope of the present invention is indicated by the claims, not the description of the above-described embodiment, and should be understood to include all modifications that fall within the meaning and scope of the claims.

100 Logistics equipment 10, 10A to 10F Storage shelf 15, 15A to 15C Article transport device 16 Travel rail 20 In-warehouse/out-of-warehouse conveyor 30 Travel rail 35 Transport cart 70 Surveillance system 71 Drone 71A Frame 71B Center section 71C to 71F Rotor 711 Flight controller 711A Control section 711A1 Memory section 711B Sensor group 711C GPS receiver 712 ESC
713 Rotor motor 714 First distance sensor 715 Second distance sensor 716 Camera 717 Communication unit 718 Battery 72 Management device 721 Control unit 722 Communication unit 723 Storage unit 724 Network interface 73 Vibration sensor M Baggage D Support S Storage unit

Claims (6)

A flight system for use in a facility where baggage is placed on a storage shelf, the flight system including an aircraft having a photographing means,
The flying object performs a flight operation to photograph the baggage while flying along a predetermined flight route,
Object recognition of the luggage is performed for each of the captured image obtained by photographing during the flight operation and a luggage image which is an image of the luggage acquired in advance,
The flying object photographs a downward direction while flying along the flight route,
Based on the information obtained by the photographing, an abnormal state regarding the position or orientation of the luggage , including a state in which the luggage has fallen from the storage shelf , is detected;
A flight system characterized by creating a distribution map of the locations where the abnormal conditions occurred based on the results of the execution of the abnormal condition detection process. The flight system of claim 1 , wherein the baggage image is acquired as an image showing the baggage in a normal loaded state . 3. The flight system according to claim 1 , wherein the baggage image is an image taken when the baggage is placed on the storage shelf . A flight system as described in any one of claims 1 to 3, characterized in that , based on the result of the object recognition, a difference between the placement state of the luggage in the captured image and the placement state of the luggage in the luggage image is recognized. 5. The flight system of claim 4, wherein the difference is a difference between an orientation of the luggage in the captured image and an orientation of the luggage in the luggage image . 6. The flight system according to claim 4, wherein when the difference exceeds a predetermined value, the detection result is reported to a predetermined reporting destination .