JP7790197B2 - Maintenance support system, maintenance support device, and maintenance support method - Google Patents

Description

This disclosure relates to a maintenance support system, a maintenance support device, and a maintenance support method.

Patent Document 1 discloses a method of using augmented reality (AR) to assist in the insertion and removal of connector plug terminals into connector adapter terminals in a connector connection board for an optical fiber wiring module. This method involves placing position detection markers on a connector connection board in which numerous connector adapter terminals are arranged in a matrix, capturing an image of the connector connection board with a camera, and then recognizing the image of the position detection marker in the captured image using a wiring work assistance device that receives the captured image. The wiring work assistance device then sets the coordinates of the center position of the recognized position detection marker in the captured image as the origin reference. The wiring work assistance device then generates image patterns for the coordinate grid and target grid, and offsets the generated image patterns from the origin reference based on a drawing position offset amount contained in pre-stored definition information, superimposing them on the captured image.

JP 2012-247700 A

However, the method disclosed in Patent Document 1 requires the provision of position detection markers for image recognition on the connector connection board.

A maintenance support system according to one aspect of the present disclosure is a maintenance support system that supports the maintenance of a communication device having a housing with multiple communication ports on one surface thereof, and includes: a maintenance support terminal including a camera and a first display; a recognition unit that uses a machine learning trained model to recognize the surface, including functional units of the communication device, in an image of the surface captured by the camera; a position measurement unit that measures the position in real space of the surface recognized by the trained model; a virtualization unit that determines a representative position of the surface based on the position of the surface measured by the position measurement unit and places a first virtual surface, which is a virtual three-dimensional model of the surface, at a position corresponding to the representative position in a virtual space corresponding to the real space; and a first display control unit that superimposes the first virtual surface placed in the virtual space by the virtualization unit and the image of the surface captured by the camera on the first display.

A maintenance support device according to one aspect of the present disclosure is a maintenance support device that supports the maintenance of a communications device having a housing with multiple communications ports on one side thereof, and includes: a recognition unit that uses a machine learning trained model to recognize the side, including functional units of the communications device, in an image of the side captured by a camera; a virtualization unit that determines a representative position of the side based on the position in real space of the side recognized by the trained model, and places a virtual surface, which is a virtual 3D model of the side, at a position corresponding to the representative position in a virtual space corresponding to the real space; and a display control unit that generates an augmented reality image in which the virtual surface placed in the virtual space by the virtualization unit is superimposed on the image of the side captured by the camera.

A maintenance support method according to one aspect of the present disclosure is a maintenance support method for supporting the maintenance of a communication device having a housing with multiple communication ports on one side thereof, and includes the steps of: recognizing, in an image of the side captured by a camera, the side including functional units of the communication device using a machine learning trained model; determining a representative position of the side based on the position in real space of the side recognized by the trained model; and placing a virtual surface, which is a virtual 3D model of the side, at a position corresponding to the representative position in a virtual space corresponding to the real space; and generating an augmented reality image by superimposing the virtual surface placed in the virtual space on the image of the side captured by the camera.

The present disclosure can be realized not only as a maintenance support system and maintenance support device having the above-described characteristic configuration, and as a maintenance support method having characteristic processing steps, but also as a computer program that causes a computer to execute the above-described method, as a maintenance management terminal included in a maintenance support system, or as part of a maintenance support device as a semiconductor integrated circuit.

According to the present disclosure, there is no need to provide a marker for image recognition on the communication device.

1 is a diagram illustrating an example of a configuration of a maintenance support system according to an embodiment. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a maintenance support terminal according to the embodiment. FIG. 2 is a diagram illustrating an example of a panel of a communication device according to an embodiment. FIG. 3B shows a virtual panel corresponding to the panel of FIG. 3A. 3B is a diagram showing an AR image in which the panel shown in FIG. 3A and the virtual panel shown in FIG. 3B are superimposed. FIG. 10 is a diagram showing an example of an arrangement pattern of upper-level ventilation openings. 10A and 10B are diagrams showing other examples of the arrangement pattern of the upper vent holes. FIG. 10 is a diagram showing a first example of an arrangement pattern of lower vent holes. FIG. 10 is a diagram showing a second example of an arrangement pattern of lower vent holes. FIG. 10 is a diagram showing a third example of an arrangement pattern of lower vent holes. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a server according to the embodiment. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a maintenance management terminal according to the embodiment. FIG. 2 is a functional block diagram illustrating an example of functions of the maintenance support system according to the embodiment. 1A and 1B are diagrams for explaining partial elements of a panel of a communication device according to an embodiment. 10A and 10B are diagrams for explaining measurement of the position of a partial element by a position measurement unit. 10A and 10B are diagrams for explaining the arrangement of virtual panels in a virtual space by a virtualization unit according to the embodiment. 10A and 10B are diagrams for explaining the relationship between the position of a panel in real space and the position of a virtual panel in virtual space. FIG. 1 is a diagram illustrating a first example of the positional relationship of partial elements in a communication device. FIG. 10 is a diagram illustrating a second example of the positional relationship of partial elements in a communication device. FIG. 10 is a diagram illustrating a third example of the positional relationship of partial elements in a communication device. FIG. 10 is a diagram illustrating an example of an AR image displayed on a display. FIG. 10 is a diagram illustrating an example of an AR image when an abnormality occurs in a communication port. FIG. 10 is a diagram illustrating an example of an AR image when some communication ports are selected. FIG. 10 is a diagram showing an example of an AR image in which a communication port is specified and work information is displayed. FIG. 10 is a diagram illustrating an example of an AR image in which port information is displayed. FIG. 10 is a diagram showing an example of an AR image displaying information on a first step in dealing with an abnormality. FIG. 10 is a diagram showing an example of an AR image displaying information on a second stage of dealing with an abnormality. FIG. 10 is a diagram showing an example of an AR image displaying information on a third stage of dealing with an abnormality. FIG. 10 is a diagram illustrating an example of an AR image in which information on the completion of dealing with an abnormality is displayed. 10 is a flowchart illustrating an example of an AR display process performed by the maintenance support terminal according to the embodiment. 10 is a flowchart illustrating an example of a port abnormality process performed by the maintenance support terminal according to the embodiment. 10 is a flowchart illustrating an example of a selection condition display process by the maintenance support terminal according to the embodiment. 10 is a flowchart illustrating an example of a port information display process by the maintenance support terminal according to the embodiment. 10 is a flowchart illustrating an example of a tutorial process by the maintenance support terminal according to the embodiment. FIG. 10 is a sequence diagram illustrating an example of a work instruction process performed by the maintenance support system according to the embodiment. 10A and 10B are diagrams illustrating modified examples of the panel of the communication device according to the embodiment.

Overview of the embodiments of the present disclosure
The following provides an outline of embodiments of the present disclosure.

(1) The maintenance support system according to this embodiment is a maintenance support system that supports the maintenance of a communication device having a housing with multiple communication ports on one side thereof, and includes: a maintenance support terminal including a camera and a first display; a recognition unit that recognizes the side including the functional units of the communication device in an image of the side captured by the camera using a trained model of machine learning; a position measurement unit that measures the position in real space of the side recognized by the trained model; a virtualization unit that determines a representative position of the side based on the position of the side measured by the position measurement unit and places a first virtual surface, which is a virtual three-dimensional model of the side, at a position corresponding to the representative position in a virtual space corresponding to the real space; and a first display control unit that superimposes the first virtual surface placed in the virtual space by the virtualization unit and the image of the side captured by the camera on the first display. Because the trained model recognizes the side including the functional units of the communication device from the image of the side of the communication device, it is possible to display the side in AR without providing any elements (markers) for image recognition on the side.

(2) The recognition unit may recognize partial elements of the one surface using the trained model, the position measurement unit may measure the real-space position of the partial elements recognized by the trained model, and the virtualization unit may determine the representative position based on the position of the partial elements measured by the position measurement unit. As a result, the trained model recognizes partial elements including functional parts of the communication device from an image of one surface of the communication device, making it possible to display the one surface in AR without providing elements (markers) for image recognition on the one surface. Furthermore, even if the entire surface is not captured, as long as partial elements are captured, the recognition unit can recognize the partial elements and display the one surface in AR.

(3) The trained model may be generated by machine learning using the image of the surface from which the partial elements have been identified as input data. The trained model generated in this manner can recognize the partial elements of the surface of the communication device.

(4) The recognition unit may recognize multiple partial elements of the surface in the image of the surface captured by the camera using a machine learning trained model, the position measurement unit may measure the position of each of the multiple partial elements in the real space, and the virtualization unit may position the first virtual surface in the virtual space based on the position of each of the multiple partial elements in the real space. In this way, by recognizing the multiple partial elements, the first virtual surface can be positioned in the virtual space in accordance with the position of the surface in the real space.

(5) The virtualization unit may determine whether the multiple partial elements recognized by the trained model are the partial elements of the same communication device based on the positions of each of the multiple partial elements in the real space, and may not place the first virtual surface in the virtual space if it determines that the multiple partial elements are not the partial elements of the same communication device. As a result, if the positional relationship between the multiple recognized partial elements is an impossible positional relationship between partial elements of the same communication device, the first virtual surface is not placed in the virtual space. Therefore, it is possible to prevent the placement of an inappropriate first virtual surface.

(6) The first virtual surface may include a plurality of component models that are three-dimensional models of the plurality of communication ports, and the first display control unit may be capable of switching the color of each of the plurality of component models. This allows the state of a communication port, etc. to be expressed by the color of the corresponding component model.

(7) The component model may be set to a normal color when the corresponding communication port is normal and an abnormal color when the corresponding communication port is abnormal, and the maintenance support system may further include a notification receiving unit that receives a notification of an abnormal communication port in the communication device, and the first display control unit may switch the color of the component model corresponding to the communication port from the normal color to the abnormal color when the notification receiving unit receives a notification of an abnormal communication port. This allows a user to recognize the status of the corresponding communication port from the color of the component model on the first virtual plane.

(8) The notification receiving unit may be capable of receiving a notification of the status of the communication port in the communication device, and the first display control unit, when the notification receiving unit receives a notification of an abnormal status of the communication port, may cause the first display to display first handling information indicating a first-stage handling method for the abnormality, and when the status of the communication port changes as a result of the first-stage handling method being executed and the notification receiving unit receives a notification of the changed status of the communication port, may switch the first handling information displayed on the first display to second handling information indicating a second-stage handling method for the abnormality. This allows a user (maintenance worker) to proceed with the work of handling the abnormality of the communication port according to the AR display on the first display.

(9) The part models may be set with a non-selection color, which is the color when the corresponding communication port is not selected, and a selection color, which is the color when the corresponding communication port is selected, and when the first display control unit receives selection conditions for selecting the communication port, the first display control unit may switch the color of the part models among the plurality of part models that correspond to the communication ports that match the selection conditions from the non-selection color to the selection color. This allows the user to recognize communication ports that match the selection conditions by the color of the part models on the first virtual surface.

(10) The maintenance support terminal may further include a first input device, and when the first input device receives input specifying the component model on the first virtual plane, the first display control unit may cause the first display to display attribute information indicating the attributes of a transceiver attached to a communication port corresponding to the component model, status information indicating the status of the communication port, and communication volume information indicating the communication volume of the communication port. This allows a user to specify a component model on the first virtual plane, thereby causing the first display to display the attribute information, status information, and communication volume information of the communication port corresponding to the component model.

(11) The maintenance support system further includes a maintenance management terminal located at a location different from the location of the communication device, the maintenance management terminal accepting input of designation of the communication port of the communication device and work information indicating work to be performed on the designated communication port, and transmitting the designation information indicating the designated communication port and the input work information, the component model being set to a non-designated color that is a color when the corresponding communication port is not designated and a designated color that is a color when the corresponding communication port is designated, the maintenance support system further including a receiving unit that receives the designation information and the work information transmitted from the maintenance management terminal, the first display control unit switching the color of the component model corresponding to the communication port designated in the designation information received by the receiving unit from the non-designated color to the designated color, and displaying the work information received by the receiving unit on the first display. This allows an administrator using the maintenance management terminal to remotely designate a communication port and instruct work. The user can proceed with the work instructed by the administrator according to the AR display on the first display.

(12) The maintenance management terminal may include a second input device, a second display, and a second display control unit, and the second display control unit may cause a second virtual surface, which is a virtual three-dimensional model of the surface, to be displayed on the second display. The maintenance management terminal may receive input from the second input device specifying a communication port for the second virtual surface, thereby receiving the designation of the communication port. This allows the administrator to specify a communication port using the second virtual surface displayed on the maintenance management terminal.

(13) A maintenance support device according to this embodiment is a maintenance support device that supports the maintenance of a communication device having a housing with multiple communication ports on one side thereof, and includes: a recognition unit that uses a machine learning trained model to recognize the side including functional units of the communication device in an image of the side captured by a camera; a virtualization unit that determines a representative position of the side based on the position in real space of the side recognized by the trained model and places a virtual surface, which is a virtual 3D model of the side, at a position corresponding to the representative position in a virtual space corresponding to the real space; and a display control unit that generates an augmented reality image in which the virtual surface placed in the virtual space by the virtualization unit is superimposed on the image of the side captured by the camera. Because the trained model recognizes the side including the functional units of the communication device from the image of the side of the communication device, it is possible to display the side in AR without providing elements (markers) for image recognition on the side.

(14) A maintenance assistance method according to this embodiment is a maintenance assistance method for assisting in the maintenance of a communication device having a housing with multiple communication ports on one side thereof, and includes the steps of: recognizing, in an image of the side captured by a camera, the side including functional units of the communication device using a machine learning trained model; determining a representative position of the side based on the position in real space of the side recognized by the trained model; and placing a virtual surface, which is a virtual three-dimensional model of the side, at a position corresponding to the representative position in a virtual space corresponding to the real space; and generating an augmented reality image in which the virtual surface placed in the virtual space and the image of the side captured by the camera are superimposed. Because the trained model recognizes the side including the functional units of the communication device from the image of the side of the communication device, it is possible to display the side in AR without providing elements (markers) for image recognition on the side.

<Details of the embodiment of the present disclosure>
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, detailed descriptions of embodiments of the present invention will be given with reference to the accompanying drawings. At least some of the embodiments described below may be combined in any desired manner.

[1. Maintenance support system]
1 is a diagram illustrating an example of the configuration of a maintenance support system according to an embodiment. The maintenance support system 10 supports a user (maintenance worker) in maintaining communication devices 200A, 200B, 200C, 200D, 200E, 200F, 200G, and 200H. The maintenance support system 10 includes a maintenance support terminal 100, a server 300, and a maintenance management terminal 400.

Communication devices 200A, 200B, 200C, 200D, 200E, 200F, 200G, and 200H are installed in communication center 20, which is a facility of a telecommunications carrier. Communication devices 200A, 200B, 200C, and 200D are mounted in rack 250A, and communication devices 200E, 200F, 200G, and 200H are mounted in rack 250B. Note that in the following description, communication devices 200A, 200B, 200C, 200D, 200E, 200F, 200G, and 200H may be collectively referred to as "communication devices 200," and racks 250A and 250B may be collectively referred to as "rack 250."

The communication device 200 includes multiple communication ports. The communication ports are provided on the front of the housing of the communication device 200. The front of the housing of the communication device 200 is composed of a single panel. The communication device 200 is, for example, an optical communication device, and an optical transceiver can be inserted (attached) to each communication port.

Rack 250 is equipped with various types of communication devices 200. For example, communication devices 200A and 200B are a first type of communication device, and communication devices 200C and 200D are a second type of communication device. The configuration of the front panel of the communication device differs depending on the model.

The maintenance support terminal 100 is an example of a maintenance support device. The maintenance support terminal 100 is used by a user at the communication center 20 to perform maintenance work on the communication device 200. The maintenance support terminal 100 is a portable terminal with communication capabilities, such as a tablet, smart glasses, or smartphone.

The server 300 and the maintenance management terminal 400 are installed, for example, in a management center 30, which is a facility of a management company that manages the communication device 200. The management center 30 is installed, for example, in a remote location from the communication center 20.

The server 300 and maintenance management terminal 400 are communicatively connected to the maintenance support terminal 100 via the network 500. The server 300 provides the maintenance support terminal 100 with information and functions used in maintenance work on the communication device 200. The maintenance management terminal 400 is used by an administrator. The administrator uses the maintenance management terminal 400 to support users in performing maintenance work on the communication device 200.

[2. Configuration of the maintenance support terminal]
2 is a block diagram showing an example of the hardware configuration of a maintenance support terminal according to this embodiment. The maintenance support terminal 100 includes a processor 101, a non-volatile memory 102, a volatile memory 103, an input device 104, a display 105, a graphics processing unit 106, a communication interface (I/F) 107, a camera 108, a position measurement unit 109, a gyro sensor 112, and an acceleration sensor 113.

The volatile memory 103 is, for example, a semiconductor memory such as an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory). The non-volatile memory 102 is, for example, a flash memory, a hard disk, or a ROM (Read Only Memory). The non-volatile memory 102 stores a maintenance support application (APP) 120 and data used to execute the maintenance support APP 120. The non-volatile memory 102 stores a trained model 600 generated by machine learning. The trained model 600 is used by the maintenance support APP 120. The non-volatile memory 102 stores data for virtual panels 700A, 700B, 700C, etc., which are three-dimensional virtual models of the panel of the communication device 200. The virtual panels 700A, 700B, 700C, etc. are examples of first virtual surfaces. The virtual panels 700A, 700B, 700C, etc. are used by the maintenance support APP 120. Non-volatile memory 102 stores virtual panels 700A, 700B, and 700C for each model of communication device 200. In the following description, virtual panels 700A, 700B, and 700C may be collectively referred to as "virtual panels 700."

The panel of the communication device 200 and the virtual panel 700 will now be described. Figure 3A is a diagram showing an example of a panel of a communication device according to an embodiment. The panel 210 includes an external port 211, a communication port 212, and air vents 221 and 222.

The external port 211 is located on the left edge of the panel 210. The external port 211 is a communication port for connecting to the network 500. For example, the external port 211 is used for communication with an external maintenance device, such as the maintenance support terminal 100, connected to the network 500.

The communication ports 212 allow optical transceivers to be attached and detached. Multiple communication ports 212 are arranged on the panel 210, separated into two levels, upper and lower. The communication ports 212 include two types of communication ports: one compatible with 1 Gbps optical transceivers and one compatible with 10 Gbps optical transceivers.

Multiple ventilation holes 221, 222 are provided on the upper part of the panel 210, above the communication port 212. The ventilation holes 221, 222 are intake holes that introduce air into the housing of the communication device 200, or exhaust holes that expel air from the housing of the communication device 200.

Figure 3B is a diagram showing a virtual panel corresponding to the panel in Figure 3A. Virtual panel 700 is composed of multiple part models 701, 702, and 703. Part model 701 is a three-dimensional virtual model of the portion of panel 210 that includes external port 211 at the left edge. Part model 702 is a three-dimensional virtual model of the communication port 212 portion of panel 210. Part model 703 is a three-dimensional model that defines the outer edge of panel 210. Part models 701 and 702 are placed in part model 703 at positions where their corresponding portions are placed in panel 210. In other words, part model 701 is placed at the left edge of part model 703. Part model 702 is placed in part model 703 at a position corresponding to the position of communication port 212 on panel 210.

The virtual panel 700 described above is placed in a virtual three-dimensional space (virtual space) set in the maintenance support terminal 100 to correspond to the real space. As a result, as will be described below, the image of the panel 210 captured by the camera 108 and the virtual panel 700 are displayed superimposed on each other. Figure 3C is a diagram showing an AR image in which the panel shown in Figure 3A and the virtual panel shown in Figure 3B are superimposed. In the AR image, the position of the outline of the panel 210 matches the position of the part model 703. The position of the portion of the panel 210 including the external port 211 at the left edge matches the position of the part model 701. Furthermore, the positions of each of the multiple communication ports 212 match the positions of each of the multiple part models 702.

The virtual panel 700 is transparent. Therefore, in the AR image, the panel 210 appears through the virtual panel 700.

Returning to Figure 3A, the ventilation openings 221, 222 are arranged in two rows, upper and lower. The upper ventilation opening 221 is used to identify the model of the communication device 200. In other words, the ventilation opening 221 also serves as an identification code for the model of the communication device 200. The lower ventilation opening 222 is used to identify the position within the panel 210. In other words, the ventilation opening 222 also serves as an identification code for the position within the panel 210.

For example, multiple ventilation openings 221 are arranged in a row horizontally, and multiple ventilation openings 222 are arranged in a row horizontally. Hereinafter, the vertical length of one ventilation opening 221, 222 will be referred to as "height," and its horizontal length will be referred to as "width." The ventilation openings 221, 222 are provided on the panel 210 so as to satisfy certain requirements. These requirements include that the width of the ventilation openings 221, 222 must be within a range defined by predetermined upper and lower limits, that a certain number or more of ventilation openings 221 and 222 must be provided per section of a certain length horizontally, and that the shape of the ventilation openings 221, 222 must be selected from a plurality of predetermined shapes.

In the following description, the shape of the ventilation openings 221, 222 is assumed to be rectangular. However, the shape of the ventilation openings 221, 222 may be other than rectangular, for example, circular, triangular, or star-shaped.

Figure 4A is a diagram showing an example of the arrangement pattern of the upper vents, and Figure 4B is a diagram showing another example of the arrangement pattern of the upper vents. The arrangement pattern of the vents 221 is determined for each model of communication device 200. For example, the arrangement pattern of the vents 221A shown in Figure 4A is a pattern specific to the models of communication devices 200A and 200B, and the arrangement pattern of the vents 221B shown in Figure 4B is a pattern specific to the models of communication devices 200C and 200D. Therefore, the maintenance support terminal 100 can identify the model of the communication device 200 by image recognition of the arrangement pattern of the vents 221.

The ventilation opening 221 is selected from two shapes: a square opening and a rectangular opening. A square opening is a square opening with a width equal to its height. A rectangular opening has a width greater than its height. The height of a square opening and a rectangular opening is the same. The width of a rectangular opening is equal to the width of two square openings. When two square openings are placed side by side, a fixed gap is provided between adjacent square openings. The width of a rectangular opening is equal to the sum of the widths of the two square openings and the gap between them. Furthermore, in addition to square openings and rectangular openings, closed square openings are used in the arrangement pattern of the ventilation openings 221. A closed square opening is a closed area (with no holes) that has the same shape and size as a square opening. The arrangement pattern of the air vents 221 is defined by the combination and order of square openings, rectangular openings, and closed square openings in a section (hereinafter referred to as a "unit section") that is the width of three square openings in the left-right direction, and the same pattern is repeated for each unit section. The number of closed square openings in one unit section is either 1 or 0. The arrangement pattern of the air vents 221 in one unit section is assigned to the model of the communication device 200. In other words, the arrangement pattern of the air vents 221 in one unit section is the model code of the communication device 200. In this case, eight different model codes can be defined.

A blank section of a certain length is provided between adjacent unit sections. The blank section has a width greater than the distance between adjacent openings (square openings or rectangular openings) in the unit section. By providing blank sections, the unit sections can be recognized by image recognition.

The arrangement pattern of the vents 221A shown in FIG. 4A includes two square openings and one closed square opening in the unit section. The unit section is defined by three regions, "first region," "second region," and "third region," arranged from left to right. The arrangement pattern of the vents 221A is a pattern in which square openings are arranged in each of the first and second regions, and a closed square opening is arranged in the third region. The arrangement pattern of the vents 221B shown in FIG. 4B is a pattern in which square openings are arranged in the first region, and rectangular openings are arranged in the second and third regions. Because the same arrangement pattern of the vents 221 is repeated for each unit section, the same pattern of the vents 221 is arranged in multiple positions on the panel 210. Therefore, even if the camera 108 captures an image of only a portion of the panel 210, the model of the communication device 200 can be identified based on the pattern of the vents 221 included in the captured image.

Figure 5A shows a first example of a layout pattern for the lower vents, Figure 5B shows a second example of a layout pattern for the lower vents, and Figure 5C shows a third example of a layout pattern for the lower vents. The lower vents 222 are used to identify their position on the panel 210 (left, center, right). Each model of communication device 200 has a specific layout pattern for the vents 222 for each of the left, center, and right sections of the panel 210. For example, the layout pattern for vents 222A shown in Figure 5A is specific to the left section of the panel 210, the layout pattern for vents 222B shown in Figure 5B is specific to the center section of the panel 210, and the layout pattern for vents 222C shown in Figure 5C is specific to the right section of the panel 210. Therefore, by performing image recognition of the arrangement pattern of the air vents 222, the maintenance support terminal 100 can identify the position (left, center, right) of the recognized part on the panel 210.

The arrangement pattern of the ventilation openings 222A shown in Figure 5A is a pattern in which square openings are arranged in each of the first and second regions, and a closed square opening is arranged in the third region. The arrangement pattern of the ventilation openings 222B shown in Figure 5B is a pattern in which square openings are arranged in the first region, and rectangular openings are arranged in the second and third regions. The arrangement pattern of the ventilation openings 222C shown in Figure 5C is a pattern in which square openings are arranged in the first region, closed square openings in the second region, and square openings in the third region. Note that in the above example, the unit section is an area equivalent to three square openings, but it may also be an area equivalent to four or more square openings. However, the unit section should be of a size that allows for the definition of model codes at least equal to the number of models of the communication device 200.

Returning to FIG. 2, the maintenance support terminal 100 is configured with a computer, and some or all of the functions of the maintenance support terminal 100 are realized by the processor 101 executing the maintenance support APP 120, which is a computer program stored in the computer's storage device. The maintenance support APP 120 can be stored in a recording medium such as flash memory, ROM, or CD-ROM. By the processor 101 executing the maintenance support APP 120, the maintenance support terminal 100 can display the communication device 200 in AR.

The processor 101 is, for example, a CPU (Central Processing Unit). However, the processor 101 is not limited to a CPU. The processor 101 may also be a GPU (Graphics Processing Unit). The processor 101 may also be, for example, an ASIC (Application Specific Integrated Circuit) or a programmable logic device such as a gate array or FPGA (Field Programmable Gate Array). In this case, the ASIC or programmable logic device is configured to be able to execute processing similar to that of the maintenance support APP 120.

Display 105 is an example of a first display. Graphics processing unit 106 is connected to display 105 and controls the display on display 105. Graphics processing unit 106 includes, for example, a GPU and VRAM (Video RAM), stores data to be displayed on display 105 in VRAM, periodically reads one frame of video data from VRAM, and generates a video signal. The generated video signal is output to display 105, and the video is displayed on display 105. The functions of graphics processing unit 106 may be included in processor 101. A portion of the volatile memory 103 may be used as VRAM.

The display 105 includes, for example, a liquid crystal monitor or an OEL (organic electroluminescence) monitor. The display 105 can display text or graphic information. The input device 104 is an example of a first input device. The input device 104 includes, for example, a capacitive or pressure-sensitive touchpad overlaid on the display 105. The input device 104 may also be a keyboard and a pointing device such as a mouse. The input device 104 is used to input information to the maintenance support terminal 100.

The communication I/F 107 is, for example, a wireless communication interface. Specifically, the communication I/F 107 is a wireless LAN interface, a Bluetooth interface, or a wireless communication interface compliant with 5G (fifth generation mobile communication system) or 4G (fourth generation mobile communication system). The communication I/F 107 can be connected wirelessly to the network 500. The maintenance support terminal 100 can communicate with the server 300 and the maintenance management terminal 400 via the communication I/F 107.

Camera 108 includes an image sensor, such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), and a lens. Camera 108 generates captured images and outputs the images. For example, camera 108 can continuously generate still images at a predetermined frame rate and output the still images at the same frame rate to output a moving image. Hereinafter, the image output from camera 108 will also be referred to as a "camera image."

The position measurement unit 109 measures the position of an object outside the maintenance support terminal 100, i.e., the coordinates in three-dimensional space (real space). For example, the position measurement unit 109 measures the position of the object using a ray casting method. The position measurement unit 109 includes a laser irradiation unit 110 and a light receiving unit 111. The position measurement unit 109 measures the position of the object by irradiating a laser from the laser irradiation unit 110 toward the object and receiving the laser light (reflected light) reflected from the object with the light receiving unit 111.

The gyro sensor 112 detects the angular velocity of the maintenance support terminal 100. The acceleration sensor 113 detects the acceleration of the maintenance support terminal 100. The maintenance support terminal 100 can calculate the position and orientation (azimuth angle) of the maintenance support terminal 100 from the detection values of the gyro sensor 112 and the acceleration sensor 113.

[3. Server Configuration]
6 is a block diagram showing an example of the hardware configuration of a server according to this embodiment. The server 300 includes a processor 301, a non-volatile memory 302, a volatile memory 303, and a communication I/F 304.

Volatile memory 303 is, for example, a semiconductor memory such as SRAM or DRAM. Non-volatile memory 302 is, for example, a flash memory, a hard disk, or a ROM. Non-volatile memory 302 stores a server program 305 and data used to execute the server program 305. Non-volatile memory 302 also includes a communication device database (DB) 306. The communication device DB 306 stores information for supporting maintenance of the communication device 200. Server 300 is configured with a computer, and some or all of the functions of server 300 are performed by processor 301 executing server program 305, a computer program stored in the computer's storage device. Server program 305 can be stored on a recording medium such as flash memory, ROM, or CD-ROM. By processor 301 executing server program 305, server 300 can provide information and functions for supporting maintenance of the communication device 200 to the maintenance support terminal 100.

Processor 301 is, for example, a CPU. However, processor 301 is not limited to a CPU. Processor 301 may also be a GPU. Processor 301 may also be, for example, an ASIC, or a programmable logic device such as a gate array or FPGA. In this case, the ASIC or programmable logic device is configured to be able to execute processing similar to that of server program 305.

The communication I/F 304 is a communication interface such as an Ethernet interface, a wireless LAN interface, or a Bluetooth interface. The communication I/F 304 is connected to the network 500. The server 300 can communicate with the maintenance support terminal 100 and the maintenance management terminal 400 via the communication I/F 304.

The communication device 200 is connected to the network 500. The server 300 can communicate with the communication device 200 via the communication I/F 304.

[4. Configuration of maintenance management terminal]
7 is a block diagram showing an example of the hardware configuration of a maintenance management terminal according to this embodiment. The maintenance management terminal 400 includes a processor 401, a non-volatile memory 402, a volatile memory 403, an input device 404, a display 405, a graphics processing unit 406, and a communication I/F 407.

The volatile memory 403 is, for example, a semiconductor memory such as an SRAM or a DRAM. The non-volatile memory 402 is, for example, a flash memory, a hard disk, a ROM, etc. The non-volatile memory 402 stores the maintenance management APP 410 and data used to execute the maintenance management APP 410. The non-volatile memory 402 stores data for virtual panels 700A, 700B, 700C, etc. The data for virtual panels 710A, 710B, 710C, etc. is used by the maintenance management APP 410. The virtual panels 710A, 710B, 710C, etc. are opaque three-dimensional virtual models of the panels of the communication device 200. The virtual panels 710A, 710B, 710C, etc. contain the same data as the virtual panels 700A, 700B, 700C, etc., except that they are opaque. In the following description, virtual panels 710A, 710B, and 710C may be collectively referred to as "virtual panel 710."

The maintenance management terminal 400 is configured with a computer, and some or all of the functions of the maintenance management terminal 400 are realized by the processor 401 executing the maintenance management APP 410, which is a computer program stored in the computer's storage device. The maintenance management APP 410 can be stored in a recording medium such as flash memory, ROM, or CD-ROM. When the processor 401 executes the maintenance management APP 410, the maintenance management terminal 400 can display a three-dimensional model of the panel 210 of the communication device 200.

The processor 401 is, for example, a CPU. However, the processor 401 is not limited to a CPU. The processor 401 may also be a GPU. The processor 401 may also be, for example, an ASIC, or a programmable logic device such as a gate array or FPGA. In this case, the ASIC or programmable logic device is configured to be able to execute processing similar to that of the maintenance management APP 410.

The graphics processing unit 406 is connected to the display 405 and controls the display on the display 405. The functions of the graphics processing unit 406 may be included in the processor 401. A portion of the volatile memory 403 may be used as VRAM.

The input device 404 is an example of a second input device, and the display 405 is an example of a second display. The display 405 includes, for example, a liquid crystal panel or an OEL panel. The display 405 can display text or graphic information. The input device 404 includes, for example, a keyboard and a pointing device such as a mouse. The input device 404 may also be a capacitive or pressure-sensitive touchpad overlaid on the display 405. The input device 404 is used to input information to the maintenance management terminal 400.

The communication I/F 404 is a communication interface such as an Ethernet interface, a wireless LAN interface, or a Bluetooth interface. The communication I/F 404 is connected to the network 500. The maintenance management terminal 400 can communicate with the maintenance support terminal 100 and the server 300 via the communication I/F 404.

[5. Functions of the maintenance support system]
8 is a functional block diagram showing an example of functions of the maintenance support system according to the embodiment. When the processor 101 of the maintenance support terminal 100 executes the maintenance support APP 120, the maintenance support terminal 100 functions as a recognition unit 121, a virtualization unit 122, a display control unit 123 (first display control unit), a notification acceptance unit 124, and a receiving unit 125. The recognition unit 121 and the virtualization unit 122 are realized by the processor 101. The display control unit 123 is realized by the processor 101 and the graphics processing unit 106. The notification acceptance unit 124 and the receiving unit 125 are realized by the processor 101 and the communication I/F 107. When the processor 401 of the maintenance management terminal 400 executes the maintenance management APP 410, the maintenance management terminal 400 functions as a display control unit 411 (second display control unit) and a communication unit 412. The display control unit 411 is realized by the processor 401 and the graphics processing unit 406. The communication unit 412 is realized by the processor 401 and the communication I/F 407 .

When a user performs maintenance on the communication device 200, the user captures an image of the panel 210 of the communication device 200 using the camera 108 of the maintenance support terminal 100. The camera image output from the camera 108 is provided to the recognition unit 121. The recognition unit 121 recognizes partial elements of the panel 210 in the image of the panel 210 captured by the camera 108 using the trained model 600. A partial element is an element that includes a functional unit of the communication device 200. One example of a functional unit is the communication port 212, another example is the external port 211, and yet another example is the air vents 221 and 222.

Figure 9 is a diagram illustrating the partial elements of the panel of a communication device according to this embodiment. In the example of Figure 9, the leftmost region of panel 210 is partial element 230A, the leftmost region of the center of panel 210 is partial element 230B, the rightmost region of the center of panel 210 is partial element 230C, and the rightmost region of panel 210 is partial element 230D.

Partial element 230A includes an external port 211 and ventilation holes 221 and 222 as functional parts. Partial elements 230B, 230C, and 230D each include a communication port 212 and ventilation holes 221 and 222 as functional parts. The trained model 600 recognizes partial elements 230A, 230B, 230C, and 230D based on the shape of the images of partial elements 230A, 230B, 230C, and 230D. Note that in the following description, partial elements 230A, 230B, 230C, and 230D may be collectively referred to as "partial elements 230."

An example of the trained model 600 will be described below. The trained model 600 is composed of a neural network. The trained model 600 includes an input layer, an intermediate layer, and an output layer. The input data of the trained model 600 is a camera image including a partial element 230. The trained model 600 recognizes the image of the partial element 230 in the camera image and locates the position of the image of the recognized partial element 230 in the camera image. The trained model 600 identifies the model of the communication device 200 from the pattern of the air vent 221 included in the recognized partial element 230. The output data of the trained model 600 is the identified model of the communication device 200 and position information in the image of the recognized partial element 230, i.e., the two-dimensional coordinate values of the screen, which is the display screen.

The construction of the trained model 600 will now be described. The trained model 600 is generated by machine learning using, as input data, an image of the panel 210 in which the partial elements 230 have been identified. The machine learning is performed, for example, by a learning device different from the maintenance support terminal 100. Specifically, an image including the partial elements 230 and the model name of the communication device 200 are input to the learning device as training data. The training data is created by semantic annotation that identifies the partial elements 230 in each of images captured of the communication device 200 from various angles. The machine learning is performed repeatedly using multiple training data. When a camera image including the partial elements 230 is input, the trained model 600 constructed by machine learning outputs position information of the partial elements 230 in the image and model information of the communication device 200.

A reference point 240 is predefined for the partial element 230. For example, when an image of partial element 230A shown in FIG. 9 is provided to the trained model 600, the trained model 600 outputs position information (two-dimensional coordinate values on the screen) of the reference point 240A of partial element 230A as position information of partial element 230A in the camera image. When an image of partial element 230B is provided to the trained model 600, the trained model 600 outputs position information of the reference point 240B of partial element 230B in the camera image. When an image of partial element 230C is provided to the trained model 600, the trained model 600 outputs position information of the reference point 240B of partial element 230C in the camera image. When an image of partial element 230D is provided to the trained model 600, the trained model 600 outputs position information of the reference point 240C of partial element 230D in the camera image.

Returning to Figure 8, the position measurement unit 109 measures the position in real space of the partial element 230 recognized by the trained model 600. Figure 10 is a diagram for explaining measurement of the position of a partial element by the position measurement unit. In the example shown in Figure 10, partial element 230A of panel 210 is image-recognized. The laser irradiation direction is identified from the position (two-dimensional coordinate values on the screen) of reference point 240A of partial element 230A in the camera image displayed on display 105. The laser irradiation unit 110 of the position measurement unit 109 irradiates a laser in the identified direction (toward reference point 240A). The laser is reflected at reference point 240A, and the reflected light is received by the light receiving unit 111 of the position measurement unit 109. By irradiating the laser and receiving the reflected light, the position measurement unit 109 measures the position of reference point 240A of partial element 230A in real space (three-dimensional coordinate system xyz).

Returning to Figure 8, the trained model 600 is provided with camera images of at least two partial elements 230 in one communication device 200. The position measurement unit 109 measures the positions of the reference points 240 of the at least two partial elements 230. For example, in Figure 9, the trained model 600 recognizes any one of the combination of partial elements 230A and 230D, the combination of partial elements 230A and 230B, and the combination of partial elements 230C and 230D.

The virtualization unit 122 determines a representative position of the panel 210 based on the position of the partial element 230 measured by the position measurement unit 109. An example of the representative position of the panel 210 is the center point of the panel 210. Figure 11 is a diagram for explaining the arrangement of a virtual panel in virtual space by the virtualization unit according to this embodiment. Figure 11 is a plan view of the panel 210 as viewed from above. For example, when partial elements 230A and 230D are recognized by the trained model 600, the position measurement unit 109 measures the position of reference point 240A (10, 50, -20) and the position of 240D (80, 47, -15) in the real space xyz. The virtualization unit 122 determines the midpoint (45, 48.5, -17.5) of reference points 240A and 240D in the real space xyz as the representative position 210P.

The virtualization unit 122 selects a virtual panel 700 from among the multiple virtual panels 700A, 700B, and 700C that matches the model of the communication device 200 recognized by the trained model 600.

The virtualization unit 122 places the selected virtual panel 700 at a position corresponding to the representative position 210P in the virtual space x'y'z', which corresponds to the real space xyz. The virtual space x'y'z' has virtual coordinate axes x', y', z' that correspond to the x, y, and z coordinate axes of the real space xyz. In the example of Figure 11, the virtualization unit 122 places the representative position 210P of the virtual panel 700 at coordinates (45, 48.5, -17.5) in the virtual space x'y'z', matching the representative position 210P (45, 48.5, -17.5) of the panel 210 in the real space xyz.

More specifically, the virtualization unit 122 positions the virtual panel 700 in the virtual space x'y'z' at an angle corresponding to the angle of the panel 210 in the real space xyz, based on the positions of each of the multiple partial elements 230 in the real space xyz. In the example of FIG. 11, the virtualization unit 122 positions the reference point 240A on the virtual panel 700 at coordinates (10, 50, -20) in the virtual space x'y'z' to match the position (10, 50, -20) of the reference point 240A of the partial element 230A in the real space xyz, and positions the reference point 240A on the virtual panel 700 at coordinates (80, 47, -15) in the virtual space x'y'z' to match the position (80, 47, -15) of the reference point 240D of the partial element 230D in the real space xyz.

Here, the angle of the panel 210 refers to the angle in the horizontal direction (e.g., the angle with respect to the x-axis). Figure 12 is a diagram illustrating the relationship between the position of the panel in real space and the position of the virtual panel in virtual space. In this embodiment, it is assumed that the communication device 200 is installed so that the panel 210 is vertical. In other words, the panel 210 is always vertical and not tilted. Therefore, by placing the virtual panel 700 in the virtual space x'y'z' so that the position of the reference point 240 of the partial element 230 in the real space xyz coincides with the position corresponding to the reference point 240 of the virtual panel 700 in the virtual space x'y'z' as described above, the angle of the panel 210 in the real space xyz can be aligned with the angle in the virtual space x'y'z'.

Returning to FIG. 8 , the virtualization unit 122 determines whether the multiple partial elements 230 recognized by the trained model 600 are partial elements 230 of the same communication device based on the positions of the multiple partial elements 230 measured by the position measurement unit 109. In a specific example, the virtualization unit 122 determines whether the positions of the multiple partial elements 230 measured by the position measurement unit 109 meet a judgment condition, which is a condition for the positional relationship of the multiple partial elements 230. If the positions of the multiple partial elements 230 measured by the position measurement unit 109 meet the judgment condition, the virtualization unit 122 determines that the multiple partial elements 230 recognized by the trained model 600 are partial elements 230 of the same communication device. If the positions of the multiple partial elements 230 measured by the position measurement unit 109 do not meet the judgment condition, the virtualization unit 122 determines that the multiple partial elements 230 recognized by the trained model 600 are not partial elements 230 of the same communication device.

Figure 13A is a diagram showing a first example of the positional relationship of partial elements in a communication device. In this embodiment, the communication device 200 is a rectangular parallelepiped, and it is assumed that the top and bottom surfaces of the communication device 200 are arranged horizontally in the rack 250. In Figure 13A, the heights (z coordinates) of partial elements 230A_1 and 230D_1 included in panel 210_1 of communication device 200_1 are the same. The height of partial element 230A_1 included in panel 210_1 of communication device 200_1 and the height of partial element 230D_2 included in panel 210_2 of communication device 200_2 are different from each other. When partial elements 230A_1 and 230D_1 are recognized by the trained model 600, the difference in z-coordinate between reference point 240A_1 of partial element 230A_1 and reference point 240D_1 of partial element 230D_1 is within the allowable range, so the virtualization unit 122 determines that partial elements 230A_1 and 230D_1 are partial elements of the same communication device 200_1. When partial elements 230A_1 and 230D_2 are recognized by the trained model 600, the difference in z-coordinate between reference point 240A_1 of partial element 230A_1 and reference point 240D_2 of partial element 230D_2 exceeds the allowable range, so the virtualization unit 122 determines that partial elements 230A_1 and 230D_2 are not partial elements of the same communication device.

Figure 13B is a diagram showing a second example of the positional relationship of partial elements in a communication device. In the example shown in Figure 13B, two communication devices 200_3 and 200_4 are aligned in the horizontal direction (x direction) and arranged facing the same direction with their positions shifted in the depth direction (y direction). For simplicity of explanation, in the example of Figure 13B, the longitudinal direction of panels 210_3 and 210_4 is taken as the x direction, and the depth direction of communication devices 200_3 and 200_4 is taken as the y direction. The depth direction positions (y coordinates) of partial elements 230A_3 and 230D_3 included in panel 210_3 of communication device 200_3 are the same. The depth direction positions of partial elements 230A_4 and 230D_4 included in panel 210_4 of communication device 200_4 are the same. The depth direction position of partial element 230A_3 included in panel 210_3 of communication device 200_3 is different from the depth direction position of partial element 230D_4 included in panel 210_4 of communication device 200_4. For example, when a user successively captures images of partial elements 230A_3 and 230D_3 of panel 210_3 of one communication device 200_3 with camera 108, partial elements 230A_3 and 230D_3 are recognized by trained model 600. In this case, the difference in the x-coordinate between the reference point 240A_3 of the partial element 230A_3 and the reference point 240D_3 of the partial element 230D_3 matches the length of the panel 210_3, and the difference in the y-coordinate between the reference point 240A_3 and the reference point 240D_3 is within the allowable range. Therefore, the virtualization unit 122 determines that the partial elements 230A_3 and 230D_3 are partial elements of the same communication device 200_3. It is conceivable that a user may capture a video using the camera 108 while changing the position or orientation of the maintenance support terminal 100. A partial element recognized in a certain screen of the video is accumulated in three-dimensional space. In other words, even if the field of view of the camera 108 changes after the partial element is captured and the partial element is no longer included in the screen, the partial element is not deleted from the three-dimensional space. For example, if a user captures an image of partial element 230A_3 of panel 210_3 of communication device 200_3 with camera 108, then moves to the front of communication device 200_4 and points camera 108 toward panel 210_4 of communication device 200_4 to capture an image of partial element 230D_4, partial element 230A_3 and partial element 230D_4 are captured. In this case, partial elements 230A_3 and 230D_4 are recognized by trained model 600. The difference in the x-coordinate between reference point 240A_3 of partial element 230A_3 and reference point 240D_4 of partial element 230D_4 matches the length of panel 210_3, but because the difference in the y-coordinate between reference point 240A_3 and reference point 240D_4 exceeds the allowable range, the virtualization unit 122 determines that partial elements 230A_3 and 230D_4 are not partial elements of the same communication device.

Figure 13C is a diagram showing a third example of the positional relationship of partial elements in a communication device. In the example shown in Figure 13D, two communication devices 200_5 and 200_6 are aligned in the horizontal direction (x direction) and positioned in opposite directions with a shift in the depth direction (y direction). In the example of Figure 13C, for simplicity of explanation, the longitudinal direction of panels 210_5 and 210_6 is defined as the x direction, and the depth direction of communication devices 200_5 and 200_6 is defined as the y direction. The depth direction positions (y coordinates) of partial elements 230A_5 and 230D_5 included in panel 210_5 of communication device 200_5 are the same. The depth direction positions of partial elements 230A_6 and 230D_6 included in panel 210_6 of communication device 200_6 are the same. The depth direction position of partial element 230A_5 included in panel 210_5 of communication device 200_5 is different from the depth direction position of partial element 230D_6 included in panel 210_6 of communication device 200_6. When partial elements 230A_5 and 230D_5 are recognized by trained model 600, the difference in the x coordinate between reference point 240A_5 of partial element 230A_5 and reference point 240D_5 of partial element 230D_5 matches the length of panel 210_5, and the difference in the y coordinate between reference point 240A_5 and reference point 240D_5 is within the allowable range, so the virtualization unit 122 determines that partial elements 230A_5 and 230D_5 are partial elements of the same communication device 200_5. When partial elements 230A_5 and 230D_6 are recognized by trained model 600, the difference in the x-coordinate between reference point 240A_5 of partial element 230A_5 and reference point 240D_6 of partial element 230D_6 is smaller than the allowable range. Therefore, even if the difference in the y-coordinate between reference point 240A_5 and reference point 240D_6 is within the allowable range, virtualization unit 122 determines that partial elements 230A_5 and 230D_6 are not partial elements of the same communication device.

As described above, if it is determined that the two recognized partial elements 230 are not partial elements of the same communication device, the virtualization unit 122 does not perform placement of the virtual panel 700 in the virtual space x'y'z'.

Returning to FIG. 8 , the display control unit 123 superimposes on the display 105 the virtual panel 700 placed in the virtual space x'y'z' by the virtualization unit 122 and the image of the panel 210 captured by the camera 108. That is, the virtual panel 700 is superimposed only on the panel 210 that appears (is image-recognized) in the image captured by the camera 108. More specifically, the display control unit 123 superimposes the virtual panel 700 placed in the virtual space x'y'z' by the virtualization unit 122 on the image of the panel 210 captured by the camera 108, as if it were captured by a camera in the virtual space x'y'z'. FIG. 14 is a diagram showing an example of an AR image displayed on a display. For simplicity's sake, FIG. 14 omits the optical fiber cable extending from the optical transceiver 280 inserted into the communication port 212 from partway through. The display 105 displays an AR image 900 in which the virtual panel 700 is superimposed on the camera image 910. The position of the virtual panel 700 in the virtual space x'y'z' is aligned with the position of the panel 210 in the real space xyz, so in the AR image 900, the virtual panel 700 is overlaid on the position of the panel 210 in the camera image 910. However, because the virtual panel 700 is transparent, the user cannot recognize that the virtual panel 700 is overlaid on the camera image 910.

Returning to FIG. 8, the notification receiving unit 124 is capable of receiving notification of the status of the communication port 212 in the communication device 200. The notification receiving unit 124 receives notification of an abnormal communication port 212 in the communication device 200. In a specific example, if an abnormality occurs in the communication port 212, the communication device 200 detects the abnormality. The communication device 200 notifies the server 300 of the abnormality in the communication port 212. The notification information of the abnormality state includes the port number of the communication port 212 in which the abnormality occurred and the type of abnormality. The server 300 records the notified abnormal state in the communication device DB 306 (see FIG. 6). The communication device DB 306 stores identification information of the communication device 200 (e.g., serial number), the port number of the communication port in which the abnormality occurred, and the type of abnormality, in association with each other. When the server 300 receives notification of an abnormality in the communication port 212 from the communication device 200, it notifies the maintenance support terminal 100 of the abnormality in the communication port 212. The notification reception unit 124 receives notification of an abnormality in the communication port 212 from the server 300.

The notification from the server 300 of an abnormality in the communication port 212 may be a push-type notification or a pull-type notification. In other words, the server 300 may notify the maintenance support terminal 100 of an abnormality in the communication port 212 without the maintenance support terminal 100 making an inquiry to the server 300, or the server 300 may notify the maintenance support terminal 100 of an abnormality in the communication port 212 when an inquiry is made from the maintenance support terminal 100.

The part model 702 is set to a normal color and an abnormal color. The normal color is the color used when the communication port 212 corresponding to the part model 702 is normal. The abnormal color is the color used when the communication port 212 corresponding to the part model 702 is abnormal. When the notification receiving unit 124 receives a notification of an abnormal communication port 212, the display control unit 123 switches the color of the part model 702 corresponding to the communication port 212 from the normal color to the abnormal color.

Figure 15 is a diagram showing an example of an AR image when a communication port abnormality occurs. In this embodiment, when a communication port 212 is normal, the part model 702 corresponding to that communication port 212 is transparent. Therefore, the normal color is transparent. When a communication port 212 abnormality is notified from the server 300, the color of the part model 702a corresponding to the communication port 212 where the abnormality occurred changes to an opaque abnormal color. The abnormal color is, for example, red. In the AR image 900 on the display 105, the abnormal communication port 212 is displayed in red, and the normal communication port 212 is not displayed in red. Therefore, the user can recognize the status of the corresponding communication port 212 from the color of the part model 702 on the virtual panel 700.

Returning to Figure 8, a non-selected color and a selected color are set for the part model 702. The non-selected color is the color when the communication port 212 corresponding to the part model 702 is not selected. The selected color is the color when the communication port 212 corresponding to the part model 702 is selected. When the display control unit 123 receives selection conditions for selecting a communication port 212, it switches the color of the part model 702 corresponding to the communication port 212 that matches the selection conditions from the non-selected color to the selected color, among the multiple part models 702.

Figure 16 shows an example of an AR image when some communication ports are selected. In the example of Figure 16, the selection condition is "communication port 212 compatible with 1 Gbps optical transceiver 280." Selection conditions are entered by performing a specified operation on the maintenance support terminal 100. The user can input a display instruction for the selection condition input section into the input device 104 of the maintenance support terminal 100. When this instruction is input to the input device 104 of the maintenance support terminal 100, the display control unit 123 superimposes the selection condition input section 920 on the AR image 900 on the display 105. The selection condition input section 920 has two check boxes for selecting 1 Gbps and 10 Gbps. The user can enter selection conditions by selecting either check box.

The display control unit 123 selects a communication port 212 that matches the selection conditions from among the multiple communication ports 212 of the communication device 200. For example, the maintenance support terminal 100 can acquire information about each communication port 212 from the communication device DB 306 of the server 300, and use the acquired information to select a communication port 212 that matches the selection conditions.

In this embodiment, if a communication port 212 is not selected, the part model 702 corresponding to that communication port 212 is transparent. Therefore, the non-selected color is transparent. When selection conditions for a communication port 212 are input, the color of the part model 702b corresponding to the communication port 212 that matches the selection conditions changes to an opaque selection color. The selection color is, for example, yellow. In the example of FIG. 16, the part model 702b corresponding to the communication port 212 corresponding to the 1 Gbps optical transceiver 280 is displayed in the selection color. In the AR image 900 on the display 105, the communication port 212 that matches the selection conditions is displayed in yellow, and the communication port 212 that does not match the selection conditions is not displayed in yellow. Therefore, the user can recognize the communication ports 212 that match the selection conditions by the color of the part model 702 on the virtual panel 700.

Returning to FIG. 8 , the maintenance management terminal 400 can accept input of work information indicating the designation of a communication port 212 of the communication device 200 and the work to be performed on the designated communication port 212. In one specific example, the display control unit 411 of the maintenance management terminal 400 causes the display 405 to display a virtual panel 710 corresponding to the panel 210 of the communication device 200. The maintenance management terminal 400 accepts the designation of the communication port 212 when the input device 404 accepts input specifying the communication port 212 of the virtual panel 710. In other words, the administrator can select a component model corresponding to the communication port 212 by tapping, clicking, or other operations on the virtual panel 710 displayed on the display 405. When a component model is selected, the maintenance management terminal 400 accepts the designation of the communication port 212 corresponding to the selected component model.

The administrator can use the input device 404 to input work information into the maintenance management terminal 400. The work information is text information that includes the work content.

The communication unit 412 of the maintenance management terminal 400 transmits the input work information and designation information indicating the designated communication port 212 to the server 300. In a specific example, the designation information includes the port number of the designated communication port 212 or information for identifying the selected part model. The server 300 forwards the received designation information and work information to the maintenance support terminal 100. The receiving unit 125 of the maintenance support terminal 100 receives the designation information and work information transmitted from the maintenance management terminal 400 and forwarded by the server 300.

The part model 702 is set to a non-designated job and a designated color. The non-designated color is the color when the communication port 212 corresponding to the part model 702 is not designated. The designated color is the color when the communication port 212 corresponding to the part model 702 is designated. The display control unit 123 switches the color of the part model corresponding to the communication port 212 designated in the designation information received by the receiving unit 125 from the non-designated color to the designated color. Furthermore, the display control unit 123 causes the display 105 to display the work information received by the receiving unit 125.

Figure 17 shows an example of an AR image in which a communication port is specified and work information is displayed. In this embodiment, if a communication port 212 is not specified, the part model 702 corresponding to that communication port 212 is transparent. Therefore, the unspecified color is transparent. If some communication ports 212 are specified, the color of the part model 702c corresponding to the specified communication ports 212 changes to an opaque specified color. The specified color is, for example, blue. In the AR image 900 on the display 105, the specified communication port 212 is displayed in blue, and the unspecified communication ports 212 are not displayed in blue. Therefore, the user can recognize the specified communication port 212 by the color of the part model 702 on the virtual panel 700.

The display control unit 123 superimposes a work instruction section 930 containing work information on the AR image 900 on the display 105. In the example shown in FIG. 17, the work instruction section 930 displays work information including the date and time the work information was entered, the affiliation and name of the administrator who issued the work instruction, and the work content. The user can recognize the content of the work that should be performed on the specified communication port 212.

Returning to FIG. 8 , when the input device 104 receives input specifying a component model 702 in the virtual panel 700, the display control unit 123 causes the display 105 to display attribute information, status information, and communication volume information (hereinafter, information including attribute information, status information, and communication volume information will be referred to as "port information") related to the communication port 212 corresponding to the component model 702. The attribute information is information indicating the attributes of the optical transceiver 280 attached to the communication port 212 corresponding to the specified component model 702. The status information is information indicating the status of the communication port 212. The communication volume information is information indicating the communication volume of the communication port 212. As a result, by specifying a component model 702 in the virtual panel 700, the user can cause the display 105 to display the attribute information, status information, and communication volume information of the communication port 212 corresponding to the component model 702.

For example, the display control unit 123 may switch the color of the part model 702 corresponding to the communication port 212 specified by the user to a specified color. The specified color of the part model 702 specified by the user may be the same as or different from the specified color of the part model 702c specified by the administrator.

Figure 18 shows an example of an AR image displaying port information. The example in Figure 18 displays port information for the component model 702a corresponding to the communication port 212 where the abnormality occurred in the example in Figure 15.

The display control unit 123 superimposes a port information display unit 940 containing port information on the AR image 900 on the display 105. In the example shown in FIG. 18, the attribute information includes the vendor name, model number, serial number, type (compatible communication standard), transmission power, reception power, bias current, and temperature of the optical transceiver 280. The status information includes whether the transmission power is normal or abnormal, whether the reception power is normal or abnormal, whether the bias current is normal or abnormal, and whether the temperature is normal or abnormal. The communication volume information includes the number of received octets, the number of received unicast packets, the number of received broadcast packets, the number of received multicast packets, the number of received packets discarded, the number of reception errors, the number of transmitted octets, the number of transmitted unicast packets, the number of transmitted broadcast packets, the number of transmitted multicast packets, the number of transmitted packets discarded, and the number of transmission errors.

The port information may include a graph showing numerical information of the communication port 212 in a time series. For example, the display control unit 123 may display transmission power, reception power, bias current, or temperature in a time series graph.

In the example of Figure 18, an abnormal temperature has occurred in the optical transceiver 280 inserted into the communication port 212 corresponding to the component model 702a. "ALARM," indicating an abnormality, is displayed in the port information display section 940 in response to the temperature in the status information. This allows the user to recognize that an abnormal temperature has occurred in the optical transceiver 280 inserted into the communication port 212 displayed in red.

Returning to FIG. 8 , when the notification receiving unit 124 receives notification of an abnormal state of the communication port 212, the display control unit 123 causes the display 105 to display first handling information indicating a first-stage method of dealing with the abnormality. When the state of the communication port 212 changes as a result of the first-stage handling method being executed and the notification receiving unit 124 receives notification of the changed state of the communication port 212, the display control unit 123 switches the first handling information displayed on the display 105 to second handling information indicating a second-stage method of dealing with the abnormality. This allows the user to proceed with the work of dealing with the abnormality of the communication port 212 in accordance with the AR display on the display 105.

Returning to Figure 18, to the right of "ALARM" for the temperature in the status information of the port information display section 940, there is a button 940a for calling up a troubleshooting tutorial. The user can select button 940a using the input device 104. When button 940a is selected, a tutorial is displayed that shows step-by-step instructions on how to deal with abnormalities.

Figures 19A to 19D are diagrams showing examples of AR images displaying information for dealing with an abnormality. When button 940a in Figure 18 is selected, the display control unit 123 queries the communication device DB 306 of the server 300 and acquires first action information indicating a first-stage action method for dealing with an abnormality in the communication port 212. As shown in Figure 19A, the display control unit 123 superimposes a tutorial section 950A containing the acquired first action information on the AR image 900 on the display 105. The tutorial section 950A includes text information as the first action information: "The transceiver needs to be replaced. Please remove the transceiver in port #3."

The tutorial section 950A enables the user to recognize that the first step in recovering from the abnormality is to remove the optical transceiver 280 from the communication port 212 with port number 3, which is displayed in red in the AR image 900. The user removes the optical transceiver 280 from the communication port 212 with port number 3 in accordance with the first action information.

Returning to FIG. 8, when the optical transceiver 280 is removed from the communication port 212 corresponding to port number 3, the communication device 200 detects the removal of the optical transceiver 280 from that communication port 212. The communication device 200 notifies the server 300 of status information indicating that the optical transceiver 280 has been removed from the communication port 212 corresponding to port number 3. The server 300 records the notified status information in the communication device DB 306 and notifies the maintenance support terminal 100 of the status information. Furthermore, the server 300 obtains second handling information indicating a second-stage handling method from the communication device DB 306 and transmits it to the maintenance support terminal 100.

When the notification receiving unit 124 receives the status information and second handling information of the communication device 200, the display control unit 123 switches the tutorial section 950A of the AR image 900 to a tutorial section 950B containing the acquired second handling information, as shown in FIG. 19B. The tutorial section 950B contains the following text information as the second handling information: "It has been confirmed that the transceiver in port #3 has been removed. Please insert a transceiver into port #3."

The tutorial section 950B allows the user to recognize that the second-stage solution to recover from the abnormality is to insert a new optical transceiver 280 into the communication port 212 with port number 3. The user inserts the new optical transceiver 280 into the communication port 212 with port number 3 in accordance with the second solution information.

Returning to FIG. 8, when the optical transceiver 280 is inserted into the communication port 212 with port number 3, the communication device 200 detects the insertion of the optical transceiver 280 into the communication port 212. The communication device 200 begins initial configuration of the inserted optical transceiver 280 and notifies the server 300 of status information indicating that the optical transceiver 280 has been inserted into the communication port 212 with port number 3. The server 300 records the notified status information in the communication device DB 306 and notifies the maintenance support terminal 100 of the status information. Furthermore, the server 300 obtains third handling information indicating a third-stage handling method from the communication device DB 306 and transmits it to the maintenance support terminal 100.

When the notification receiving unit 124 receives the status information and third handling information of the communication device 200, the display control unit 123 switches the tutorial section 950B of the AR image 900 to a tutorial section 950C containing the acquired third handling information, as shown in FIG. 19C. The tutorial section 950C contains the following text information as the third handling information: "Insertion of a transceiver into port #3 has been confirmed. Initial settings for the transceiver are being performed. Please wait a moment."

The tutorial section 950C allows the user to recognize that the third-stage solution to recover from the abnormality is to wait until the initial setup of the optical transceiver 280 inserted into the communication port 212 with port number 3 is complete. The user waits in accordance with the third solution information.

Returning to FIG. 8, when the initial setup of the optical transceiver 280 inserted into the communication port 212 of port number 3 is completed, the communication device 200 detects that the initial setup has been completed successfully. Furthermore, if the abnormality in the communication port 212 of port number 3 has been resolved, the communication device 200 detects that the communication port 212 is normal. The communication device 200 notifies the server 300 of status information indicating that the initial setup of the optical transceiver 280 in the communication port 212 of port number 3 has been completed and that the communication port is normal. The server 300 records the notified status information in the communication device DB 306 and notifies the maintenance support terminal 100 of the status information. Furthermore, the server 300 obtains completion information from the communication device DB 306 indicating that the response method has been completed, and transmits this information to the maintenance support terminal 100.

When the notification receiving unit 124 receives the status information and termination information of the communication device 200, the display control unit 123 switches the tutorial section 950C of the AR image 900 to a tutorial section 950D containing the acquired termination information, as shown in FIG. 19D. The tutorial section 950D contains the text information "Initial settings of the transceiver have been completed successfully. Recovery from the abnormal state has been confirmed" as termination information. The tutorial section 950D allows the user to recognize that recovery from the abnormality has been completed.

[6. Operation of Maintenance Support System]
The following describes the operation of the maintenance support system 10 according to the embodiment. The processor 101 of the maintenance support terminal 100 executes the maintenance support APP 120 to perform the following AR display processing, port abnormality processing, selection condition display processing, port information display processing, tutorial processing, and work instruction display processing.

Figure 20 is a flowchart showing an example of AR display processing by the maintenance support terminal according to an embodiment.

The panel 210 of the communication device 200 is imaged by the camera 108 of the maintenance support terminal 100. The processor 101 captures the camera image 910 output from the camera 108 (step S101).

The processor 101 inputs the camera image 910 into the trained model 600. The trained model 600 recognizes the partial element 230 and recognizes the model of the communication device 200 from the pattern of the air vent 221 (step S102).

The processor 101 uses the position measurement unit 109 to measure the positions of the reference points 240 in the real space xyz for the two partial elements 230 (step S103).

The processor 101 determines the representative position 210P of the panel 210 from the positions of the two reference points 240 (step S104).

The processor 101 selects a virtual panel 700 corresponding to the recognized model and places the selected virtual panel 700 in virtual space x'y'z' (step S105). At this time, the processor 101 places the virtual panel 700 in virtual space x'y'z by aligning the representative position 210P (center point) of the virtual panel 700 corresponding to the representative position 210P of the panel 210 with the representative position 210P in real space xyz. Furthermore, the processor 101 places the virtual panel 700 in virtual space x'y'z' so that the position of the reference point 240 of the partial element 230 in real space xyz coincides with the position of the reference point 240 of the virtual panel 700 in virtual space x'y'z', and aligns the angle of the panel 210 in real space xyz with the angle of the virtual space x'y'z'.

The processor 101 generates an AR image 900 by superimposing the virtual panel 700 on the camera image 910, and displays the AR image 900 on the display 105 (step S106). The processor 101 repeats the above AR display process at predetermined sampling intervals.

The following port abnormality processing, selection condition display processing, port information display processing, tutorial processing, and work instruction display processing are executed while the AR image 900 is displayed on the display 105.

Figure 21 is a flowchart showing an example of port abnormality processing by the maintenance support terminal according to an embodiment.

When the communication device 200 detects an abnormality in the communication port 212, it notifies the server 300 of the abnormality in the communication port 212. The server 300 forwards the abnormality notification in the communication port 212 to the maintenance support terminal 100. The maintenance support terminal 100 receives the abnormality notification in the communication port 212 of the communication device 200 (step S201).

The processor 101 switches the color of the component model 702 corresponding to the communication port 212 in which the abnormality occurred to an abnormal color (step S2020). This completes the port abnormality processing.

Figure 22 is a flowchart showing an example of the selection condition display process performed by the maintenance support terminal according to the embodiment.

The user inputs a display instruction for the selection condition input section 920 to the input device 104. This causes the selection condition input section 920 to be displayed superimposed on the AR image 900. The user inputs selection conditions for the communication port 212 by selecting checkboxes in the selection condition input section 920. The processor 101 accepts the input of the selection conditions (step S301).

The processor 101 obtains information about each communication port 212, for example, from the communication device DB 306 of the server 300, and selects a communication port 212 that meets the selection conditions from among the multiple communication ports 212 of the communication device 200 (step S302).

The processor 101 switches the color of the part model 702 corresponding to the selected communication port 212 to the selected color (step S303). This completes the selection condition display process.

Figure 23 is a flowchart showing an example of the port information display process performed by the maintenance support terminal according to the embodiment.

The user specifies the part model 702 (communication port 212) in the virtual panel 700 in the AR image 900 by tapping or clicking on it. The processor 101 accepts the specification of the part model 702 (step S401).

The processor 101 obtains port information for the communication port 212 corresponding to the specified component model 702, for example, by querying the communication device DB 306 of the server 300 (step S402).

The processor 101 processes the acquired port information (step S403). For example, the processor 101 creates a graph from time-series numerical information on transmit power, receive power, bias current, or temperature.

The processor 101 displays the port information display section 940 containing the port information superimposed on the AR image 900 (step S404). This completes the port information display process.

Figure 24 is a flowchart showing an example of tutorial processing by the maintenance support terminal according to an embodiment.

When the component model 702 corresponding to the communication port where the abnormality occurred is displayed in an abnormal color in the AR image 900 as a result of the abnormal port processing described above, and the port information display section 940 is displayed as a result of the port information display processing described above, the user can instruct the display of a tutorial by selecting button 940a (see FIG. 18). The processor 101 accepts a tutorial display instruction from the user (step S501).

The processor 101 acquires countermeasure information that shows step-by-step countermeasure methods for dealing with an abnormality in the communication port 212 (step S502). Next, the processor 101 displays a tutorial section including the acquired countermeasure information superimposed on the AR image 900 (step S503).

The user responds to the abnormality in accordance with the response information. This changes the state of the communication port 212, and the communication device 200 detects this change in state. When the communication device 200 detects the change in the state of the communication port 212, it notifies the server 300 of the new state of the communication port 212. The server 300 forwards the state notification of the communication port 212 to the maintenance support terminal 100.

The processor 101 determines whether a status notification has been received for the communication port 212 of the communication device 200 (step S504). If the maintenance support terminal 100 has not received a status notification (NO in step S504), the processor 101 returns to step S504.

If the maintenance support terminal 100 receives a status notification (YES in step S504), the processor 101 checks the notified status and determines whether the abnormality in the communication port 212 has been resolved (step S505). If the abnormality in the communication port 212 has not been resolved (NO in step S505), the processor 101 returns to step S502 and acquires the next stage of handling information. This causes the displayed handling information to switch to the next stage.

If the abnormality in the communication port 212 has been resolved (YES in step S505), the processor 101 switches the color of the part model 702 corresponding to the communication port 212 whose abnormality has been resolved to a normal color (step S506) and stops displaying the troubleshooting information (step S507). This completes the tutorial processing.

Figure 25 is a sequence diagram showing an example of work instruction processing by the maintenance support system according to the embodiment. In the work instruction processing, the maintenance management terminal 400 executes work instruction notification processing, and the maintenance support terminal 100 executes work instruction display processing.

In the work instruction notification process, the processor 401 of the maintenance management terminal 400 displays the virtual panel 710 on the display 405 in response to instructions input by the administrator (step S601).

The administrator specifies the part model corresponding to the communication port 212 for which the work instruction is to be issued on the virtual panel 710. The processor 401 accepts the part model specification (step S602).

The administrator uses the input device 404 to input work information, which is text information indicating the work to be instructed, into the maintenance management terminal 400. The processor 401 accepts the input of the work information (step S603).

The maintenance management terminal 400 transmits the specification information indicating the communication port 212 corresponding to the specified part model and the input work information to the server 300 (step S604). The server 300 receives the specification information and work information and transfers the received specification information and work information to the maintenance support terminal 100 (step S605).

The maintenance support terminal 100 receives the specification information and work information. In the work instruction display process, the processor 101 of the maintenance support terminal 100 switches the color of the part model 702 corresponding to the communication port 212 specified in the specification information to the specified color (step S606). The processor 101 displays the work instruction section 930 including the received work information superimposed on the AR image 900 (step S607).

7. Modifications
FIG. 26 is a diagram illustrating a modified example of a panel of a communication device according to an embodiment. A communication device 200a according to this modified example allows for the attachment and detachment of communication modules. A panel 210a of the communication device 200a is provided with openings 213A, 213B, and 213C. Panels 270A, 270B, and 270C of the attached communication modules 260A, 260B, and 260C are exposed through the openings 213A, 213B, and 213C. That is, a portion of the panel 210a is the panel 270A, 270B, and 270C of the communication modules 260A, 260B, and 260C attached to the communication device 200a. A plurality of communication ports 212 are provided on each of the panels 270A, 270B, and 270C of the communication modules 260A, 260B, and 260C.

A ventilation opening 221aA is provided in panel 270A of communication module 260A. A ventilation opening 221aB is provided in panel 270B of communication module 260B. A ventilation opening 221aC is provided in panel 270C of communication module 260C. Ventilation openings 221aA, 221aB, and 221aC are each provided above communication port 212, i.e., at the top end of panels 270A, 270B, and 270C.

Ventilation openings 221aA, 221aB, and 221aC are used to identify the models of communication modules 260A, 260B, and 260C. In other words, ventilation openings 221aA, 221aB, and 221aC also serve as identification codes for the models of communication modules 260A, 260B, and 260C. Ventilation openings 221aA provided in panel 230A of communication module 260A are arranged in a pattern unique to communication module 260A. Ventilation openings 221aB provided in panel 230B of communication module 260B are arranged in a pattern unique to communication module 260B. Ventilation openings 221aC provided in panel 230C of communication module 260C are arranged in a pattern unique to communication module 260C.

As a result, in a communication device 200a capable of mounting communication modules 260A, 260B, and 260C, the models of communication modules 260A, 260B, and 260C can be identified by image recognition of the ventilation openings 221aA, 221aB, and 221aC.

A plurality of ventilation holes 222a are provided above openings 213A, 213B, and 213C on panel 210a of communication device 200a. That is, the plurality of ventilation holes 222a are provided in a different portion of panel 210a from panels 230A, 230B, and 230C of the communication module. Ventilation holes 222a are used to identify positions within panel 210a. In other words, ventilation holes 222a also serve as identification codes for positions within panel 210a. The arrangement pattern and uses of ventilation holes 222a are the same as those of ventilation holes 222 described in the embodiment.

In the above-described embodiment, the maintenance support terminal 100 functions as the recognition unit 121, virtualization unit 122, and display control unit 123, places the virtual panel 700 in the virtual space x'y'z', and generates the AR image 900, but this is not limited to this. The server 300 may also be configured to place the virtual panel 700 in the virtual space x'y'z', generate the AR image 900, and transmit data for displaying the AR image 900 to the maintenance support terminal 100. In other words, the server 300 may function as the recognition unit 121, virtualization unit 122, and display control unit 123.

In the above-described embodiment, the trained model 600 identified the model of the communication device 200 from the pattern of the vents 221 included in the partial elements 230, but this is not limited to this. For example, the model name or model code may be printed on the panel 210, and the model of the communication device 200 may be identified by image recognition of the model name or model code. As yet another example, the model of the communication device 200 may be identified by image recognition of the shape of all or part of the panel 210. In this case, the vents 221 may be arranged in a pattern common to multiple models of communication device 200, rather than in a pattern specific to the model of the communication device 200.

In the above-described embodiment, the maintenance support terminal 100 performs image recognition of the arrangement pattern of the ventilation holes 222 to identify the position (left, center, right) of the recognized portion on the panel 210, but this is not limited to this. For example, port numbers may be printed near the communication ports 212 on the panel 210, and the position on the panel 210 of the communication port corresponding to the recognized port number may be identified by image recognition of the port numbers. In this case, there is no need to change the pattern of the ventilation holes 222 for each position on the panel 210; for example, the ventilation holes 222 may be arranged in the same pattern across the entire panel 210.

In the above-described embodiment, the position of the partial element 230 is measured using a ray casting method, but this is not limiting. For example, the position of the partial element 230 may be measured by capturing an image of the partial element 230 with a compound eye camera.

In the above-described embodiment, a video was captured by the camera 108, and the AR display process was sequentially repeated to display the AR image 900 in the form of a video, in which the virtual panel 700 was superimposed on the camera image 910. However, this is not limited to this. The virtual panel 700 may be superimposed on a still image captured by the camera 108, and the AR image of the still image may be displayed.

In the above-described embodiment, partial elements 230 of panel 210 are captured by camera 108, and the partial elements 230 included in camera image 910 are recognized by trained model 600, but this is not limited to this. The entire panel 210 may also be captured by camera 108, and the entire panel 210 included in camera image 910 may be recognized by trained model 600. In this case, for example, a specific position of the recognized panel 210 (e.g., the top left and top right corners of panel 210) may be measured, and a representative position may be determined based on the measured specific position.

[8. Supplementary Notes]
The embodiments disclosed herein are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims rather than the above-described embodiments, and includes meanings equivalent to the claims and all modifications within the scope thereof.

10 Maintenance support system 20 Communication center 30 Management center 100 Maintenance support terminal (maintenance support device)
101 Processor 102 Non-volatile memory 103 Volatile memory 104 Input device (first input device)
105 Display (first display)
106 Graphics processing unit (first display control unit)
107 Communication interface (communication I/F)
108 Camera 109 Position measurement unit 110 Laser irradiation unit 111 Light receiving unit 112 Gyro sensor 113 Acceleration sensor 120 Maintenance support application (maintenance support APP)
121 Recognition unit 122 Virtualization unit 123 Display control unit 124 Notification reception unit 125 Reception unit 200, 200A, 200B, 200C, 200D, 200E, 200F, 200G, 200H, 200_1, 200_2, 200_3, 200_4, 200_5, 200_6, 200a Communication device 210, 210_1, 210_2, 210_3, 210_4, 210_5, 210_6, 210a Panel 210P Representative position 211 External port 212 Communication port 213A, 213B, 213C Opening 221, 221A, 221B, 221aA, 221aB, 221aC Ventilation holes 222, 222A, 222B, 222C, 222a Ventilation holes 230, 230A, 230B, 230C, 230D, 230A_1, 230D_1, 230D_2, 230A_3, 230D_3, 230A_4, 230D_4, 230A_5, 230D_5, 230A_6, 230D_6 Partial elements 240, 240A, 240B, 240C, 240D, 240A_1, 240D_1, 240D_2, 240A_3, 240D_3, 240D_4, 240A_5, 240D_5, 240D_6 Reference points 250, 250A, 250B Rack 260A, 260B, 260C Communication module 270A, 270B, 270C Panel (one side)
280 Optical transceiver 300 Server 301 Processor 302 Non-volatile memory 303 Volatile memory 304 Communication interface (communication I/F)
305 Server program 306 Communication device database (communication device DB)
400 Maintenance management terminal 401 Processor 402 Non-volatile memory 403 Volatile memory 404 Input device (second input device)
405 Display (second display)
406 Graphics processing unit (second display control unit)
407 Communication interface (communication I/F)
410 Maintenance management APP
411 Display control unit 412 Communication unit 500 Network 600 Trained model 700, 700A, 700B, 700C Virtual panel (first virtual surface)
701, 702, 702a, 702b, 702c, 703 Part models 710, 710A, 710B, 710C Virtual panels (second virtual surfaces)
900 AR image 910 Camera image 920 Selection condition input section 930 Work instruction section 940 Port information display section 940a Button 950A, 950B, 950C, 950D Tutorial section

Claims (16)

A maintenance support system that supports maintenance of a communication device having a plurality of communication ports on one surface of a housing,
a maintenance support terminal including a camera and a first display;
a recognition unit that recognizes the surface including the functional unit of the communication device in the image of the surface captured by the camera using a trained model of machine learning;
a position measurement unit that measures a position in the real space of the one surface recognized by the trained model;
a virtualization unit that determines a representative position of the one surface based on the position of the one surface measured by the position measurement unit, and places a first virtual surface that is a virtual three-dimensional model of the one surface at a position corresponding to the representative position in a virtual space that corresponds to the real space;
a first display control unit that causes the first virtual surface arranged in the virtual space by the virtualization unit and an image of the surface captured by the camera to be superimposed on the first display;
Equipped with
Maintenance support system. The recognition unit recognizes a partial element of the one surface using the trained model,
The position measurement unit measures a real-space position of the partial element recognized by the trained model,
the virtualization unit determines the representative position based on the position of the partial element measured by the position measurement unit.
The maintenance support system according to claim 1 . The trained model is generated by machine learning using the image of the entire surface in which the partial elements are identified as input data.
The maintenance support system according to claim 2 . the recognition unit recognizes a plurality of partial elements of the surface in the image of the surface captured by the camera using a trained model of machine learning;
the position measurement unit measures the position of each of the plurality of partial elements in the real space;
the virtualization unit arranges the first virtual surface in the virtual space based on positions of the plurality of partial elements in the real space.
The maintenance support system according to claim 2 or 3. the virtualization unit determines whether the plurality of partial elements recognized by the trained model are the partial elements of the same communication device based on the positions of each of the plurality of partial elements in the real space, and when it determines that the plurality of partial elements are not the partial elements of the same communication device, does not place the first virtual surface in the virtual space.
The maintenance support system according to claim 4 . the first virtual surface includes a plurality of part models that are three-dimensional models of the plurality of communication ports;
the first display control unit is capable of switching the color of each of the plurality of part models;
The maintenance support system according to any one of claims 1 to 5. a normal color is set to the component model when the corresponding communication port is normal, and an abnormal color is set to the component model when the corresponding communication port is abnormal;
the maintenance support system further includes a notification receiving unit that receives a notification of an abnormal communication port in the communication device;
the first display control unit, when the notification receiving unit receives a notification of an abnormal communication port, switches a color of the component model corresponding to the communication port from the normal color to the abnormal color;
The maintenance support system according to claim 6. the notification receiving unit is capable of receiving a notification of a state of the communication port in the communication device,
The first display control unit
when the notification receiving unit receives a notification of an abnormal state of the communication port, displaying first response information indicating a first-stage response method for the abnormality on the first display;
when the state of the communication port changes as a result of the first-stage countermeasure method being executed and the notification receiving unit receives a notification of the state of the communication port after the change, the first countermeasure information displayed on the first display is switched to second countermeasure information indicating a second-stage countermeasure method for the abnormality;
The maintenance support system according to claim 7. a non-selection color, which is a color when the corresponding communication port is not selected, and a selection color, which is a color when the corresponding communication port is selected, are set for the component model;
when the first display control unit receives selection conditions for selecting the communication port, it switches a color of the part model, among the plurality of part models, corresponding to the communication port that matches the selection conditions from the non-selected color to the selected color;
The maintenance support system according to claim 6. the maintenance support terminal further includes a first input device;
when the first input device receives an input specifying the part model on the first virtual plane, the first display control unit causes the first display to display attribute information indicating an attribute of a transceiver attached to a communication port corresponding to the part model, status information indicating a status of the communication port, and communication volume information indicating a communication volume of the communication port.
The maintenance support system according to any one of claims 6 to 9. the maintenance support system further includes a maintenance management terminal that is located at a location different from a location where the communication device is located;
the maintenance management terminal accepts input of designation of the communication port of the communication device and work information indicating work to be performed on the designated communication port, and transmits designation information indicating the designated communication port and the input work information;
a non-designated color, which is a color when the corresponding communication port is not designated, and a designated color, which is a color when the corresponding communication port is designated, are set to the component model;
the maintenance support system further includes a receiving unit that receives the designation information and the work information transmitted from the maintenance management terminal,
the first display control unit switches a color of the part model corresponding to the communication port designated in the designation information received by the receiving unit from the non-designated color to the designated color, and displays the work information received by the receiving unit on the first display.
The maintenance support system according to claim 6. the maintenance management terminal includes a second input device, a second display, and a second display control unit;
the second display control unit causes the second display to display a second virtual surface that is a virtual three-dimensional model of the one surface;
the maintenance management terminal accepts the designation of the communication port by the second input device accepting an input designating the communication port of the second virtual plane;
The maintenance support system according to claim 11. The aspect does not include elements used solely for image recognition.
The maintenance support system according to any one of claims 1 to 5. The recognition unit identifies a model of the communication device using the trained model,
the first virtual surface is a virtual three-dimensional model of the one surface of the identified model of the communication device;
The maintenance support system according to any one of claims 1 to 5. A maintenance support device that supports maintenance of a communication device having a plurality of communication ports on one surface of a housing,
a recognition unit that recognizes the surface including the functional unit of the communication device in an image of the surface captured by a camera using a trained model of machine learning;
a virtualization unit that determines a representative position of the one surface based on the position in real space of the one surface recognized by the trained model, and places a virtual surface that is a virtual three-dimensional model of the one surface at a position corresponding to the representative position in a virtual space corresponding to the real space;
a display control unit that generates an augmented reality image by superimposing the virtual surface arranged in the virtual space by the virtualization unit and the image of the surface captured by the camera;
Equipped with
Maintenance support equipment. A maintenance support method for supporting maintenance of a communication device having a plurality of communication ports on one surface of a housing, comprising:
a step of recognizing the surface including the functional unit of the communication device in an image of the surface captured by a camera using a trained model of machine learning;
determining a representative position of the one surface based on the position in real space of the one surface recognized by the trained model, and placing a virtual surface, which is a virtual three-dimensional model of the one surface, at a position corresponding to the representative position in a virtual space corresponding to the real space;
generating an augmented reality image by superimposing the virtual surface arranged in the virtual space and the image of the surface captured by the camera;
Including,
Maintenance support method.