JP7738240B2 - Information processing system, information processing method, and program - Google Patents

Description

This disclosure relates to an information processing system, an information processing method, and a program. More specifically, this disclosure relates to an information processing system, an information processing method, and a program that perform information processing related to the creation of a small blackboard (electronic small blackboard) to be embedded in photographed images of a construction site.

Patent Document 1 discloses a photography terminal that takes construction photos. The display unit of the photography terminal displays a layout diagram in which the locations where equipment will be installed are indicated with icons. When a user taps an icon on the layout diagram, a startup unit of the photography terminal activates the camera function and takes a photo of the construction target area. An annotation unit of the photography terminal appends the construction information linked to the icon tapped by the user to the image taken with the camera function and sends the image to a server.

Japanese Patent Application Laid-Open No. 2021-141388

In recent years, with the digitalization of construction photographs, it has become common to attach small blackboards (electronic small blackboards) with construction work details written on them to construction photographs. When taking construction photographs using the photography terminal described in Patent Document 1, if a photograph is taken of equipment at an installation location that has not been registered in advance, the user must select an appropriate style of small blackboard and perform input work to write down the work details, which is time-consuming for the user to perform input work.

The purpose of this disclosure is to provide an information processing system, information processing method, and program that can reduce the effort required for user input.

An information processing system according to one aspect of the present disclosure includes an acquisition unit, a voice recognition unit, and a small blackboard selection unit. The acquisition unit acquires voice data of voice uttered by a user. The voice recognition unit performs voice recognition processing on the voice data. The small blackboard selection unit selects a target format corresponding to a target task from a plurality of small blackboard formats corresponding to a plurality of types of tasks. Each of the plurality of small blackboard formats includes a plurality of input items according to the type of task. The voice recognition unit recognizes the voice data to recognize at least a portion of the plurality of input items and the plurality of pieces of input information entered into each of the plurality of input items. The small blackboard selection unit selects the target format from the plurality of small blackboard formats based on the recognition results of the voice recognition unit.

An information processing method according to one aspect of the present disclosure includes an acquisition step, a voice recognition step, and a small blackboard selection step. In the acquisition step, voice data of a voice uttered by a user is acquired. In the voice recognition step, voice recognition processing is performed on the voice data. In the small blackboard selection step, a target format corresponding to a target task is selected from a plurality of small blackboard formats corresponding to a plurality of types of tasks. Each of the plurality of small blackboard formats includes a plurality of input items according to the type of task. In the voice recognition step, the voice data is voice recognized to recognize at least a portion of the plurality of input items and the plurality of pieces of input information entered into each of the plurality of input items. In the small blackboard selection step, the target format is selected from the plurality of small blackboard formats based on the recognition results of the voice recognition step.

One aspect of the program disclosed herein is a program for causing one or more processors of a computer system to execute the information processing method.

This disclosure can reduce the effort required for user input processing.

FIG. 1 is a block diagram of an information processing system according to an embodiment. FIG. 2 is an explanatory diagram showing an example of a screen display of the information processing system. FIG. 3 is an explanatory diagram showing an example of a composite image of the information processing system. FIG. 4 is an explanatory diagram showing a first example of the small blackboard format of the information processing system. FIG. 5 is an explanatory diagram showing a second example of the small blackboard format of the information processing system. FIG. 6 is an explanatory diagram showing a third example of the small blackboard format of the information processing system. FIG. 7 is an explanatory diagram showing an example of a small blackboard created by the information processing system. FIG. 8 is a flowchart showing an example of processing in the information processing system.

(Embodiment)
An information processing system 1, an information processing method, and a program according to an embodiment will be described below with reference to the accompanying drawings. However, the embodiment described below is merely one of various embodiments of the present disclosure. The embodiment described below can be modified in various ways depending on the design, etc., as long as the object of the present disclosure can be achieved.

(overview)
The "target space" referred to in this disclosure is a construction site of a building (e.g., an apartment building, a detached house, an office building, a hospital, a factory, a commercial facility, etc.) before, during, or after construction. However, the "target space" is not limited to this example.

During construction work such as the construction of a new building, remodeling, or renovation, a portion of the target space (construction site) of the building that is the subject of construction is photographed, and the captured images are used to confirm and inspect the condition of the building and the construction status. Information processing system 1 presents information related to this photography work to workers (users), thereby assisting in the confirmation and inspection of the condition of the building and the construction status. In other words, information processing system 1 assists in the confirmation and inspection associated with the construction of a building.

Information processing system 1 has the function of processing photographed images of a portion of the target space of a building and information related to the photographed images to assist in confirmation and inspection during the construction of the building. Specifically, information processing system 1 has the function of creating a small blackboard (electronic small blackboard) to be combined with photographed images of the construction site. Users such as workers and inspectors take photographs of various parts of the target space before, during, or after construction. Small blackboards are created by entering work information related to the construction at the construction site into a small blackboard (electronic small blackboard) format (target format) that corresponds to the construction site being photographed.

In this embodiment, as an example, the building is an apartment building, and a dwelling unit in the apartment building is being renovated. In this case, the target space is the apartment building (construction site), which is the construction site for the renovation. When renovating a dwelling unit, a current situation survey is first conducted to confirm the condition of the dwelling unit before the renovation. Next, demolition work is carried out to remove unnecessary parts and equipment from the current dwelling unit, and the main renovation work is carried out. The main work includes carpentry work, equipment work, piping work, wiring work, electrical work, painting work, interior work, etc. Once the main work is completed, a final inspection is carried out to inspect the dwelling unit after renovation. During each of the current situation survey, demolition work, main work, and final inspection, photography may be carried out to confirm and inspect the condition of the building and the construction status, etc.

As shown in FIG. 1, the information processing system 1 of this embodiment includes an acquisition unit 13, a voice recognition unit 111, and a small blackboard selection unit 112. The acquisition unit 13 acquires voice data of voice uttered by the user. The voice recognition unit 111 performs voice recognition processing on the voice data. The small blackboard selection unit 112 selects a target form corresponding to a target task from a plurality of small blackboard forms A4 (see FIGS. 4 to 6) corresponding to multiple types of tasks, respectively. Each of the plurality of small blackboard forms A4 includes a plurality of input items D11 to D14 corresponding to the type of task. The voice recognition unit 111 recognizes the plurality of input items D11 to D14 and at least a portion of the plurality of pieces of input information entered into each of the plurality of input items D11 to D14 by voice recognizing the voice data. The small blackboard selection unit 112 selects a target form from the plurality of small blackboard forms A4 based on the recognition results of the voice recognition unit 111. In the following explanation, when the three types of small blackboard forms A4 shown in Figures 4 to 6 are individually described, the small blackboard forms A4 shown in Figures 4 to 6 may be referred to as small blackboard forms A41, A42, and A43, respectively.

According to this embodiment, the small blackboard selection unit 112 selects the target format based on the recognition results of the speech recognition unit 111, which recognizes at least a portion of the multiple input items D11 to D14 and the multiple pieces of input information. Therefore, the small blackboard selection unit 112 can select an appropriate small blackboard format A4 as the target format from multiple small blackboard formats A4 based on the recognition results of the speech recognition unit 111, eliminating the need for the user to select the small blackboard format A4 themselves and reducing the user's input processing effort.

Furthermore, functions similar to those of the information processing system 1 can be embodied in an information processing method. The information processing method of this embodiment is executed by the information processing system 1 and includes an acquisition step, a voice recognition step, and a small blackboard selection step. In the acquisition step, voice data of a voice uttered by a user is acquired. In the voice recognition step, voice recognition processing is performed on the voice data. In the small blackboard selection step, a target form corresponding to the target task is selected from a plurality of small blackboard forms A4 corresponding to multiple types of tasks, respectively. Each of the plurality of small blackboard forms A4 includes a plurality of input items D11 to D14 (see Figures 4 to 6) corresponding to the type of task. In the voice recognition step, the voice data is voice recognized to recognize at least a portion of the plurality of input items D11 to D14 and the plurality of pieces of input information entered into the plurality of input items D11 to D14, respectively. In the small blackboard selection step, a target form is selected from the plurality of small blackboard forms A4 based on the recognition results of the voice recognition step.

The information processing method can also be embodied as a program. The program of this embodiment is a program for causing one or more processors of a computer system to execute the information processing method. The program may be recorded on a non-transitory recording medium readable by the computer system.

(detail)
(1) Overall Configuration First, the overall configuration of the information processing system 1 will be described with reference to FIG.

The information processing system 1 is, for example, a tablet terminal, a smartphone, or a dedicated construction photography terminal used to take construction photos.

The information processing system 1 includes a computer system having one or more processors and memory. At least some of the functions of the information processing system 1 are realized by the processor of the computer system executing a program stored in the memory of the computer system. The program may be stored in memory, provided via a telecommunications line such as the Internet, or provided on a non-transitory recording medium such as a memory card.

As shown in FIG. 1, the information processing system 1 includes a processing unit 11, a memory unit 12, an acquisition unit 13, an operation unit 14, a display unit 15, an image capture unit 16, and a communication unit 17.

(2) Processing Unit The processing unit 11 includes one or more processors of the information processing system 1. The processing unit 11 executes a program to realize a predetermined function.

The processing unit 11 is configured to perform overall control of the information processing system 1. For example, the processing unit 11 controls the display of images by the display unit 15. The processing unit 11 also controls the photographing operation by the photographing unit 16.

In this embodiment, the processing unit 11 has functions such as a voice recognition unit 111, a small blackboard selection unit 112, a small blackboard creation unit 113, and an image synthesis unit 114. Note that the voice recognition unit 111, the small blackboard selection unit 112, the small blackboard creation unit 113, and the image synthesis unit 114 merely indicate functions realized by the processing unit 11, and do not indicate actual configurations. Each function of the processing unit 11 will be explained below.

(2.1) Voice Recognition Unit The voice recognition unit 111 performs voice recognition on the voice data acquired by the acquisition unit 13, for example, using a voice recognition dictionary file stored in the storage unit 12. The voice recognition dictionary file stored in the storage unit 12 includes data on words related to construction work being carried out in the target space, as well as words related to input items and input information on the electronic blackboard to be displayed overlaid on construction photos. The voice recognition dictionary file stored in the storage unit 12 may further include word data on operation commands for causing the information processing system 1 to perform a desired operation.

Note that it is not essential that the storage unit 12 of the information processing system 1 stores a voice dictionary file, and the voice recognition unit 111 may perform voice recognition using a voice dictionary file stored on the cloud.

(2.2) Small Blackboard Selection Unit The small blackboard selection unit 112 selects a small blackboard form A4 corresponding to a target task from a plurality of small blackboard forms A4 corresponding to a plurality of types of tasks based on the recognition result of the voice recognition unit 111.

For example, if the recognition result of the voice recognition unit 111 includes information on one or more input items, the small blackboard selection unit 112 selects the small blackboard form A4 that has one or more input items included in the recognition result as the target form, which is the small blackboard form for the target task.

Furthermore, if the recognition result of the voice recognition unit 111 includes one or more pieces of input information corresponding to one or more input items, the small blackboard selection unit 112 determines the corresponding one or more input items from the one or more pieces of input information obtained by voice recognition.The small blackboard selection unit 112 then selects the small blackboard form A4 including one or more input items as the target form, which is the small blackboard form for the target task.

Furthermore, if the recognition result of the voice recognition unit 111 includes one or more first input items and one or more pieces of second input information corresponding to one or more second input items, the small blackboard selection unit 112 determines one or more corresponding second input items from the one or more pieces of second input information obtained by voice recognition.The small blackboard selection unit 112 then selects the small blackboard form A4, which includes one or more first input items and one or more second input items, as the target form, which is the small blackboard form for the target task.

Note that if the recognition results of the voice recognition unit 111 alone are not sufficient to narrow down the target format to one, the small blackboard selection unit 112 may select two or more candidate formats based on the recognition results of the voice recognition unit 111. In other words, the small blackboard selection unit 112 may perform processing to narrow down multiple small blackboard formats A4 to two or more candidate formats based on the recognition results of the voice recognition unit 111. In this case, the processing unit 11 displays the two or more candidate formats selected by the small blackboard selection unit 112 on the display unit 15 and accepts a selection operation by the user. When the operation unit 14 accepts a selection operation by the user to select a target format from two or more candidate formats, the small blackboard selection unit 112 selects a target format corresponding to the target task based on the selection operation.

In this way, the small blackboard selection unit 112 selects two or more candidate forms as target forms from multiple small blackboard forms A4 based on the recognition results of the voice recognition unit 111. Then, when the operation unit 14 receives a user selection operation to select a target form from two or more candidate forms, the small blackboard selection unit 112 selects a target form corresponding to the target work based on the selection operation. Because the candidate forms are narrowed down by the small blackboard selection unit 112, the user's selection operation can be reduced compared to when the user selects a target form from multiple small blackboard forms.

(2.3) Small Blackboard Creation Unit The small blackboard creation unit 113 creates the small blackboard A5 by inputting multiple pieces of input information into multiple input fields included in the target form selected by the small blackboard selection unit 112. Specifically, the voice recognition unit 111 recognizes the multiple pieces of input information to be input into the multiple input fields by voice recognition of the voice data, and the small blackboard creation unit 113 creates the small blackboard A5 by inputting the multiple pieces of input information into the multiple input fields based on the recognition results of the voice recognition unit 111. In other words, the small blackboard creation unit 113 creates text data of the input information based on the recognition results of the voice recognition unit 111 and displays this text data in the input fields of the corresponding input fields, thereby creating image data for the small blackboard A5.

Figure 7 shows an example of image data of a small blackboard A5 created by the small blackboard creation unit 113. The small blackboard creation unit 113 creates image data of a small blackboard A5 that displays, for example, text data of input information in the second display area AR2 of the small blackboard format A41 shown in Figure 4, and text data indicating the name of the item being photographed (e.g., a distribution board) in the third display area AR3.

(2.4) Image Composition Unit The image composition unit 114 creates a composite image A6 (see FIG. 3 ) with the small blackboard A5 added by combining the captured image A3 captured at the shooting position A2 with the image data of the small blackboard A5 created by the small blackboard creation unit 113. The image composition unit 114 stores the created composite image A6 in the memory unit 12.

The position of the small blackboard A5 added to the captured image A3 can be changed as needed; for example, it is preferable to place the small blackboard A5 in a position that does not overlap with the subject captured at the capture position A2. The composite image A6 shown in Figure 3 is an image in which the image data of the small blackboard A5 is composited onto the captured image A3 of the distribution board, and the image data of the small blackboard A5 is composited in a position that does not overlap with the distribution board.

(3) Storage Unit The storage unit 12 stores information used in the information processing system 1. The storage unit 12 stores, for example, a space diagram A1, a shooting position A2, a captured image A3, and a plurality of small blackboard forms A4.

The space diagram A1 is a diagram representing the target space. The space diagram A1 is, for example, a floor plan of the target space, and if the target space is a dwelling unit, it may also be a floor plan of the dwelling unit. The space diagram A1 may be a two-dimensional drawing, a three-dimensional drawing, or a virtual image representing the target space.

Photographing position A2 is a position where a photograph of a subject in the target space is taken by the photographing unit 16, and is set on the space map A1. Photographing position A2 may be set in advance on the space map A1, or the user may add or delete photographing positions A2. In the example screen shown in Figure 2, multiple photographing positions A2 are set on the space map A1, but the number of photographing positions A2 may be one or more and can be changed as appropriate. The example screen in Figure 2 displays six markers M1 to M6 representing six photographing positions A2, respectively, but the number of markers can be changed as appropriate depending on the number of photographing positions A2. In the following description, the multiple markers M1 to M4 displayed superimposed on the space map A1 may be collectively referred to as marker Mn (n is a natural number).

The memory unit 12 stores the captured images A3 captured by the image capture unit 16 at each of the multiple image capture positions A2. The memory unit 12 stores the captured images A3 in association with information about the image capture positions A2 at which the captured images A3 were captured.

The memory unit 12 pre-stores multiple small blackboard forms A4. There are many types of construction work to be carried out in the target space, including carpentry work, equipment work, piping work, wiring work, electrical work, painting work, and interior construction work. Each type of construction work involves many different tasks, such as checking the current situation, checking the removal status, checking the delivery of materials, recording the construction status, and recording the post-construction status. Because the input items to be written on the small blackboard A5, which is superimposed on the photographed image A3 of the construction site, differ depending on the type of work, the memory unit 12 stores multiple small blackboard forms A4 corresponding to multiple types of work. Each of the multiple small blackboard forms A4 has a rectangular display area modeled after a conventional construction blackboard, and is provided with multiple input items for entering information about the corresponding work.

The multiple input fields here include subject, photo category, work type, location, etc. The name of the work being done on the target space is input into "subject." Information indicating the timing of the photo shoot at shooting location A2 (before removal, before construction, during construction, after construction, etc.) is input into "photo category." Information indicating the type of work being done at shooting location A2 (for example, carpentry work, equipment work, piping work, wiring work, electrical work, painting work, interior work, etc.) is input into "location." The name of shooting location A2 is input into "location."

Figures 4 to 6 show an example of a small blackboard form A4. A first display area AR1 that displays the names of multiple input items is provided in the upper left of each small blackboard form A4, and a second display area AR2 that displays multiple pieces of input information entered for each of the multiple input items is provided in the upper right of each small blackboard form A4. Finally, a third display area AR3 that displays the name of the item to be worked on is provided below each small blackboard form A4.

In the small blackboard form A41 shown in Figure 4, four input items D11 to D14 are displayed in the first display area AR1: subject, photo category, type of work, and location. In the small blackboard form A42 shown in Figure 5, three input items D11 to D13 are displayed in the first display area AR1: subject, photo category, and type of work. In the small blackboard form A43 shown in Figure 6, two input items D11 and D13 are displayed in the first display area AR1: subject and type of work.

The memory unit 12 also stores image data of the small blackboard (electronic small blackboard) A5 created by the small blackboard creation unit 113, in association with the shooting position A2. The memory unit 12 also stores a composite image A6 created by combining the captured image A3 captured by the imaging unit 16 with the image data of the small blackboard A5. The memory unit 12 also stores the composite image A6 in association with the shooting position A2 of the captured image A3.

It is preferable that the memory unit 12 have rewritable non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read Only Memory) or flash memory.

(4) Acquisition Unit The acquisition unit 13 includes a microphone that converts the voice uttered by the user into electrical voice data, and outputs the voice data to the voice recognition unit 111.

When capturing image A3 at shooting position A2, if the user vocalizes the input items or input information for the small blackboard A5 to be superimposed on captured image A3, the acquisition unit 13 converts the user's voice into voice data and outputs it to the voice recognition unit 111.

In addition, even if the user issues an operation command to operate the information processing system 1 by voice, the acquisition unit 13 converts the operation command into voice data and outputs it to the voice recognition unit 111. When the voice recognition unit 111 recognizes the voice data of the operation command, the processing unit 11 operates according to the content of the operation command, so that the information processing system 1 can be operated by voice.

Note that it is not essential that the acquisition unit 13 include a microphone; the acquisition unit 13 may acquire audio data input from an external microphone.

(5) Operation Unit The operation unit 14 is a user interface device that accepts user operations. For example, the operation unit 14 includes at least one of a touch panel, a button, and a key switch that accepts user operations (manual operations).

When the operation unit 14 receives a user operation, it outputs an operation signal corresponding to the user operation to the processing unit 11. The processing unit 11 performs processing corresponding to the operation signal. For example, the processing unit 11 controls the display unit 15 or the imaging unit 16 in response to the operation signal.

If the operation unit 14 has a touch panel that accepts manual operations, user operations on the touch panel include, for example, tapping, double tapping, long tapping, dragging, swiping, flicking, pinching in, and pinching out.

(6) Photography Unit The photography unit 16 includes a two-dimensional image sensor, such as a CCD (Charge Coupled Devices) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. The photography unit 16 functions as a camera that captures color or monochrome images (still images) and generates data for the captured image A3. The photography unit 16 is used to capture images of the construction site of the target construction work. The photography unit 16 stores the data for the captured image A3 in the memory unit 12.

For example, if a touch panel display implements the functions of both the operation unit 14 and the display unit 15, when the user taps a marker Mn displayed on the screen of the display unit 15, the image capture unit 16 enters a standby state for capturing an image. Furthermore, when the user performs a predetermined operation on the operation unit 14, the image capture unit 16 captures an image. The storage unit 12 stores the data of the captured image A3 in association with information about the capture position A2 corresponding to the marker Mn tapped by the user.

(7) Communication Unit The communication unit 17 includes a communication interface device. The communication unit 17 is capable of communicating with an external system via the communication interface device. In this disclosure, "capable of communication" means being able to send and receive signals directly or indirectly via a network or a repeater, etc., using an appropriate communication method such as wired communication or wireless communication. Note that wired communication refers to wired communication via, for example, a twisted pair cable, a dedicated communication line, or a LAN (Local Area Network) cable. Wireless communication refers to wireless communication that complies with standards such as Wi-Fi (registered trademark) or low-power wireless (specified low-power wireless) that does not require a license.

The communication unit 17 exchanges data with external systems. For example, the communication unit 17 acquires data for the space diagram A1 from an external system. For example, the communication unit 17 acquires data for the space diagram A1, which is a diagram of the target space, from an external CAD system or external server, and stores the data for the space diagram A1 in the storage unit 12. Note that the space diagram A1 in this embodiment is a floor plan of a dwelling unit, as shown in Figure 2.

(8) Display Unit The display unit 15 is a display device using an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diode), or the like, and has a screen for displaying images. The display unit 15 is controlled by the processing unit 11.

The touch panel display may also function as both the operation unit 14 and the display unit 15.

As shown in FIG. 2, the display unit 15 displays a space diagram A1, which is a diagram of the target space. Specifically, the display unit 15 displays the space diagram A1 on which multiple markers Mn (M1 to M6) are superimposed. The multiple markers Mn correspond one-to-one to the multiple shooting positions A2. Each of the multiple markers Mn is displayed at the same position as the corresponding shooting position A2, and represents the location of the corresponding shooting position A2. Note that the display position of a marker Mn is, for example, the same position as the corresponding shooting position A2, but is not limited to a position that exactly matches the shooting position A2, and may be near the shooting position A2.

Note that sequence information indicating the shooting order of the multiple shooting positions A2 is superimposed on each of the multiple markers Mn (M1 to M6). In this embodiment, the sequence information of the multiple shooting positions A2 is represented by consecutive numbers such as 1, 2, 3, ....

For example, at the position corresponding to shooting position A2 where the first photograph is taken, a marker M1 and the number [1] as sequence information are displayed superimposed on the spatial diagram A1. Similarly, at the position corresponding to shooting position A2 where the nth photograph is taken (n is a natural number), a marker Mn and the number [n] as sequence information are displayed superimposed on the spatial diagram A1.

The display unit 15 is also used to display image data for the small blackboard A5. The display unit 15 is also used to display the composite image A6.

(9) Processing Flow Next, the processing flow of the information processing system 1 in the work of taking photos for construction confirmation will be described with reference to Fig. 8. Note that the flowchart shown in Fig. 8 is merely an example, and the order of each step may be changed as appropriate, and steps may be added or omitted as appropriate.

Information such as a spatial diagram A1, multiple shooting positions A2, and multiple small blackboard forms A4 is input into the information processing system 1 (step ST1), and the memory unit 12 stores the information input in step ST1.

When the necessary information is input in step ST1, the information processing system 1 starts processing to display the spatial diagram A1 on the display unit 15. That is, the processing unit 11 performs display processing to display the spatial diagram A1 on the display unit 15, with markers Mn superimposed on multiple shooting positions A2 (step ST2).

When the display unit 15 is displaying the spatial diagram A1 and the user taps the marker Mn or any position on the spatial diagram A1 as a new shooting position A2 (step ST3: Yes), the operation unit 14 accepts the operation information, and the processing unit 11 activates the shooting unit 16. After that, when the operation unit 14 accepts a predetermined operation, the shooting unit 16 captures the subject (step ST4), and the captured image A3 captured by the shooting unit 16 is stored in the memory unit 12.

Once the photographing unit 16 has performed the photographing process, the processing unit 11 begins the process of creating a small blackboard A5 to be superimposed on the photographed image A3.

The information processing system 1 waits for voice input of input items to be displayed on the small blackboard A5 or input information related to the input items (step ST5). When the user vocalizes the input items or input information (step ST5: Yes), the acquisition unit 13 acquires the voice data, and the voice recognition unit 111 performs voice recognition processing (step ST6).

At this time, the small blackboard selection unit 112 performs a selection process to select the small blackboard form A4 containing the input items entered by voice or the input items corresponding to the input information as the target form corresponding to the target task based on the recognition results of the voice recognition unit 111 (step ST7).

If the target format is narrowed down to one in the selection process of step ST7 (step ST8: Yes), the processing unit 11 displays an image of the target format on the display unit 15 (step ST12).

If the selection process of step ST7 fails to narrow down the target formats to one (step ST8: No), the small blackboard selection unit 112 selects two or more small blackboard formats A4 that contain input items input by voice or input items corresponding to the input information as candidate formats that are candidates for the target format. At this time, the processing unit 11 displays images of the two or more candidate formats on the display unit 15 (step ST9) and waits for a selection operation by the user (step ST10). When the user performs a selection operation to select a target format from the two or more candidate formats displayed on the display unit 15, for example, using the operation unit 14 (step ST10: Yes), the operation unit 14 accepts the user's selection operation. Then, the small blackboard selection unit 112 determines the candidate format selected by the user as the target format based on the operation information accepted by the operation unit 14 (step ST11), and the processing unit 11 displays an image of the target format on the display unit 15 (step ST12).

When an image of the target form is displayed on the display unit 15, the user vocally inputs input information for any of the multiple input fields in the target form for which no input information has been entered. The information processing system 1 waits for the vocal input of input information (step ST13), and when the user vocally inputs the input information, the voice recognition unit 111 performs voice recognition processing (step ST14).

When input information is entered by voice for all of the input items provided in the target form, the small blackboard creation unit 113 creates image data for a small blackboard A5 that displays the text data for the input information in multiple input fields corresponding to the multiple input items (step ST15). The small blackboard creation unit 113 stores the created image data for the small blackboard A5 in the memory unit 12.

The image synthesis unit 114 then creates a composite image A6 by superimposing the image data of the small blackboard A5 onto the captured image A3 captured by the imaging unit 16 (step ST16), and stores the created composite image A6 in the memory unit 12. The processing unit 11 then displays the composite image A6 on the display unit 15 (step ST17), allowing the user to check the created composite image A6.

Here, when the operation unit 14 accepts an end operation from the user (step ST18: Yes), the processing unit 11 ends the process of displaying the spatial diagram A1 on the display unit 15.

On the other hand, when the operation unit 14 accepts a predetermined operation by the user (step ST19: Yes), the processing unit 11 returns to step ST2 and causes the display unit 15 to display the spatial diagram A1. While the display unit 15 is displaying the spatial diagram A1, if the user taps the marker Mn or taps any position on the spatial diagram A1 as a new shooting position A2 (step ST3: Yes), the information processing system 1 repeatedly executes the processing from step ST4 onwards described above.

In this way, when the user speaks an input item in step ST5, the small blackboard selection unit 112 selects the target format based on the recognition result of the voice recognition unit 111. Then, in step ST13, when the user speaks input information related to the input item, the small blackboard creation unit 113 identifies the input information for the input item based on the recognition result of the voice recognition unit 111, and creates a small blackboard A5 that displays text data or symbols indicating the input information in the input field of the target format.

In addition, in step ST5, if the user speaks input information corresponding to an input item, the small blackboard selection unit 112 identifies the input item corresponding to the input information based on the recognition result of the voice recognition unit 111, and selects a target form that includes the input item. The small blackboard creation unit 113 then creates a small blackboard A5 that displays text data or graphic symbols indicating the input information spoken by the user in step ST5 in the input field of the target form. In this case, the user only needs to speak the input information corresponding to the input item, further reducing the user's input work.

(Modification of the embodiment)
Modifications of the embodiment are listed below. The following modifications may be implemented in appropriate combination.

The shape of the marker Mn is not limited to a pin shape and may be any shape.

The sequence information displayed superimposed on the markers Mn is not limited to numbers, but may be, for example, letters such as the alphabet, or figures. The color of the markers Mn may also indicate the order in which the photographs are taken. In other words, the color of the markers Mn may indicate the sequence information.

Furthermore, the display unit 15 may display a marker Mn representing a shooting position A2 where image capture has not yet been completed and a marker Mn representing a shooting position A2 where image capture has been completed in different display modes (e.g., color, shape, etc.).

The entity that executes the information processing system 1 or information processing method in this disclosure includes a computer system. The computer system is primarily composed of a processor and memory as hardware. At least a portion of the functions of the entity that executes the information processing system 1 or information processing method in this disclosure are realized by the processor executing a program recorded in the computer system's memory. The program may be pre-recorded in the computer system's memory, provided via a telecommunications line, or provided recorded on a non-transitory recording medium readable by the computer system, such as a memory card, optical disc, or hard disk drive. The processor of the computer system is composed of one or more electronic circuits, including a semiconductor integrated circuit (IC) or a large-scale integrated circuit (LSI). The integrated circuits, such as ICs and LSIs, are referred to by different names depending on the degree of integration, and include integrated circuits called system LSIs, very large-scale integration (VLSI), or ultra-large-scale integration (ULSI). Furthermore, FPGAs (Field-Programmable Gate Arrays), which are programmed after the LSI is manufactured, or logic devices that allow the internal connections of the LSI to be reconfigured or the circuit partitions within the LSI to be reconfigured, can also be used as processors. Multiple electronic circuits may be integrated into a single chip or distributed across multiple chips. Multiple chips may be integrated into a single device or distributed across multiple devices. The computer system referred to here includes a microcontroller having one or more processors and one or more memories. Therefore, a microcontroller may also be composed of one or more electronic circuits, including semiconductor integrated circuits or large-scale integrated circuits.

Furthermore, it is not essential for the information processing system 1 that multiple functions are integrated into a single device, and multiple components of the information processing system 1 may be distributed across multiple devices. For example, a device including the imaging unit 16 may be provided separately from a device including the display unit 15.

At least some of the functions of the information processing system 1, for example, the image processing functions of the processing unit 11, may be realized by a server or the cloud (cloud computing), etc.

(summary)
As described above, the information processing system (1) of the first aspect includes an acquisition unit (13), a voice recognition unit (111), and a small blackboard selection unit (112). The acquisition unit (13) acquires voice data of a voice uttered by a user. The voice recognition unit (111) performs voice recognition processing on the voice data. The small blackboard selection unit (112) selects a target form corresponding to a target task from a plurality of small blackboard forms (A4) corresponding to a plurality of types of tasks, respectively. Each of the plurality of small blackboard forms (A4) includes a plurality of input items (D11 to D14) corresponding to the type of task. The voice recognition unit (111) recognizes the plurality of input items (D11 to D14) and at least a portion of the plurality of pieces of input information entered into each of the plurality of input items (D11 to D14) by voice recognition of the voice data. The small blackboard selection unit (112) selects a target form from the plurality of small blackboard forms (A4) based on the recognition result of the voice recognition unit (111).

This reduces the amount of input work required by the user.

The information processing system (1) of the second aspect is the same as that of the first aspect, but further includes an operation unit (14) that accepts user operations. The small blackboard selection unit (112) selects two or more candidate formats as target formats from a plurality of small blackboard formats (A4) based on the recognition results of the voice recognition unit (111). When the operation unit (14) accepts a user's selection operation to select a target format from two or more candidate formats, the small blackboard selection unit (112) selects a target format corresponding to the target task based on the selection operation.

In this embodiment, the user selects the target style from two or more candidate styles narrowed down by the small blackboard selection unit (112), thereby reducing the effort required for the user's selection operation.

The information processing system (1) of the third aspect, in the first or second aspect, further includes a small blackboard creation unit (113) that creates a small blackboard (A5) by inputting multiple pieces of input information into multiple input items (D11 to D14) included in the target form.

This reduces the amount of input work required by the user.

In the information processing system (1) of the fourth aspect, in the third aspect, the voice recognition unit (111) recognizes multiple pieces of input information entered into multiple input items (D11 to D14) by voice recognizing voice data. The small blackboard creation unit (113) creates a small blackboard (A5) by entering multiple pieces of input information into the multiple input items (D11 to D14) based on the recognition results of the voice recognition unit (111).

This reduces the amount of input work required by the user.

The information processing system (1) of the fifth aspect is the third or fourth aspect, and further includes a photographing unit (16) and a memory unit (12). The photographing unit (16) photographs the construction site of the target construction work. The memory unit (12) stores a composite image (A6) obtained by combining the photographed image (A3) photographed by the photographing unit (16) with a small blackboard (A5).

This aspect reduces the amount of input work required by the user when generating a composite image (A6) that combines a photographed image (A3) of the construction site of the target construction project with a small blackboard (A5).

The information processing system (1) of the sixth aspect is the fifth aspect, further comprising a display unit (15) that displays the composite image (A6).

According to this embodiment, the user can check the composite image (A6) displayed on the display unit (15).

The information processing method of the seventh aspect includes an acquisition step, a voice recognition step, and a small blackboard selection step. In the acquisition step, voice data of voice uttered by the user is acquired. In the voice recognition step, voice recognition processing is performed on the voice data. In the small blackboard selection step, a target form corresponding to the target task is selected from a plurality of small blackboard forms (A4) corresponding to a plurality of types of tasks, respectively. Each of the plurality of small blackboard forms (A4) includes a plurality of input items (D11 to D14) corresponding to the type of task. In the voice recognition step, the voice data is voice recognized to recognize the plurality of input items (D11 to D14) and at least a portion of the plurality of pieces of input information entered into each of the plurality of input items (D11 to D14). In the small blackboard selection step, a target form is selected from the plurality of small blackboard forms (A4) based on the recognition results in the voice recognition step.

This reduces the amount of input work required by the user.

The program of the eighth aspect is a program for causing one or more processors of a computer system to execute the information processing method of the seventh aspect.

This reduces the amount of input work required by the user.

Not limited to the above aspects, various configurations (including modified examples) of the information processing system (1) according to the embodiment can be embodied as an information processing method performed by the information processing system (1), a (computer) program, or a non-transitory recording medium on which a program is recorded, etc.

The configurations according to the second to sixth aspects are not essential for the information processing system (1) and may be omitted as appropriate.

1 Information processing system 12 Storage unit 13 Acquisition unit 14 Operation unit 15 Display unit 16 Photography unit 111 Voice recognition unit 112 Small blackboard selection unit 113 Small blackboard creation unit A3 Photographed image A4 Small blackboard format A5 Small blackboard A6 Composite image D11 to D14 Input items

Claims (8)

an acquisition unit that acquires voice data of a voice uttered by a user;
a voice recognition unit that performs voice recognition processing on the voice data;
a small blackboard selection unit that selects a target format corresponding to a target task from a plurality of small blackboard formats that respectively correspond to a plurality of types of tasks;
each of the plurality of small blackboard forms includes a plurality of input items according to the type of work;
the voice recognition unit recognizes the plurality of input items and at least a part of the plurality of pieces of input information input into the plurality of input items by performing voice recognition on the voice data;
the small blackboard selection unit selects the target style from the plurality of small blackboard styles based on the recognition result of the voice recognition unit.
Information processing system. further comprising an operation unit that accepts an operation by the user;
The small blackboard selection unit
Selecting two or more candidate styles as candidates for the target style from the plurality of small blackboard styles based on the recognition result of the voice recognition unit;
When the operation unit receives a selection operation by the user to select the target form from the two or more candidate forms, the operation unit selects the target form corresponding to the target task based on the selection operation.
The information processing system according to claim 1 . a small blackboard creation unit that creates a small blackboard by inputting the plurality of pieces of input information into the plurality of input items included in the target form,
The information processing system according to claim 1 . the voice recognition unit recognizes the plurality of pieces of input information input into the plurality of input items by voice recognition of the voice data;
the small blackboard creation unit creates the small blackboard by inputting the plurality of pieces of input information into the plurality of input items based on the recognition result of the voice recognition unit.
The information processing system according to claim 3 . An imaging unit that photographs the construction site of the target work ;
and a storage unit that stores a composite image obtained by combining the image captured by the image capturing unit with the small blackboard.
The information processing system according to claim 3 . Further comprising a display unit that displays the composite image.
The information processing system according to claim 5 . an acquisition step of acquiring voice data of a voice uttered by a user;
a voice recognition step of performing voice recognition processing on the voice data;
a small blackboard selection step of selecting a target form corresponding to a target task from a plurality of small blackboard forms respectively corresponding to a plurality of types of tasks,
each of the plurality of small blackboard forms includes a plurality of input items according to the type of work;
In the voice recognition step, the voice data is voice-recognized to recognize at least a part of the plurality of input items and the plurality of pieces of input information input into the plurality of input items, respectively;
In the small blackboard selection step, the target style is selected from the plurality of small blackboard styles based on the recognition result in the voice recognition step.
Information processing methods. 8. The information processing method according to claim 7, wherein the information processing method is executed by one or more processors of a computer system,
program.