JP7614926B2 - Image forming system, control method, and program - Google Patents

Description

The present invention relates to an image forming system, a control method, and a program.

Recently, information processing devices that can be linked to voice control devices such as smart speakers have become widespread. Patent Document 1 discloses a technology that converts speech to a voice control device into instructions and commands for an image forming device and operates the image forming device.

JP 2020-107130 A

However, in Japanese Patent Laid-Open No. 2006-133999, it is not assumed that a plurality of voice control devices are linked to one image forming device, and in this respect there is room for improvement.
In an environment where multiple voice control devices exist, when changing settings of an image forming device using a voice control device, operations may conflict. For example, user A starts an interactive mode with a first smart speaker to execute a job. At almost the same time, user B starts an interactive mode with a second smart speaker to execute a job. If user B then changes the settings and executes the job before user A, user A is not notified of the setting change and executes the job thinking that the settings are the original, resulting in printing that is different from what was expected.

The present invention aims to improve the usability when operating an image forming device using a voice control device.

The information processing system of the present invention is an image forming system including a plurality of voice control devices that accept user speech, a server that acquires instruction information based on speech information acquired from the plurality of voice control devices, and an image forming device that executes processing based on the instruction information acquired from the server, and is characterized in that the system includes a transmission means in one of the plurality of voice control devices that accepts an instruction to start a voice operation from a user and transmits startup information to the server, a request means in the server that receives the startup information and requests setting information from the image forming device, a storage means in the server that links and stores the setting information of the image forming device and the information of the one voice control device, and a generation means in the server that generates the instruction information using at least the stored setting information when instructing the execution of a job based on the speech information acquired from the one voice control device.

The present invention can improve usability when operating an image forming device using a voice control device.

1 is a diagram illustrating an example of the overall configuration of an image forming system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a hardware configuration of an MFP. FIG. 2 is a diagram illustrating an example of a hardware configuration of a smart speaker. FIG. 2 is a diagram illustrating an example of a hardware configuration of a cloud server. FIG. 2 is a diagram illustrating an example of a functional configuration of an MFP. FIG. 2 is a diagram illustrating an example of a functional configuration of a smart speaker. FIG. 2 is a diagram illustrating an example of a functional configuration of a cloud server. FIG. 2 is a sequence diagram showing an overall process executed by the image forming system. 10 is a flowchart showing details of a process executed by the server. FIG. 2 is a sequence diagram showing an overall process executed by the image forming system. FIG. 2 is a sequence diagram showing an overall process executed by the image forming system.

Below, the form for implementing the present invention will be explained with reference to the drawings.

<System Configuration>
Fig. 1 is a diagram showing an example of the system configuration of the image forming system of the present embodiment. As shown in Fig. 1, the image forming system 100 is composed of an MFP 101, a cloud server 103, and a plurality of smart speakers 110 and 111. The MFP 101 and the smart speakers 110 and 111 are connected to a network 104 and can communicate with each other. The cloud server 103 is also connected to the network 104 via a gateway 105 and can communicate with the MFP 101 and the smart speakers 110 and 111 on the network 104. The image forming system 100 is composed of two smart speakers 110 and 111, but may be composed of three or more devices.

The smart speakers 110 and 111 are electronic devices equipped with a voice control function, and are examples of voice control devices. The smart speakers 110 and 111 accept the user's speech and issue various instructions to the MFP 101 via the cloud server 103. In this embodiment, an independent electronic device such as the smart speakers 110 and 111 is used as the voice control device, but a terminal device such as a PC (personal computer) or a smartphone equipped with a voice control function may also be used.

The cloud server 103 is a cloud server that provides services of a voice recognition function and an operation control function of the MFP 101. The cloud server 103 performs voice recognition on speech information acquired from the smart speakers 110 and 111, and instructs the MFP 101 to execute processing based on the voice recognition result. In this way, the operation of the MFP 101 is controlled.
The MFP 101 is a multifunction printer (MFP) having multiple functions such as copying, scanning, printing, and faxing, and is an example of an image forming apparatus. Note that the image forming apparatus may be an apparatus having a single function such as a printer or scanner. The MFP 101 executes processing based on instruction information acquired from the cloud server 103.

The network 104 includes a wired or wireless LAN, a WAN, the Internet, etc. The network 104 connects the smart speakers 110 and 111, the MFP 101, and the gateway 105 to each other.
The gateway 105 is, for example, a wireless LAN router that operates according to a wireless communication method conforming to the IEEE 802.11 standard series. The gateway 105 may also be a wired LAN router that operates according to a wired communication method conforming to Ethernet standards such as 10BASE-T, 100BASE, and 1000BASE-T, instead of a wireless LAN router.

In this embodiment, the smart speakers 110 and 111 and the MFP 101 are configured to be independent of each other, but this is not limited to this configuration. For example, some of the devices and functions constituting the smart speakers 110 and 111 may be incorporated into the MFP 101 and configured as an integrated unit. In addition, in this embodiment, the smart speakers 110 and 111 and the cloud server 103 are configured to be independent of each other, but this is not limited to this configuration. For example, some of the devices and functions constituting the cloud server 103 may be incorporated into the smart speakers 110 and 111 and configured as an integrated unit. Furthermore, the cloud server 103 and the MFP 101 are configured to be independent of each other, but this is not limited to this configuration. For example, some of the devices and functions constituting the cloud server 103 may be incorporated into the MFP 101 and configured as an integrated unit.

<MFP Hardware Configuration>
2 is a block diagram showing the hardware configuration of the MFP 101 according to this embodiment. The MFP 101 is mainly composed of a control unit 200, an operation panel 209, a print engine 211, a scanner 213, and a card reader 214. The control unit 200 includes a CPU 202, a RAM 203, a ROM 204, a storage 205, a network I/F 206, a display controller 207, and an operation I/F 208. The control unit 200 also includes a print controller 210 and a scan controller 212. These blocks are connected to each other via a system bus 201.

The CPU (Central Processing Unit) 202 controls the overall operation of the MFP 101. The CPU 202 reads out programs stored in the ROM 204 etc., loads them into the RAM 203, and executes them to realize the processes shown in the sequence diagrams described below and the various functions shown in FIG. 5 described below. The RAM 203 is the main storage memory of the CPU 202, and is used as a temporary storage area for loading the programs. The ROM 204 stores programs that can be executed by the CPU 202. The storage 205 stores print data, image data, and various setting information required when executing a job.

In this embodiment, one CPU 202 uses one memory (RAM 203) to execute each process shown in the sequence diagram described below, but other configurations are also possible. For example, multiple CPUs, RAMs, ROMs, and storages can work together to execute the processes shown in the sequence diagrams described below. Also, some of the processes may be executed using hardware circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

The network I/F 206 is an interface that allows the MFP 101 to communicate with external devices via the network 104. The MFP 101 receives instruction information from the cloud server 103 via the network I/F 206 and executes processing based on the instruction information. If the instruction information is a job execution command that instructs the execution of a job, the MFP 101 executes the job. A job is an execution unit of a series of image formation processes realized using the print engine 211 or the scanner 213, and includes a print job, a copy job, and a scan job. If the instruction information is a setting change command that instructs a change to the settings of the MFP 101 main body, the MFP 101 changes the setting information stored in the storage 205 to the instructed content.

The operation I/F 208 connects an operation panel 209 and a card reader 214 to the CPU 202. The operation panel 209 combines the functions of an input device and a display device, and is configured, for example, by a touch panel display. The operation panel 209 provides the operation contents input by the user to the CPU 202 via the operation I/F 208. The display controller 207 controls the display contents of the operation panel 209 under the control of the CPU 202.

The print controller 210 connects the print engine 211 to the CPU 202. Image data to be printed is transferred to the print engine 211 via the print controller 210. The print engine 211 receives the image data to be printed under the control of the CPU 202, and forms an image based on the received image data on a sheet. The printing method of the print engine 211 may be an electrophotographic method or an inkjet method. In the case of the electrophotographic method, an electrostatic latent image is formed on a photoconductor, and then developed with toner, the toner image is transferred to a sheet, and the transferred toner image is fixed to form an image. On the other hand, in the case of the inkjet method, ink is ejected to form an image on a sheet.

The scan controller 212 connects the scanner 213 to the CPU 202. The scanner 213 reads an image on a sheet and generates image data. The image data generated by the scanner 213 is stored in the storage 205. The print engine 211 forms an image on a sheet based on the image data generated by the scanner 213 under the control of the CPU 202. The scanner 213 has a document feeder (not shown) and is capable of reading sheets placed on the document feeder while transporting them one by one.
The card reader 214 performs authentication of the user. If the user is successfully authenticated by the MFP 101, the user can operate the screen displayed on the operation panel 209 to instruct the execution of a print job, a copy job, a scan job, and the like.

<Smart speaker hardware configuration>
3 is a block diagram showing the hardware configuration of the smart speaker 110, 111 according to the present embodiment. The smart speaker 110, 111 is mainly composed of a control unit 300, a microphone 308, a speaker 310, and an LED 312. The control unit 300 includes a CPU 302, a RAM 303, a ROM 304, a storage 305, a network I/F 306, a microphone I/F 307, an audio controller 309, and a display controller 311. These blocks are connected to each other via a system bus 301. In the following, since the smart speaker 110 and the smart speaker 111 have similar hardware configurations, the smart speaker 110 will be described and the description of the smart speaker 111 will be omitted.

The CPU 302 controls the overall operation of the smart speaker 110. The CPU 302 reads out programs stored in the ROM 304 etc., deploys them in the RAM 303, and executes them to realize the processes shown in the sequence diagrams described below and the various functions shown in FIG. 6 described below. The RAM 303 is the main memory of the CPU 302, and is used as a temporary storage area for deploying programs. The ROM 304 stores programs such as a voice control program for realizing a voice control function. The storage 305 stores authentication information required for the authentication procedure with the cloud server 103, setting information for the smart speaker 110 main unit, etc.

The network I/F 306 is an interface that allows the smart speaker 110 to communicate with external devices via the network 104. The smart speaker 110 transmits speech information regarding the user's speech received by the microphone 308 to the cloud server 103 via the network I/F 306. The smart speaker 110 also receives voice synthesis data from the cloud server 103 via the network I/F 306.

The microphone I/F 307 connects the microphone 308 to the CPU 302. The microphone 308 is a voice input device that converts the voice uttered by the user around the smart speaker 110 into a voice signal. The microphone 308 is not limited to being built into the smart speaker 110, and may be configured to be linked by wired communication or wireless communication. Furthermore, the microphone 308 may be used in a plurality of units arranged at predetermined positions so that the direction of arrival of the voice uttered by the user can be calculated. The CPU 302 uses the voice signal acquired by the microphone 308 to generate coded voice data to be transmitted to the cloud server 103, and temporarily stores the generated voice data in the RAM 303. When the user finishes speaking, the CPU 302 acquires the voice data stored in the RAM 303 as speech information.

The audio controller 309 connects the speaker 310 to the CPU 302. The speaker 310 is an audio output device. The speaker 310 is not limited to being built into the smart speaker 110, but may be linked via wired or wireless communication. Under the control of the CPU 302, the audio controller 309 outputs from the speaker 310 a response sound and a voice based on voice synthesis data received from the cloud server 103.

The display controller 311 connects the LED 312 to the CPU 302. The display controller 311 controls the lighting of the LED 312 under the control of the CPU 302. For example, the display controller 311 lights up the LED 312 when the user's speech is being properly recognized. Note that a display device capable of displaying characters and figures may be used instead of the LED 312.

<Cloud server hardware configuration>
4 is a block diagram showing the hardware configuration of the cloud server 103 according to this embodiment. The cloud server 103 includes a CPU 402, a RAM 403, a ROM 404, a storage 405, and a network I/F 406. These blocks are connected to each other via a system bus 401.

The CPU 402 controls the overall operation of the cloud server 103. The CPU 402 reads out programs stored in the ROM 404 etc., deploys them in the RAM 403, and executes them to realize the processes shown in the sequence diagrams and flowcharts described below, and the various functions shown in FIG. 7 described below. The RAM 403 is the main memory of the CPU 402, and is used as a temporary storage area for deploying programs. The ROM 404 stores various programs, including a voice recognition program for implementing a voice recognition function. The storage 405 stores authentication information required for the authentication procedure for the MFP 101, as well as data such as content and files.

The network I/F 406 is an interface for communicating with external devices via the network 104. The cloud server 103 receives speech information from the smart speaker 110 via the network I/F 406, and transmits instruction information acquired based on the voice recognition results of the speech information to the MFP 101. The cloud server 103 also receives various notifications from the MFP 101 via the network I/F 406, and transmits voice synthesis data generated based on the notifications to the smart speakers 110 and 111.

In this embodiment, a single CPU 402 uses a single memory (RAM 403) to execute a program stored in ROM 404 or the like to execute the processes shown in the sequence diagrams and flowcharts described below, but other configurations are also possible. For example, multiple storage devices such as CPUs, RAMs, ROMs, and HDDs can work together to execute the processes described below. In addition, the cloud server 103 is not necessarily realized by a single server, but may be composed of multiple servers according to the functions, or may be composed of multiple devices for distributed processing.

<Functional configuration of MFP>
Fig. 5 is a diagram showing an example of the functional arrangement of the MFP 101 according to this embodiment. The MFP 101 operates as each of the functional units shown in Fig. 5 when the CPU 202 executes a program stored in the ROM 204 or the like.
The data transmission/reception unit 501 transmits and receives data to and from the cloud server 103 via the network I/F 206. Specifically, the data transmission/reception unit 501 receives data related to instructions and requests from the cloud server 103. In addition, the data transmission/reception unit 501 transmits responses to the instructions and requests to the cloud server 103.

The data analysis unit 502 analyzes the data received by the data transmission/reception unit 501. If the analysis shows that the data received from the cloud server 103 is a job execution command, the data analysis unit 502 provides the job execution command to the job control unit 503, which will be described later. If the data is a setting change command, the data analysis unit 502 provides the setting change command to the data management unit 504, which will be described later. If the data is a command requesting setting information for the MFP 101 main body, the data management unit 504 reads out the setting information for the MFP 101 main body from the storage 205. The data transmission/reception unit 501 then transmits the read setting information to the cloud server 103.

The job control unit 503 uses the print controller 210 and the scan controller 212 to give control instructions to the print engine 211 and the scanner 213. Specifically, the job control unit 503 executes printing and scanning based on the job execution command provided by the data analysis unit 502 and the job execution command provided by the operation control unit 506. Note that the job execution command provided by the data analysis unit 502 is a job execution command obtained based on the user's speech information.

The data management unit 504 stores in the storage 205 setting information and the like required for controlling the print engine 211 and the scanner 213. The setting information is data consisting of a combination of each setting item and setting value of the job executed by the job control unit 503, such as black and white/color settings and single-sided/double-sided printing settings. The data management unit 504 changes the setting information stored in the storage 205 based on setting change commands provided by the data analysis unit 502 and setting change commands provided by the operation control unit 506. The setting change commands provided by the data analysis unit 502 are setting change commands obtained based on the user's speech information.

The display control unit 505 controls the display contents of the operation panel 209 via the display controller 207. More specifically, the display control unit 505 displays UI components (buttons, pull-down lists, check boxes, etc.) that can be operated by the user on the operation panel 209. When a job is executed by the job control unit 503, the display control unit 505 updates the display contents of the operation panel 209 based on the execution status of the job. In addition, when the setting information is changed by the data management unit 504, the display contents of the operation panel 209 are updated with the changed contents.

The operation control unit 506 acquires the touched coordinates on the operation panel 209 via the operation I/F 208, and determines the instruction content corresponding to the UI part that is the target of the operation. For example, when the print execution button is touched, the operation control unit 506 recognizes that the execution of a print job has been instructed. The operation control unit 506 then reads the instructed image data and setting information from the storage 205 via the data management unit 504, and generates a job execution command to print the image data using the read image data and setting information. The generated job execution command is provided to the job control unit 503. Also, for example, when the setting change button is touched, the operation control unit 506 generates a setting change command to change the setting information to the instructed content. The generated setting change command is provided to the data management unit 504.

<Functional configuration of smart speaker>
Fig. 6 is a block diagram showing the functional configuration of the smart speakers 110 and 111 according to this embodiment. The smart speakers 110 and 111 operate as the functional units shown in Fig. 6 when the CPU 302 executes a program stored in the ROM 304 or the like. In the following, since the smart speaker 110 and the smart speaker 111 have the same functional configuration, only the smart speaker 110 will be described and a description of the smart speaker 111 will be omitted.

The data transmission/reception unit 601 transmits and receives data to and from the cloud server 103 via the network I/F 306. Specifically, the data transmission/reception unit 501 transmits speech information acquired by a voice acquisition unit 604 (described later) to the cloud server 103, and receives a response thereto from the cloud server 103. Specifically, the data transmission/reception unit 501 sequentially transmits speech information acquired by a voice control unit 603 (described later) to the cloud server 103 during an ongoing dialogue session with the cloud server 103. In addition, the data transmission/reception unit 501 transmits startup information to the cloud server 103. The timing of transmitting the startup information is managed by the voice control unit 603.
The data management unit 602 stores in the storage 305 setting information and the like required for controlling the microphone 308, the speaker 310, and the LED 312. The setting information is, for example, the volume setting of the sound reproduced by the sound reproduction unit 605 described later.

The audio acquisition unit 604 generates audio data encoded in a predetermined format such as MP3 using the audio signal acquired by the microphone 308, and temporarily stores the generated audio data in the RAM 303. The above-described processing by the audio acquisition unit 604 is called audio processing. The timing of starting and ending the audio processing is managed by the audio control unit 603, which will be described later. Note that a general-purpose streaming format may be used to generate the audio data.
The voice reproducing unit 605 reproduces the synthesized voice data included in the response received by the data transmitting/receiving unit 601 on the speaker 310 via the audio controller 309. The above-described processing by the voice reproducing unit 605 is referred to as a reproduction process.
The LED display unit 606 turns on and off the LED 312. The timing of turning on and off the LED display unit 606 is managed by the audio control unit 603, which will be described later.

The operation start detection unit 607 receives a voice operation start instruction from the user and transmits an operation start notification to the voice control unit 603. Specifically, the operation start detection unit 607 detects the utterance of a wake word from the voice signal acquired by the microphone 308. The wake word is a word uttered by the user when activating the voice recognition function. The user can operate the MFP 101 by uttering the wake word and then speaking what they want to do (for example, "I want to copy"). The operation start detection unit 607 may detect a voice operation start instruction by pressing an operation start key (not shown) on the smart speaker 110.

The speech end determination unit 608 determines the timing of the end of speech, and when it is determined that the speech has ended, it transmits a speech end notification to the voice control unit 603. For example, when the user's voice is interrupted for a predetermined time (e.g., 3 seconds), it is determined that the user's speech has ended. Note that the determination of the end of speech is not limited to a method of determining based on the time during which there is no speech, and may be performed based on whether or not the utterance of a predetermined word is detected. For example, when a predetermined word such as "yes," "no," "OK," "cancel," "end," "start," or "start" is received, the speech end determination unit 608 determines that the speech has ended without waiting for a predetermined time. The end of speech processing may also be determined by the cloud server 103. In this case, the end of the operation may be determined based on the meaning and context of the user's speech content.

The audio control unit 603 performs overall control of the audio acquisition unit 604 , audio playback unit 605 , and LED display unit 606 .
First, when the voice control unit 603 receives an operation start notification from the operation start detection unit 607, it starts processing an interactive session with the cloud server 103. First, the voice control unit 603 reads information about its own device from the storage 305. The information about its own device is, for example, an ID number such as a real number that is individually set in the smart speaker 110, and a device name. Hereinafter, the information about its own device will be referred to as device information. Then, the data transmission/reception unit 601 transmits the device information together with the startup information to the cloud server 103.
Next, the voice control unit 603 controls the voice acquisition unit 604 to start voice processing, and controls the LED display unit 606 to turn on the LED 312. Furthermore, upon receiving a speech end notification from the speech end determination unit 608, the voice control unit 603 controls the voice acquisition unit 604 to end voice processing, and controls the LED display unit 606 to turn off the LED 312.

When the user finishes speaking, the voice control unit 603 acquires the voice data temporarily stored in the voice acquisition unit 604 as speech information, and controls the data transmission/reception unit 601 to transmit the acquired speech information together with device information to the cloud server 103. The data transmission/reception unit 601 then waits for a response from the cloud server 103. The response from the cloud server 103 is, for example, a response message consisting of a header section indicating that it is a response, and voice synthesis data. When the data transmission/reception unit 601 receives the response, the voice control unit 603 controls the voice playback unit 605 to start playback processing.

After the playback process is completed, the voice control unit 603 controls the voice acquisition unit 604 to start voice processing. As a result, while the dialogue session with the cloud server 103 continues, the user can continue to talk about what he or she wants to do without uttering the wake word. The cloud server 103 determines whether the dialogue session has ended. The dialogue session ends when the cloud server 103 sends a dialogue session end notification to the smart speaker 110. The state from the end of a dialogue session to the start of the next dialogue session is referred to as a standby state. The voice control unit 603 is always in a standby state until it receives an operation start notification from the operation start detection unit 607.

<Functional configuration of cloud server>
Fig. 7 is a block diagram showing the functional configuration of the cloud server 103 according to this embodiment. The cloud server 103 operates as each of the functional units shown in Fig. 7 by the CPU 402 executing a program stored in the ROM 404 or the like.
The data transmission/reception unit 701 transmits and receives data between the smart speakers 110 and 111 and the cloud server 103 via the network I/F 406. In the following, it is assumed that the cloud server 103 receives startup information from the smart speaker 110 of the smart speakers 110 and 111. When the data transmission/reception unit 701 receives the startup information from the smart speaker 110, it transmits a request command for setting information of the MFP 101 main body to the MFP 101. When the data transmission/reception unit 701 receives speech information from the smart speaker 110, it provides the speech information to a voice recognition unit 703 described later. The data transmission/reception unit 701 transmits an instruction command for the MFP 101 acquired by an instruction command acquisition unit 704 described later to the MFP 101.

The data transmission/reception unit 701 also receives responses to request commands and instruction commands from the MFP 101. The data transmission/reception unit 701 provides the setting information of the MFP 101 main body received as a response to the request command to the data management unit 702. The data transmission/reception unit 701 also provides responses to instruction commands to the voice synthesis unit 705, which will be described later. The data transmission/reception unit 701 transmits the response converted into voice synthesis data by the voice synthesis unit 705 to the smart speaker 110.

The data management unit 702 stores and manages in the storage 405 parameters necessary for the voice recognition processing by the voice recognition unit 703 and parameters necessary for the voice synthesis processing by the voice synthesis unit 705. The data management unit 702 also stores in the RAM 403 and manages a table in which the setting information of the MFP 101 main body provided from the data transmission/reception unit 701 and the device information received from the smart speaker 110 are linked and stored. When the data management unit 702 receives the setting information of the MFP 101 main body from the data transmission/reception unit 701 during an ongoing dialogue session, it updates the setting information stored in the table with the provided setting information.

The voice recognition unit 703 performs voice recognition processing. First, the voice recognition unit 703 converts the user's speech information received by the data transmission/reception unit 701 into text data. For example, the voice recognition unit 703 converts the user's voice data into phonemes using an acoustic model, and then converts the phonemes into actual text data using a language model. The voice recognition unit 703 then performs morpheme analysis on the converted text data. For example, the voice recognition unit 703 derives a morpheme string from a dictionary containing information such as the grammar and parts of speech of the language, and then determines the part of speech of each morpheme. The voice recognition unit 703 provides the results of the above voice recognition processing to the instruction command acquisition unit 704.

The instruction command acquisition unit 704 acquires instruction information for the MFP 101 from the result of the voice recognition unit 703. Specifically, when the instruction command acquisition unit 704 determines that the instruction from the user is an instruction to execute a job, it reads out setting information of the MFP 101 main body that is linked to the device information of the smart speaker 110 that sent the spoken information. The instruction command acquisition unit 704 then generates a job execution command using the read setting information. For example, it generates a command to print image data with the MFP 101. Note that the image data to be printed is stored in an external server (not shown) on the Internet, the storage 405 of the cloud server 103, the storage 205 of the MFP 101, or the like.

When the instruction command acquisition unit 704 determines that the instruction from the user is a setting change instruction, it generates a setting change command using the instruction from the user. For example, it generates a command to change the print settings of the MFP 101 from single-sided printing to double-sided printing.

The voice synthesis unit 705 generates voice synthesis data. For example, the voice synthesis unit 705 converts a response from the MFP 101 received by the data transmission/reception unit 701 into text data, and converts this into voice synthesis data using a voice database. Furthermore, if the response is to a setting change command, the voice synthesis unit 705 generates voice synthesis data and controls the data transmission/reception unit 701 to send a request command for setting information of the MFP 101 main body to the MFP 101.

<Overall processing performed by the image processing system, part 1>
8 is a sequence diagram showing a process executed by the image forming system 100 in this embodiment. In the following description, each process (step) is represented by adding an S to the beginning, and the description of the process (step) is omitted. In FIG. 8, a case where an instruction is given to the MFP 101 using one smart speaker 110 will be described.

First, when the smart speaker 110 receives a wake word from a user (S801), the smart speaker 110 transmits device information (information about its own device) and startup information to the cloud server 103 (S802, S803). This starts an interactive session.
When the cloud server 103 receives the startup information from the smart speaker 110, it transmits a request command for setting information of the MFP 101 main body to the MFP 101 (S804). The MFP 101 acquires the setting information in response to the request command (S805) and transmits the setting information to the cloud server 103 (S806). When the cloud server 103 receives the setting information of the MFP 101 main body, it associates the received setting information with the device information received from the smart speaker 110 in S802 and stores them in the RAM 403 (S807).

Next, when the smart speaker 110 receives a speech from the user (S808), it transmits the device information and the speech information to the cloud server 103 (S809, S810). For example, the smart speaker 110 acquires the speech information based on the user's speech such as "Print file A" uttered following the wake word received in S801. Note that the device information transmitted in S809 is the same value as the device information transmitted in S803.

Next, the cloud server 103 performs voice recognition processing on the speech information received from the smart speaker 110 (S811) and determines the instruction content from the result of the voice recognition processing (S812). Here, since the instruction content is an instruction to execute a job, the cloud server 103 generates job information using the setting information stored in the RAM 403 in association with the device information received together with the speech information (S813). Next, when the cloud server 103 transmits the generated job information to the MFP 101 (S814), the MFP 101 transitions the display of the operation panel 209 to a job execution screen (S815) and transmits a job execution start notification to the cloud server 103 (S816). The cloud server 103 converts the received job execution start notification into voice synthesis data (S817) and transmits the voice synthesis data to the smart speaker 110 (S818). Next, the smart speaker 110 plays the voice synthesis data received from the cloud server 103 (S819).

The MFP 101 then performs job execution processing in accordance with the job information received in S814 (S820). When execution processing is completed, the MFP 101 transitions the display on the operation panel 209 to a job execution completion screen (S821) and transmits a job execution completion notification to the cloud server 103 (S822). The cloud server 103 converts the received job execution completion notification into voice synthesis data (S823) and transmits the voice synthesis data to the smart speaker 110 (S824). The smart speaker 110 then plays back the voice synthesis data received from the cloud server 103 (S825). For example, the smart speaker 110 plays back the voice synthesis data saying "Printed" through the speaker 310.

Next, the cloud server 103 determines whether to end the interaction session (S826), and if so, sends an interaction session end notification to the smart speaker 110 (S827). When the smart speaker 110 receives the interaction session end notification from the cloud server 103, it ends the interaction session (S828). The smart speaker 110 then transitions to a standby state. This completes the series of processes shown in the sequence diagram of FIG. 8.

<Processing performed by the cloud server>
Fig. 9 is a flowchart showing details of the processing executed by the cloud server 103 according to this embodiment. Although the case where the subject that executes each process shown in the flowchart of Fig. 9 is the CPU 402 of the cloud server 103 will be described, the subject that executes each process is not limited to this, and each process may be executed by a CPU of another device.
First, when the cloud server 103 receives data from the smart speaker 110, in S901, the CPU 402 determines whether the received data is startup information or speech information. If the CPU 402 determines that the data is startup information, the process proceeds to S902, and if the CPU 402 determines that the data is speech information, the process proceeds to S905. The startup information is, for example, a command to confirm the startup of a voice recognition function.
In step S<b>902 , the CPU 402 transmits a request command for setting information to the MFP 101 .
Next, in step S903, the CPU 402 acquires setting information of the MFP 101 main body as a response to the request command.
Next, in S904, the CPU 402 associates the acquired setting information of the main body of the MFP 101 with the device information of the smart speaker 110 that transmitted the startup information, and stores the association in a table on the RAM 403. After that, the process proceeds to S921.

In S<b>905 , the CPU 402 performs voice recognition processing on the speech information received from the smart speaker 110 .
Next, in step S906, the CPU 402 determines whether the instruction content is an instruction to execute a job based on the result of the voice recognition process in step S905. If the CPU 402 determines that the instruction content is an instruction to execute a job, the process proceeds to step S907. If the CPU 402 determines that the instruction content is not an instruction to execute a job, the process proceeds to step S912.
In S907, the CPU 402 reads out the setting information of a record that matches the device information of the smart speaker 110 that is the sender of the speech information, from the records stored in the table on the RAM 403.
Next, in S908, the CPU 402 generates a job execution command using the setting information read in S907. Note that in S907, if there is no record that matches the device information of the smart speaker 110 that is the sender of the speech information, or if there is a record but it does not include the setting items used in the job execution instruction, the job execution command is generated using default setting information.
Next, in step S<b>909 , the CPU 402 transmits the generated job execution command to the MFP 101 .
Next, in step S910, the CPU 402 receives a response notification to the command sent in step S909.
Next, in S911, the CPU 402 performs voice synthesis processing on the response notification received in S910 to generate voice synthesis data, and transmits the generated voice synthesis data to the smart speaker 110 that is the sender of the speech information. Here, for example, a voice such as "The file has been printed" is synthesized. After that, the process proceeds to S921.

In S912, the CPU 402 determines, based on the result of the voice recognition process in S905, whether or not the instruction content is an instruction to change the settings of the main body of the MFP 101. If the CPU 402 determines that the instruction content is an instruction to change the settings of the main body of the MFP 101, the process proceeds to S913, and if the CPU 402 determines that the instruction content is not an instruction to change the settings, the process proceeds to S920.
In S913, the CPU 402 generates a setting change command using the contents of the instruction from the user.
Next, in step S<b>914 , the CPU 402 transmits the generated setting change command to the MFP 101 .
Next, in S915, the CPU 402 receives a response notification to the command sent in S914.
Next, in S916, the CPU 402 performs voice synthesis processing on the response notification received in S915 to generate voice synthesis data, and transmits the generated voice synthesis data to the smart speaker 110 that is the transmission source of the speech information. Here, for example, a voice such as "The settings have been changed" is synthesized.
Next, in step S<b>917 , the CPU 402 transmits a request command for setting information to the MFP 101 .
Next, in S918, the CPU 402 acquires the changed setting information as a response to the request command.
Next, in S919, the CPU 402 updates the setting information of the record stored in the table on the RAM 403 that matches the device information of the smart speaker 110 that transmitted the speech information with the changed setting information acquired in S918. This makes it possible to execute the job using the changed settings if the user continues to instruct execution of the job after changing the settings. Then, the process proceeds to S921.

In S920, the CPU 402 performs necessary voice synthesis processing according to the state of the smart speaker 110, etc., to generate voice synthesis data, and transmits the generated voice synthesis data to the smart speaker 110 that is the sender of the speech information. Here, for example, a voice such as "Please speak again" is synthesized.
Next, in S921, the CPU 402 determines whether to end the interaction session. If the CPU 402 determines that the interaction session is to be ended, the process proceeds to S922. If the CPU 402 determines that the interaction session is to be continued, the process returns to S901, and the process waits for the transmission of speech information from the smart speaker 110.
In S922, the CPU 402 discards from the RAM 403, among the records held in the table on the RAM 403, any records that match the device information of the smart speaker 110 to be disconnected from the interaction session. Then, the series of processes shown in the flowchart in FIG. 9 ends.

<Overall processing performed by the image processing system, part 2>
10A and 10B are sequence diagrams showing the processing executed by the image forming system 100 in this embodiment. In FIG. 10A and FIG. 10B, a case will be described in which a user A starts an interaction session with the smart speaker 110 to execute a job, and a user B starts an interaction session with the smart speaker 111 at approximately the same time.

8 is executed in response to a voice operation start instruction from user A who operates the smart speaker 110. As a result, the setting information of the MFP 101 at the time when the smart speaker 110 starts the dialogue session is stored in the cloud server 103 in association with the information of the smart speaker 110.
8 is executed in response to a voice operation start instruction from user B who operates the smart speaker 111. As a result, the setting information of the MFP 101 at the time when the smart speaker 111 starts the dialogue session is stored in the cloud server 103 in association with the information of the smart speaker 111.

Next, when the smart speaker 111 receives a speech from user B (S1015), it transmits device information and speech information to the cloud server 103 (S1016, S1017). Next, the cloud server 103 performs a voice recognition process on the received speech information (S1018) and determines the instruction content from the result of the voice recognition process (S1019). Here, since the instruction content is an instruction to change the main body settings, the cloud server 103 generates change information of the instructed content (S1020). When the cloud server 103 transmits the generated change information to the MFP 101 (S1021), the MFP 101 uses the received change information to change the setting information stored in the storage 205 (S1022). After that, the MFP 101 transitions the display of the operation panel 209 to a change notification screen (S1023) and transmits a setting change notification to the cloud server 103 (S1024). The cloud server 103 converts the received setting change notification into voice synthesis data (S1025) and transmits the voice synthesis data to the smart speaker 111 (S1026). The smart speaker 111 then plays back the voice synthesis data received from the cloud server 103 (S1027).

Next, the cloud server 103 determines whether to end the interactive session with the smart speaker 111 (S1028), and if so, sends an interactive session end notification to the smart speaker 111 (S1029). When the smart speaker 111 receives the interactive session end notification, it ends the interactive session (S1030). The smart speaker 111 then transitions to a standby state.

Next, in S1031 to S1051, the same processes as S808 to S828 in Fig. 8 are executed in response to a job execution instruction from user A who operates the smart speaker 110. This completes the series of processes in the sequence diagrams shown in Fig. 10A and 10B.
As shown in the sequence diagram above, the cloud server 103 acquires and stores the setting information of the MFP 101 main body at the timing when user A starts an interactive session with the smart speaker 110. When user A subsequently instructs execution of a job using the smart speaker 110, the cloud server 103 instructs execution of the job using the stored setting information. As a result, even if user B, who started an interactive session at approximately the same time as user A, instructs to change the settings before user A instructs execution of the job, user A can execute the job with the settings before the change. In other words, printing can be performed as user A intended.

According to the image forming system 100 of this embodiment as described above, when changing the settings of an image forming device using a voice control device in an environment where multiple voice control devices exist, conflicts in operations are eliminated. Therefore, usability can be improved when operating an image forming device using a voice control device.

As a first modified example of this embodiment, the MFP 101 may also read and store the setting information of the MFP 101 main body in the RAM 203 when the CPU 202 of the MFP 101 detects the start of operation of the MFP 101 (for example, when user A logs in). In this case, when user A operates the operation panel 209 to instruct execution of a job, the MFP 101 executes the job using the setting information stored in the RAM 203. As a result, even if user B uses the smart speaker 110 to instruct a setting change while user A is operating the operation panel 209, and the setting information of the MFP 101 main body is changed, user A can execute the job with the setting contents before the change.

The present invention has been described above in conjunction with embodiments, but the above embodiments are merely illustrative examples of how the present invention may be implemented, and the technical scope of the present invention should not be interpreted in a limiting manner based on these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

Other Embodiments
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

100: Image forming system, 101: MFP, 110, 111: Smart speaker, 103: Cloud server

Claims (5)

An image forming system including a plurality of voice control devices that receive user utterances, a server that acquires instruction information based on utterance information acquired from the plurality of voice control devices, and an image forming device that executes a process based on the instruction information acquired from the server,
A transmission means for receiving a voice operation start instruction from a user in one of the plurality of voice control devices and transmitting start information to the server;
a requesting unit in the server for receiving the startup information and requesting setting information from the image forming apparatus;
a storage unit in the server for storing setting information of the image forming apparatus and information of the one of the voice control devices in association with each other;
a generating unit configured to generate the instruction information by using at least the stored setting information when instructing execution of a job based on the speech information acquired from the one voice control device in the server;
An image forming system comprising: the request means, when instructing a change in setting information of the image forming apparatus based on the speech information acquired from the one voice control apparatus, requests the image forming apparatus for the changed setting information;
2. The image forming system according to claim 1, wherein said storage unit updates the stored setting information to the changed setting information. The image forming system according to claim 1 or 2, characterized in that the instruction to start the voice operation is given by the user uttering a wake word. A control method for an image forming system including a plurality of voice control devices that receive user utterances, a server that acquires instruction information based on utterance information acquired from the plurality of voice control devices, and an image forming device that executes a process based on the instruction information acquired from the server, comprising:
a transmission step of receiving a voice operation start instruction from a user in one of the plurality of voice control devices and transmitting start information to the server;
a request step of receiving the startup information in the server and requesting setting information from the image forming apparatus;
a storage step of storing, in the server, setting information of the image forming apparatus and information of the one of the voice control devices in association with each other;
a generating step of generating, in the server, the instruction information by using at least the stored setting information when instructing execution of a job based on the speech information acquired from the one voice control device;
A control method comprising: A program for causing a computer to function as each of the means of the image forming system according to any one of claims 1 to 3.

Images