JP7201419B2 - Image processing system, information processing device, information processing method and program - Google Patents

Description

The present disclosure relates to an image processing system, an information processing device, an image processing device, an information processing method, an image processing method, and a program that generate image data by capturing an image of a subject.

2. Description of the Related Art Techniques for generating information in real time about an object existing in a video made up of a plurality of images in a video processing system are known (see Patent Document 1, for example). According to this technique, a first local feature amount consisting of an i-dimensional feature vector based on a feature point in an image of a recognition target, and a second local feature amount consisting of a j-dimensional feature vector based on the first image. and a third local feature amount consisting of a k-dimensional feature vector based on a second image different from the first image. Determine the appearance rate of

JP 2015-92302 A

However, in Patent Literature 1 described above, only the appearance rate of objects existing in the video is determined, and the association between the image and other information is not known when checking the image. For this reason, there has been a demand for a technique that can grasp the relevance to an image when checking the image.

The present disclosure has been made in view of the above, and provides an image processing system, an information processing device, an image processing device, an information processing method, which can grasp the relationship between an image and other information when checking the image. An object of the present invention is to provide an image processing method and program.

In order to solve the above-described problems and achieve the object, an image processing system according to the present disclosure provides first image data obtained by photographing a natural object by a first device and handwritten memos by a second device. a communication unit for receiving second image data obtained by photographing; and a display control unit for displaying the first image data on a display screen of a display unit, wherein the second image data is , including processing information for display on the display screen.

Further, an information processing apparatus according to the present disclosure includes a communication unit that receives first image data generated by an external device; and an imaging unit that captures an image of

Further, in the information processing apparatus according to the present disclosure, in the above disclosure, the processing information is character information synthesized when printing the first image data.

Further, in the information processing apparatus according to the present disclosure, in the above disclosure, the processing information is pasting information for pasting the first image corresponding to the first image data on the paper medium.

Further, in the information processing apparatus according to the present disclosure, in the above disclosure, the pasting information includes text information, position information indicating a position for pasting the first image on the paper medium, and size information.

Further, in the above disclosure, the information processing apparatus according to the present disclosure includes a calculating unit that calculates a pasting area for pasting the first image on the paper medium, and based on the pasting area calculated by the calculating unit: and an image processing unit that resizes the display area of the first image, wherein the imaging unit captures written information in which any one of characters, figures, and symbols written on the paper medium is written. The image is continuously captured, and the calculation unit calculates the pasting area based on the handwritten information appearing in the live view images sequentially generated by the image capturing unit.

Further, in the information processing apparatus according to the present disclosure, in the above disclosure, a recording control unit that generates an image file including the first image, the second image, and the processing information and records it in a recording unit further provide.

Further, in the above disclosure, the information processing apparatus according to the present disclosure further includes a display control unit that displays the live view image on a display unit, and the display control unit controls the first image resized by the image processing unit. An image is superimposed on the handwriting information in the live view image and displayed on the display unit.

Further, an image processing apparatus according to the present disclosure includes a first image generated by an information processing apparatus and generated by an external device, and a second image processing information having processing information for displaying the first image on a paper medium. a communication unit for receiving an image file containing an image of; a combining unit for generating combined data by combining the first image with the second image based on the processing information; and the combined data and an output unit that outputs the

Further, in the image processing apparatus according to the present disclosure, in the above disclosure, the processing information is character information synthesized when printing the first image.

Further, in the image processing apparatus according to the present disclosure, in the above disclosure, the processing information is pasting information for pasting the first image on the paper medium.

Further, an image processing method according to the present disclosure is an image processing method executed by an image processing system, in which first image data obtained by photographing a natural object by a first device and a handwritten memo by a second device are captured. and second image data obtained by photographing the image, and displaying the first image data on the display screen of the display unit, and displaying the second image data on the display screen contains processing information for

Further, an information processing method according to the present disclosure is an information processing method executed by an information processing apparatus, which receives a first image generated by an external device and displays the first image on a paper medium. A second image is captured that includes processing information for.

Further, an image processing method according to the present disclosure is an image processing method executed by an image processing apparatus, wherein a first image generated by an information processing apparatus and generated by an external device is transferred to a sheet of paper. a second image having processing information for display on a medium; and composite data by compositing the second image with the first image based on the processing information. and outputs the synthesized data.

A program according to the present disclosure is a program to be executed by an image processing system, and includes first image data obtained by photographing a natural object by a first device and handwritten memo by a second device. and displaying the first image data on a display screen of a display unit, the second image data being processing information for displaying on the display screen. include.

Further, a program according to the present disclosure is a program to be executed by an information processing apparatus, and includes processing information for receiving a first image generated by an external device and displaying the first image on a paper medium. capture a second image comprising

Further, a program according to the present disclosure is a program to be executed by an image processing apparatus, and displays a first image generated by an information processing apparatus and generated by an external device, and the first image on a paper medium. receiving an image file containing a second image having processing information for generating composite data by compositing the first image with the second image based on the processing information; Output the synthesized data.

Advantageous Effects of Invention According to the present disclosure, it is possible to grasp the relationship between an image and other information when confirming the image.

FIG. 1 is a schematic diagram showing an outline of an image processing system according to Embodiment 1. FIG. 2 is a block diagram showing a functional configuration of the capture device according to Embodiment 1. FIG. 3 is a block diagram showing a functional configuration of the mobile terminal according to Embodiment 1. FIG. 4 is a block diagram of a functional configuration of a server according to Embodiment 1. FIG. FIG. 5 is a diagram showing a situation in which a user is observing a specimen using a capture device. FIG. 6 is a diagram showing an example of an image captured by the capture device. FIG. 7 is a diagram showing an example of a notebook created by a user. FIG. 8 is a diagram showing a situation in which a user takes an image of a notebook using a portable terminal. FIG. 9 is a diagram schematically showing an example of an image generated by the server. FIG. 10 is a diagram schematically showing an example of data stored in an image file. FIG. 11A is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the mobile terminal. FIG. 11B is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the mobile terminal. FIG. 11C is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the mobile terminal. FIG. 11D is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the mobile terminal. FIG. 11E is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the mobile terminal. FIG. 11F is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the mobile terminal. 12 is a flowchart showing an overview of processing executed by the capture device according to Embodiment 1. FIG. FIG. 13 is a flow chart showing an overview of observation mode processing in FIG. 14 is a flowchart illustrating an outline of processing executed by the mobile terminal according to Embodiment 1. FIG. FIG. 15 is a flow chart showing an outline of microscope mode processing of FIG. FIG. 16 is a flow chart showing an overview of the AR display processing of FIG. 17 is a diagram schematically showing a method of calculating a pasting area calculated by a calculating unit according to Embodiment 1. FIG. 18 is a diagram schematically showing a method of calculating a pasting area calculated by a calculating unit according to Embodiment 1. FIG. 19 is a flowchart illustrating an overview of processing executed by the server according to Embodiment 1; FIG. FIG. 20 is a schematic diagram showing an outline of an image processing system according to the second embodiment. 21 is a block diagram of a functional configuration of an image processing apparatus according to Embodiment 2. FIG. FIG. 22 is a diagram showing a situation when the user attaches a memo to an image captured by the capture device and prints or sends the image. FIG. 23 is a diagram schematically showing an example of an image transmitted from the capture device to the image processing device. FIG. 24 is a diagram showing a situation in which a memo created by a user is being photographed with a portable terminal. FIG. 25 is a diagram schematically showing an example of an image of a memo captured by a mobile terminal. FIG. 26 is a diagram schematically showing an example user of an image printed or transmitted by the image processing apparatus. FIG. 31 is a diagram schematically showing an example of an image printed by the image processing apparatus. FIG. 32 is a diagram schematically showing an example of an image printed by the image processing apparatus. FIG. 29A is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the terminal device. FIG. 29B is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the terminal device. FIG. 29C is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the terminal device. FIG. 29D is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the terminal device. FIG. 29E is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the terminal device. FIG. 29F is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the terminal device. FIG. 29G is a diagram schematically showing the relationship between screen transitions and the capture device during cooperative processing between the capture device and the terminal device. 30 is a flowchart showing an outline of processing executed by the capture device according to Embodiment 2. FIG. 31 is a flowchart showing an outline of reproduction mode processing executed by the capture device according to Embodiment 2. FIG. FIG. 32 outlines the processing executed by the image processing apparatus according to the second embodiment.

Hereinafter, embodiments for implementing the present disclosure will be described in detail with drawings. It should be noted that the present disclosure is not limited by the following embodiments. In addition, each drawing referred to in the following description only schematically shows the shape, size, and positional relationship to the extent that the contents of the present disclosure can be understood. That is, the present disclosure is not limited only to the shapes, sizes and positional relationships illustrated in each drawing.

(Embodiment 1)
Advantageous Effects of Invention According to the present disclosure, it is possible to obtain an effect that it is possible to grasp the relationship between an image and other information obtained by photographing a memo or the like (such as an image obtained by photographing the memo) when confirming an image. In particular, handwritten memos have different messages depending on whether they are written with vigor or carefully. It is possible to tell.

In recent years, an environment has been developed in which users can input information using dedicated input devices such as keyboards, touch panels, mice, and voice input. There is also the simplicity and emotional charm of the handwriting, and such handwriting is emphasized in situations such as quickly taking notes with one hand as in the first embodiment of the present disclosure. It's a scene. Under these circumstances, there has been a demand for improvements in handwritten and digital information matching technology.

It is often difficult to understand what an image of a natural object that does not contain characters or symbols by itself is, and adding such handwritten information to such first image data often increases its value. . It is important not only to skip this first image data by communication and reproduce it on a paper medium, but also to make effective use of the handwritten information on this same paper medium. This technology takes into account the information generated by photographing handwritten information with a mobile terminal (external device) such as a smartphone, and this handwritten information is also reproduced on the paper medium. can be described as a second image containing the processing information of .

[Overview of image processing system]
FIG. 1 is a schematic diagram showing an outline of an image processing system according to Embodiment 1. FIG. An image processing system 1 shown in FIG. and a server 4 that receives various information from and transmits various information.

[Configuration of capture device]
First, the configuration of the capture device 2 will be described. FIG. 2 is a block diagram showing the functional configuration of the capture device 2. As shown in FIG. The capture device 2 shown in FIG. 2 is a microscope such as a video microscope, biological microscope and industrial microscope. In FIG. 2, a microscope is used as an example, but other than the microscope, an imaging device such as a digital camera, a tablet terminal device, a camcorder, an IC recorder with a photographing function, an industrial or medical endoscope may be used. can also be applied.

The capture device 2 shown in FIG. 2 includes an optical system 21, an imaging unit 22, a sound pickup unit 23, a clock 24, an operation unit 25, a communication unit 26, a recording unit 27, a display unit 28, a control a portion 29;

The optical system 21 forms a subject image and outputs the formed subject image to the user and the imaging section 22 . The optical system 21 is configured using at least one or more lenses. Also, the optical system 21 changes the focal length and focal position under the control of the controller 29 . In Embodiment 1, the optical system 21 is described as an objective lens or an eyepiece lens, but it may be an interchangeable lens or the like depending on the configuration of the capture device 2 .

Under the control of the control unit 29, the imaging unit 22 receives a subject image formed by the optical system 21 and performs photoelectric conversion to generate image data. The imaging unit 22 outputs image data to the control unit 29 . The imaging unit 22 is configured using a CMOS (Complementary Metal Oxide Semiconductor), a CCD (Charge Coupled Device), an A/D conversion circuit, and the like.

Under the control of the control unit 29, the sound pickup unit 23 acquires the voice uttered by the user, and A/D converts the acquired voice to generate voice data. The sound pickup unit 23 outputs audio data to the control unit 29 . The sound pickup unit 23 is configured using a microphone, an amplifier circuit, an A/D conversion circuit, and the like.

The clock 24 measures time and outputs the result of the time measurement to the control section 29 . The clock 24 generates date and time information related to the date and time of the image data generated by the imaging unit 22 and outputs this date and time information to the control unit 29 in addition to the time keeping function. The clock 24 is configured using a clock generator or the like.

The operation unit 25 receives an input of a user's operation and outputs a signal corresponding to the received operation to the control unit 29 . The operation unit 25 is configured using a touch panel, switches, buttons, a jog dial, a rotating lever, and the like.

Under the control of the control unit 29, the communication unit 26 receives various information input via the network 100 according to a predetermined communication standard such as Wi-Fi (Wireless Fidelity) (registered trademark) or Bluetooth (registered trademark). is output to the control unit 29, and various information input from the control unit 29 is transmitted to the outside. The communication unit 26 is configured using a communication module or the like.

The recording unit 27 records various information about the capture device 2 . The recording unit 27 is configured using a flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. The recording unit 27 includes an image file recording unit 271 for recording an image file containing image data, an audio file recording unit 272 for recording an audio file containing audio data, and a recording unit for recording the first image data and the first image data on paper. An auxiliary data recording unit 273 for recording auxiliary data including second image data including processing information for display on a medium, and a program for recording various programs executed by the capture device 2 and various data used during processing. and a recording unit 274 .

The display unit 28 displays various information regarding the capture device 2 under the control of the control unit 29 . The display section 28 displays an image corresponding to the image data generated by the imaging section 22 . The display unit 28 is configured using a display panel such as liquid crystal or organic EL (Electro Luminescence).

The control unit 29 comprehensively controls the control of each unit that configures the capture device 2 . The control unit 29 is configured using a memory and a processor having hardware. Specifically, the control unit 29 is configured using a CPU (Central Processing Unit), a memory, a GPU (Graphics Processing Unit), and the like. The control unit 29 has a shooting control unit 291 , a display control unit 292 , a recording control unit 293 and a communication control unit 294 .

The imaging control section 291 controls imaging by the imaging section 22 . The imaging control unit 291 causes the imaging unit 22 to perform imaging according to the operation of the operation unit 25 .

The display control section 292 controls the display of the display section 28 . For example, the display control unit 292 causes the display unit 28 to display various information regarding the capture device 2 .

The recording control unit 293 creates an image file in which at least the image data generated by the imaging unit 22 and the audio data generated by the sound pickup unit 23, the timing result of the clock 24, and the history information related to the operation history of the operation unit 25 are associated with each other. , and records this image file in the recording unit 27 .

The communication control section 294 controls communication of the communication section 26 . The communication control unit 294 causes the communication unit 26 to transmit various information of the capture device 2 , such as image data generated by the imaging unit 22 .

[Configuration of mobile terminal]
Next, the configuration of the mobile terminal 3 will be described. FIG. 3 is a block diagram showing the functional configuration of the mobile terminal 3. As shown in FIG. The mobile terminal 3 shown in FIG. 3 is, for example, a mobile phone with a photographing function. The mobile terminal 3 shown in FIG. 3 includes an imaging unit 31 , a clock 32 , an operation unit 33 , a communication unit 34 , a recording unit 35 , a display unit 36 and a control unit 37 .

The imaging unit 31 generates image data under the control of the control unit 37 and outputs the generated image data to the control unit 37 . The imaging unit 31 is configured using one or more optical systems and a CMOS or CCD that generates image data by receiving a subject image formed by the optical systems.

Clock 32 counts time and outputs the result of the clocking to control unit 37 . The clock 32 generates date and time information related to the date and time of the image data generated by the imaging unit 31 and outputs this date and time information to the control unit 37 in addition to the timekeeping function. The clock 32 is configured using a clock generator or the like.

The operation unit 33 receives an input of a user's operation and outputs a signal corresponding to the received operation to the control unit 37 . The operation unit 33 is configured using a touch panel, switches, buttons, and the like.

Under the control of the control unit 37, the communication unit 34 outputs various types of information input via the network 100 to the control unit 37 according to a predetermined communication standard, and transmits various types of information input from the control unit 37. Send externally. The communication unit 34 is configured using a communication module or the like.

The recording unit 35 records various information about the mobile terminal 3 . The recording unit 35 is configured using a recording medium such as a volatile memory, a nonvolatile memory, and a memory card. The recording unit 35 includes an image file recording unit 351 for recording an image file containing image data, and second image data containing first image data and processing information for displaying the first image data on a paper medium. and a program recording unit 353 for recording various programs executed by the portable terminal 3 and various data used during processing.

Under the control of the control unit 37, the display unit 36 displays various types of information about the mobile terminal 3 and images corresponding to image data. The display unit 36 is configured using a display panel such as liquid crystal or organic EL (Electro Luminescence).

The control unit 37 comprehensively controls the operation of each unit that configures the mobile terminal 3 . The control unit 37 is configured using a memory and a processor having hardware. Specifically, the control unit 37 is configured using a memory, a CPU, a GPU, and the like. The control unit 37 includes a detection unit 371, a calculation unit 372, an image processing unit 373, a synthesis unit 374, an imaging control unit 375, a display control unit 376, a recording control unit 377, a communication control unit 378, have

The detection unit 371 detects position information regarding the position of handwritten information in which any one of characters, graphics, and symbols written on a paper medium appears in an image corresponding to the image data input from the imaging unit 31 . Such handwritten information is intuitive, quick, convenient, easy to write and modify, and has advantages that cannot be ignored. By writing with vigor or by writing carefully, this handwritten character will have a different message. , it becomes possible to convey. In memos for experiments, the sense of urgency and atmosphere at that time may appear in the letters.

The calculation unit 372 calculates a pasting area for pasting the image of the capture device 2, which is the first image, on the paper medium. For example, the calculation unit 372, based on handwritten information in which any of characters, figures, and symbols written on a paper medium appearing in the live view image corresponding to the image data sequentially input from the imaging unit 31 is written, Calculate the pasting area. Such handwritten information such as characters, graphics, and areas is intuitive, quick, and convenient when instantly instructing the entire layout, and is easy to write and modify, which is an advantage that cannot be ignored.

The imaging control section 375 controls imaging by the imaging section 31 . The imaging control unit 375 causes the imaging unit 31 to perform imaging according to the operation of the operation unit 33 .

The display control section 376 controls display on the display section 36 . For example, the display control unit 376 causes the display unit 36 to display various information regarding the mobile terminal 3 .

The recording control unit 377 generates a second image including image data, which is the first image received from the capture device 2, and processing information for displaying the first image generated by the imaging unit 31 on a paper medium. Then, an image file containing pasting information for pasting the first image corresponding to the first image data on a paper medium is generated, and this image file is recorded in the recording unit 35 . Here, the pasting information includes text information, position information indicating the position for pasting the first image on the paper medium, and size information (pasting area) indicating the size.

The communication control section 378 controls communication of the communication section 34 . The communication control unit 378 causes the communication unit 34 to transmit various information including image files.

[Configuration of image processing device]
Next, the configuration of the server 4 will be described. FIG. 4 is a block diagram showing the functional configuration of the server 4. As shown in FIG. The server 4 shown in FIG. 4 is a device that receives image data via the network 100 and outputs images and data corresponding to this image data. The server 4 shown in FIG. 4 includes a communication unit 41, an output unit 42, a recording unit 43, and a control unit 44.

Under the control of the control unit 44, the communication unit 41 outputs various types of information input via the network 100 to the control unit 44 according to a predetermined communication standard, and transmits various types of information input from the control unit 44. Output to outside. The communication unit 41 is configured using a communication module or the like.

The output section 42 outputs an image corresponding to the image data under the control of the control section 44 . For example, the output unit 42 prints and outputs an image corresponding to the image data on a medium such as paper under the control of the control unit 44 . The output unit 42 is configured using a printer or the like, for example.

The recording unit 43 records various information about the server 4 . The recording unit 43 is configured using a volatile memory, a nonvolatile memory, an HDD, an SSD, and the like. The recording unit 43 has a program recording unit 431 for recording various programs executed by the server 4 and data used during processing.

The control unit 44 comprehensively controls the operation of each unit that configures the server 4 . The control unit 44 is configured using a memory and a processor having hardware. Specifically, the control unit 44 is configured using a memory, a CPU, a GPU, and the like. The control unit 44 has a synthesizing unit 441 , an output control unit 442 and a communication control unit 443 .

A synthesizing unit 441, image data that is the first image received from the capture device 2, and a second image including processing information for displaying the first image generated by the imaging unit 31 on a paper medium. , and pasting information for pasting a first image corresponding to the first image data on a paper medium to generate combined data.

The output control unit 442 and the output of the output unit 42 are controlled. The output control unit 442 causes the output unit 42 to print an image corresponding to the image data on a paper medium.

It controls communication between the communication control unit 443 and the communication unit 41 . The communication control unit 443 causes the communication unit 41 to transmit various information including image files.

[Outline of processing of the image processing system]
Next, an outline of processing executed by the image processing system 1 will be described. FIG. 5 is a diagram showing a situation in which the user is observing the specimen using the capture device 2. As shown in FIG. FIG. 6 is a diagram showing an example of an image captured by the capture device 2. As shown in FIG. FIG. 7 is an example of a note created by a user. FIG. 8 is a diagram showing the situation when the user uses the portable terminal 3 to photograph the notebook. FIG. 9 is a diagram schematically showing an example of an image generated by the server 4. As shown in FIG.

As shown in FIG. 5, the user U1 is creating a notebook N100 (for example, a lab notebook) while observing the specimen using the capture device 2. FIG. Under this circumstance, the user U1 wishes to paste the image P1 shown in FIG. 6 onto the desired paste area of the notebook N100. However, the user U1 is in the process of creating the notebook N100, and it is difficult for the user U1 to output and attach the image P1. Even if an attempt is made to paste P1, the pasting position of image P1 cannot sometimes be grasped because the image P1 is not associated with the pasting area. Furthermore, as the number of notebooks N100 and the number of images to be pasted increases, it becomes more difficult for the user U1 to grasp the relationship between the images and pasting positions, and the wrong images are pasted.

For this reason, as shown in FIG. 8, the user U1 attaches a square frame W1 for pasting the image P1 to a predetermined pasting area D1 in the notebook N100 and a character (for example, "Here is A") inside the frame W1. to write (handwriting input). Then, as shown in FIG. 8, the user U1 uses the portable terminal 3 to photograph the notebook N100. Specifically, the user U1 uses the portable terminal 3 to capture an image P2 including processing information for displaying the image P1 on the notebook N100, which is a paper medium, and generates an image file. In this case, as shown in FIG. 9, the mobile terminal 3 captures an image P2 including processing information for displaying the image P1 on the notebook N100, which is a paper medium. Here, the processing information includes pasting information for pasting to the pasting area D1 in the notebook N100. The pasting information includes text information, position information indicating the position for pasting the image P1, which is the first image, on the notebook N100, and size information. Specifically, as shown in FIG. 10, the recording control unit 377 of the mobile terminal 3 stores the file name T1, the image data T2 for recording the synthesized data obtained by synthesizing the first image data and the second image data. , text information T3, position information and size information T4 of paste information as processing information, and other meta-data T5 such as the model name and photographing time are generated. In this case, the portable terminal 3 uses well-known pattern matching or the like on the image corresponding to the image data generated by the imaging unit 31 so that the detection unit 371 detects a figure or a closed area for pasting. The position information is detected, and the calculator 372 calculates the size of the pasting area D1. Then, in the portable terminal 3, the image processing unit 373 performs image processing for resizing the display area of the image P1 based on the pasting area D1, and the synthesizing unit 374 synthesizes the image P1 and the image P2 to obtain synthesized data. A corresponding image P3 is generated, and the recording control unit 377 stores image data corresponding to the image P3 in the image file F1.

The mobile terminal 3 stores an image P3 that associates the image P1, which is the first image, with the second image P2 including processing information for displaying the image P1 on the notebook N100, which is a paper medium. Send the file to the server 4 . The server 4 performs printing by outputting the image P3 stored in the image file received from the portable terminal 3 to a printer (not shown). As a result, even when the user U1 pastes the image P1 on the notebook N100 after finishing creating the notebook N100, the image P1 and the image P2 are associated with each other so that the image P3 can be attached to the paste area D1. Since letters are written, it is easy to know where to put it. 7 to 9, the mobile terminal 3 performs image processing for resizing the display area of the image P1 based on the pasting area D1, and synthesizes the image P1 and the image P2 to create an image corresponding to the synthesized data. P3 is generated, but the server 4 performs image processing for resizing the display area of the image P1 based on the paste area D1, and synthesizes the image P1 and the image P2 to generate the image P3 corresponding to the synthesized data. may be generated.

[Overview of cooperative processing between capture device and mobile terminal]
11A to 11F are diagrams schematically showing the relationship between screen transitions and the capture device 2 during cooperative processing between the capture device 2 and the mobile terminal 3. FIG.

As shown in FIG. 11A, first, when the user U1 selects an icon A2 connected to the capture device 2 from among a plurality of icons A1 to A5 displayed by the display unit 36, the portable terminal 3 displays the icon A2 in FIG. 11B. As shown, the mobile terminal 3 displays icons A11 to A14 on the display unit 36. FIG. The icon A1 receives an input of an instruction signal for instructing another mode for performing other operations that can change settings of the mobile terminal 3 or the like. The icon A2 receives input of an instruction signal instructing a microscope mode in which image data or the like is received from the capture device 2 and linked. The icon A3 accepts input of an instruction signal instructing a moving image mode in which moving image data is received from the capture device 2 and linked. Icon A4 accepts input of an instruction signal that instructs a transmission mode for transmitting mail to another device. Icon A5 accepts an input of an instruction signal that instructs a telephone mode for talking to another device. Also, the icon A11 receives an input of an instruction signal requesting image data from an image captured by the capture device 2, which is a microscope. The icon A12 accepts an input of an instruction signal for transitioning to a mode in which characters written by the user U1 or a memo written by the user U1 are associated with an image. The icon A13 receives input of an instruction signal for transitioning to the previous display.

In the situation shown in FIG. 11B, when the icon A11 is selected by the user U1 from the icons A11 to A13 displayed by the display unit 36, the portable terminal 3 corresponds to the image data generated by the capture device 2. Acquire an image P1.

11C, when the icon A12 is selected from the icons A11 to A13 displayed by the display unit 36 by the user U1, the portable terminal 3 displays the icon as shown in FIG. 11D. displays the image P1, the icon A21, the icon A22 and the icon A13 on the display unit . The icon A21 receives an input of an instruction signal for transitioning to a mode for photographing a paper medium on which the user U1 has written characters for the image P1. The icon A22 receives an input of an instruction signal for transitioning to a mode for inputting text using the UI of the mobile terminal 3 .

Thereafter, when the icon A21 is selected by the user U1, the mobile terminal 3 captures the paper medium on which the user U1 has written characters on the imaging unit 31, and displays the image P2 corresponding to the image data generated live on the display unit 36. Display the view (see FIG. 11E). Then, when the operation unit 33 is operated by the user U1 and an image is captured, the portable terminal 3 generates an image file in which the image P1 and the image P2 are associated with each other, and displays the image P1 and the image P2 on the display unit 36. (See FIG. 11F).

[Processing of capture device]
Next, processing executed by the capture device 2 will be described. FIG. 12 is a flow chart showing an overview of the processing executed by the capture device 2. As shown in FIG.

As shown in FIG. 12, first, the case where the capture device 2 is set to the observation mode (step S1: Yes) will be described. In this case, the capture device 2 executes observation mode processing for observing the specimen (step S2). Note that the observation mode processing will be described later.

Subsequently, when an instruction signal instructing termination is input from the operation unit 25 (step S3: Yes), the capture device 2 terminates this process. On the other hand, if an instruction signal instructing termination is not input from the operation unit 25 (step S3: No), the capture device 2 returns to step S1.

In step S1, if the capture device 2 is not set to the observation mode (step S1: No), the capture device 2 proceeds to step S4.

Subsequently, when the capture device 2 is set to the communication setting mode (step S4: Yes), the capture device 2 performs communication settings for various settings for communicating with the mobile terminal 3 or the server 4 (step S5). After step S5, the capture device 2 proceeds to step S3.

In step S4, if the capture device 2 is not set to the communication setting mode (step S4: No), the capture device 2 proceeds to step S3.

[Observation mode processing]
Next, the details of the observation mode processing described in step S2 of FIG. 12 will be described. FIG. 13 is a flowchart showing an overview of observation mode processing.

As shown in FIG. 13, first, the control unit 29 causes the imaging unit 22 to perform imaging and generate image data (step S10).

Subsequently, when communication between the capture device 2 and the mobile terminal 3 is set (step S11: Yes), the communication control section 294 causes the communication section 26 to transmit the image data generated by the imaging section 22 to the mobile terminal 3. (Step S12). After step S12, the capture device 2 proceeds to step S13. On the other hand, if the capture device 2 is not set to communicate with the mobile terminal 3 (step S11: No), the capture device 2 proceeds to step S13.

Subsequently, when the operation unit 25 is operated (step S13: Yes), the imaging control unit 291 controls the parameters of the imaging unit 22 based on the operation of the operation unit 25 (step S14). After step S14, the capture device 2 proceeds to step S15. On the other hand, if the operation unit 25 has not been operated (step S13: No), the capture device 2 proceeds to step S15.

Thereafter, when an instruction signal instructing recording is input from the operation unit 25 (step S15: Yes), the imaging control unit 291 causes the imaging unit 22 to perform imaging and generate image data (step S16).

Subsequently, the recording control unit 293 records the operation history including the image data generated by the imaging unit 22 and the current parameters of the imaging unit 22 in the image file recording unit 271 (step S17).

After that, if the capture device 2 is set to transmit the image file to the mobile terminal 3 (step S18: Yes), the communication control section 294 causes the communication section 26 to transmit the image file to the mobile terminal 3 (step S19). ). After step S19, the capture device 2 returns to the main routine of FIG.

In step S15, if an instruction signal instructing recording has not been input from the operation unit 25 (step S15: No), the capture device 2 returns to the main routine of FIG.

In step S18, if the capture device 2 is not set to transmit the image file to the mobile terminal 3 (step S18: No), the capture device 2 returns to the main routine of FIG.

[Handling of mobile terminal]
Next, processing executed by the mobile terminal 3 will be described. FIG. 14 is a flow chart showing an overview of the process executed by the mobile terminal 3. As shown in FIG.

As shown in FIG. 12, first, the display control unit 376 causes the display unit 36 to display icons related to each of a plurality of modes executable by the mobile terminal 3 (step S21). Specifically, the display control unit 376 causes the display unit 36 to display the icons A1 to A5 described above with reference to FIG. 11A.

Subsequently, when the icon A3 is operated using the operation unit 33 and the microscope mode is selected (step S22: Yes), the portable terminal 3 executes microscope mode processing to connect with the capture device 2 (step S23). . Details of the microscope mode processing will be described later. After step S23, the mobile terminal 3 proceeds to step S24.

After that, when an instruction signal instructing termination is input from the operation unit 33 (step S24: Yes), the mobile terminal 3 terminates this process. On the other hand, when the instruction signal instructing the end is not input from the operation unit 33 (step S24: No), the portable terminal 3 returns to step S21 described above.

In step S22, if the icon A3 is not operated using the operation unit 33 and the microscope mode is not selected (step S22: No), the mobile terminal 3 proceeds to step S25.

Subsequently, when the operation unit 33 is operated to set the communication setting mode in which the mobile terminal 3 sets communication with the capture device 2 (step S25: Yes), the mobile terminal 3 communicates with the capture device 2. Communication settings are made (step S26).

After that, the mobile terminal 3 receives the image from the capture device 2 (step S27). After step S27, the mobile terminal 3 proceeds to step S24.

In step S25, if the operation unit 33 is not operated and the communication setting mode in which the mobile terminal 3 sets communication with the capture device 2 is not set (step S25: No), the mobile terminal 3 proceeds to step S24. Transition.

[Microscope mode processing]
Next, details of the microscope mode processing described in step S23 of FIG. 14 will be described. FIG. 15 is a flowchart showing an overview of microscope mode processing.

As shown in FIG. 15, when an image is received from the capture device 2 (step S31: Yes), the display control section 376 causes the display section 36 to display the image received from the capture device 2 (step S32).

Subsequently, when the operation unit 33 is operated to acquire an image from the capture device 2 (step S33: Yes), the mobile terminal 3 proceeds to step S34 described later. On the other hand, if the operation unit 33 is not operated and no image is acquired from the capture device 2 (step S33: No), the mobile terminal 3 proceeds to step S36, which will be described later.

In step S34, when performing AR display (step S34: Yes), the portable terminal 3 performs AR display processing (step S35). After step S35, the mobile terminal 3 proceeds to step S36.

[AR display processing]
FIG. 16 is a flow chart showing an overview of the AR display processing in step S35 of FIG.

As shown in FIG. 16, the imaging control unit 375 causes the imaging unit 31 to perform imaging (step S51).

Subsequently, the calculation unit 372 calculates a pasting area on which the first image acquired from the capture device 2 is superimposed from the image corresponding to the image data generated by the imaging unit 31 (step S52). Specifically, as shown in FIG. 17, the calculation unit 372 sets the focal distance from the center L1 of the optical system 311 of the imaging unit 31 to the light receiving surface of the image sensor 312 as F, and the image is formed on the light receiving surface of the image sensor 312. When the size of the character M1 to be displayed is Sy, the distance from the center L1 of the optical system 311 to the note N100 is D, and the size of the character M1 on the note N100 is Py, the following equation (1) holds.
Py=(Sy*D)/F (1)
Here, since the focal length F, the size Sy of the image sensor 312, and the distance D are known, the calculator 372 uses the above equation (1) to calculate the size Py of the character M1 on the note N100, that is, A sticking area can be calculated. Note that the calculation unit 372 may calculate the size Sy from the number of pixels of the character M1 on the image sensor 312 . Further, the calculation unit 372 may calculate the distance D based on detection results from a plurality of phase difference sensors (phase difference pixels) arranged on the imaging device 312 . Furthermore, the calculator 372 may calculate the focal length F based on a position sensor provided in the optical system 311 . Also, even if FIG. 18 is also a figure, the calculator 372 can similarly calculate the pasting area.

After that, the image processing unit 373 performs resizing processing for adjusting the image size of the first image to be pasted based on the pasting area (step S53).

Subsequently, the display control unit 376 superimposes and displays the first image corresponding to the image data resized by the image processing unit 373 (step S54). After step S54, the mobile terminal 3 returns to the subroutine of FIG.

Returning to FIG. 15, the description after step S36 is continued.
In step S36, if the mode of the mobile terminal 3 is a text memo for creating a text memo (step S36: Yes), the image processing unit 373 creates a text memo according to the operation of the operation unit 33 (step S37). ).

Subsequently, the recording control unit 377 generates an image file storing the text memo created by the image processing unit 373 according to the operation of the operation unit 33 and the image data (step S38). After step S38, the portable terminal 3 proceeds to step S42, which will be described later.

In step S36, if the mode of the mobile terminal 3 is text memo for creating a text memo (step S36: No), the mobile terminal 3 proceeds to step S39.

Subsequently, if the mode of the mobile terminal 3 is memo photography (step S39: Yes), the mobile terminal 3 proceeds to step S40, which will be described later. On the other hand, if the mode of the mobile terminal 3 is not memo photography (step S39: No), the mobile terminal 3 proceeds to step S42, which will be described later.

In step S40, the imaging control section 375 causes the imaging section 31 to perform imaging. In this case, the calculation unit 372 analyzes the processing information for displaying the image P1, which is the first image data, on a paper medium from the image corresponding to the image data generated by the imaging unit 31. FIG. Specifically, the calculation unit 372 performs the same processing as that shown in FIG. calculate. Furthermore, the detection unit 371 detects position information regarding the position of the pasting area on the paper medium (for example, on the notebook N100).

Subsequently, the recording control unit 377 meta-sizes paste information including size information about the size of the paste area calculated by the calculation unit 372, position information about the position of the paste area detected by the detection unit 371, and text information. It is stored as data in an image file (step S41). After step S41, the mobile terminal 3 proceeds to step S42.

After that, when an instruction signal for determining the content is input from the operation unit 33 (step S42: Yes), the recording control unit 377 records an image file in which the acquired file and the memo information are related to each other in the recording unit 35 ( step S43). For example, the recording control unit 377 stores an image file (see FIG. 10) in which the text information or metadata generated in step S37 or step S41 described above and the first image acquired from the capture device 2 are associated with each other. 35 is recorded in the auxiliary data recording unit 352 .

Subsequently, when an instruction signal for transmitting an image file in which the acquisition file and the memo information are associated to the server 4 is input from the operation unit 33 (step S44: Yes), the communication control unit 378 directs the An image file in which the acquired file and the memo information are associated is transmitted to the communication unit 34 (step S45).

Thereafter, when an instruction signal instructing termination is input from the operation unit 33 (step S46: Yes), the mobile terminal 3 terminates this process. On the other hand, when the instruction signal instructing the end is not input from the operation unit 33 (step S46: No), the portable terminal 3 returns to step S33 described above.

In step S42, when the instruction signal for determining the content has not been input from the operation unit 33 (step S42: No), the mobile terminal 3 proceeds to step S46.

In step S44, if the command signal for transmitting the image file in which the acquired file and the memo information are associated to the server 4 is not input from the operation unit 33 (step S44: No), the portable terminal 3 proceeds to step S46. do.

[Server processing]
Next, processing executed by the server 4 will be described. FIG. 19 is a flow chart showing an overview of the processing executed by the server 4. As shown in FIG.

As shown in FIG. 19, when the server 4 receives the image file from the mobile terminal 3 (step S61: Yes), the server 4 causes a printer (not shown) to print the image P3 stored in the image file F1 (step S62). After step S62, the server 4 terminates this process. On the other hand, if the image file has not been received from the mobile terminal 3 (step S61: No), the server 4 terminates this process.

According to the first embodiment described above, since the portable terminal 3 captures the second image including the processing information for displaying the first image data on a paper medium, the relationship between the image and the image can be confirmed when confirming the image. You can understand gender.

Further, according to Embodiment 1, the recording control unit 377 generates an image file including the first image, the second image, and the processing information, and records the image file in the recording unit 35. At the time of confirmation, the relevance to the image can be grasped.

Further, according to Embodiment 1, the display control unit 376 causes the display unit 36 to display the first image resized by the image processing unit 373 so as to be superimposed on the pasting area, which is the handwritten information in the live view image. The user U1 can intuitively grasp the image when pasting the first image on the paper medium.

Further, according to the first embodiment, the synthesizing unit 441 synthesizes the first image with the second image based on the processing information stored in the image file, thereby generating synthesized data. Since it is output to the output unit 42, it is possible to grasp the relationship with the image when confirming the image.

(Embodiment 2)
Next, Embodiment 2 will be described. In Embodiment 1 described above, a microscope is used as the capture device 2, but in Embodiment 2, an imaging device is used as the capture device. In the second embodiment, an environment has been prepared in which a user can input information using a dedicated input device such as a keyboard, touch panel, mouse, or voice input. Simplicity of input and correction and attractiveness of emotion also exist, and in situations such as this embodiment where handwriting that reflects the individuality of the user is used to take notes, it is easy to put heart into it, and handwriting is emphasized. This is the scene where We do not want to lose the shape and momentum information that reflects emotions and individuality, which is lost in print and text.Even in a situation where the amount of information unique to handwriting is required, there is a need to improve handwriting and digital information matching technology. rice field.

[Overview of image processing system]
FIG. 20 is a schematic diagram showing an outline of an image processing system according to the second embodiment. The image processing system 1A shown in FIG. 20 includes a capture device 2A capable of imaging a subject, a mobile terminal 3, a server 4, an image processing device 5 capable of communicating with the capture device 2A and the mobile terminal 3, Prepare. The capture device 2A is a device such as an imaging device such as a digital camera, a tablet terminal device, a camcorder, an IC recorder with a shooting function, or the like, and has the same functions as the capture device 2 according to the first embodiment described above. A description of such a configuration is omitted.

[Configuration of image processing device]
FIG. 21 is a block diagram showing the functional configuration of the image processing device 5. As shown in FIG. The image processing apparatus 5 shown in FIG. 21 includes a communication unit 51, an operation unit 52, an output unit 53, a display unit 54, a recording unit 55, and a control unit 56.

Under the control of the control unit 56, the communication unit 51 transmits various types of information input via the network 100 and outputs various types of information to the outside according to a predetermined communication standard. The communication unit 51 is configured using a communication module or the like.

The operation unit 52 receives an input of a user's operation and outputs a signal corresponding to the received operation to the control unit 56 . The operation unit 52 is configured using a keyboard, mouse, touch panel, switches, buttons, a jog dial, a rotating lever, and the like.

The output section 53 outputs an image corresponding to the image data under the control of the control section 56 . For example, the output unit 53 prints and outputs an image corresponding to the image data on a medium such as paper under the control of the control unit 56 . The output unit 53 is configured using a printer or the like, for example.

Under the control of the control unit 56, the display unit 54 displays various information about the image processing device 5 and images corresponding to image data. The display unit 54 is configured using a display panel such as liquid crystal or organic EL.

The recording unit 55 records various information about the image processing device 5 . The recording unit 55 is configured using a volatile memory, a nonvolatile memory, an HDD, an SSD, and the like. The recording unit 55 has a program recording unit 551 for recording various programs executed by the image processing apparatus 5 and data used during processing.

The control unit 56 comprehensively controls the operation of each unit that configures the image processing apparatus 5 . The control unit 56 is configured using a processor having memory and hardware. Specifically, the control unit 56 is configured using a memory, a CPU, a GPU, and the like. The control unit 56 has a synthesizing unit 561 , an output control unit 562 and a communication control unit 563 .

A synthesizing unit 561, a first image generated by the capture device 2A generated by the mobile terminal 3, and a second image having processing information for displaying the first image on a paper medium. Synthetic data is generated by synthesizing the first image with the second image based on the processing information stored in the image file.

The output control section 562 controls the output section 53 . The output control unit 562 causes the output unit 53 to print an image corresponding to the image data on a paper medium.

The communication control section 563 controls communication of the communication section 51 . The communication control unit 563 causes the communication unit 51 to transmit various information including image files.

[Outline of processing of the image processing system]
Next, an outline of processing executed by the image processing system 1A will be described. FIG. 22 is a diagram showing the situation when the user attaches a memo to an image captured by the capture device 2A and prints or sends the image. Here, by making handwritten notes, we do not want to lose the shape and momentum information that reflects emotions and individuality, assuming the amount of information unique to handwriting. is taking notes. Such handwritten information is intuitive, quick, and easy, and is easy to write and modify, and has advantages that cannot be ignored even in a system centered on digital processing.

FIG. 23 is a diagram schematically showing an example of an image transmitted from the capture device 2A to the image processing device 5. As shown in FIG. FIG. 24 is a diagram showing a situation in which a memo created by the user is being photographed by the mobile terminal 3. As shown in FIG. FIG. 25 is a diagram schematically showing an example of an image of a memo captured by the mobile terminal 3. As shown in FIG. FIG. 26 is a diagram schematically showing an example user of an image printed or transmitted by the image processing apparatus 5. As shown in FIG.

As shown in FIGS. 22 and 23, the user U1 sends the image captured by the capture device 2A to the image processing device 5. FIG. In this case, user U1 would like to print or transmit image P11 with a handwritten message. Therefore, the user U1 writes, for example, "How are you?" in the notebook N101 shown in FIG. 21 by hand. By writing with vigor or by writing carefully, this handwritten character will have a different message. , can be conveyed.

Subsequently, as shown in FIG. 24, the user U1 uses the portable terminal 3 to capture text information, which is a message written in the notebook N101. In this case, as shown in FIG. 25, the mobile terminal 3 generates an image P12 corresponding to the image data showing the message.

After that, the mobile terminal 3 transmits the image P12 to the image processing device 5 . In this case, as shown in FIG. 26, the image processing device 5 generates an image P13 corresponding to image data in which the image P11 transmitted from the capture device 2A and the image P12 transmitted from the portable terminal 3 are associated. Then, the image file containing this image P13 and the processing information is printed or transmitted via the network 100 and the server 4 to the device at the predetermined address designated by the user U1. Although the image P13 is generated using the image processing device 5 in FIGS. 22 to 26, the image P13 is generated by cooperation with the capture device 2A and the portable terminal 3, for example, without being limited to this. Then, an image file containing the image P13 and the processing information may be transmitted to the image processing device 5. FIG. In this case, the image processing device 5 outputs by printing the image P100 shown in FIG. The image P100 has the first image P1 captured by the capture device 2A printed on the front side and the second image P2 printed on the back side of the message. That is, the processing information is character information synthesized when printing the first image data, and includes information to be printed on the back side image. Further, as shown in FIG. 28, the image processing device 5 may print the image P200 shown in FIG. As for the image P200, the first image P201 and the message, which is the image P202, are printed on the same sheet. In this way, user U1, who is the sender, can send a fashionable photo containing a redundant message to the destination.

[Overview of cooperative processing between capture device and mobile terminal]
29A to 29G are diagrams schematically showing the relationship between screen transitions and the capture device 2A during cooperative processing between the capture device and the terminal device.

As shown in FIG. 29A, first, the mobile terminal 3 acquires or requests image data from the capture device 2A from among a plurality of icons A1, A3 to A5, and an icon A100 displayed by the display unit 36 by the user U1. When the icon A100 is selected, a request signal requesting image data is transmitted to the capture device 2A.

Subsequently, as shown in FIG. 29B, the capture device 2A transmits an image P11 corresponding to the image data to the mobile terminal 3 based on the request signal from the mobile terminal 3. FIG.

Thereafter, as shown in FIG. 29C, when the icon A102 is selected by the user U1 from the plurality of icons A100 to A102 displayed by the display unit 36, the portable terminal 3 displays the image P11 received from the capture device 2A. 29D).

Subsequently, when the icon A21 is selected by the user U1, the mobile terminal 3 causes the image capturing unit 31 to display the image P12 corresponding to the image data obtained by capturing the text written in the notebook N101 on the display unit 36 (FIG. 29E). ).

After that, the portable terminal 3 generates composite data by combining the image P10 and the image P11, displays an image corresponding to the composite data on the display unit 36 (FIG. 29F), and combines the image P10 and the image P11. Data is generated, and an image corresponding to the synthesized data, icon A201, icon A202 and icon A13 are displayed on the display unit 36 (FIG. 29G). The icon A201 accepts input of an instruction signal for transmitting an image file including composite data to a designated address. The icon S202 receives input of an instruction signal instructing printing by transmitting an image file to the image processing apparatus 5 .

[Processing of capture device]
Next, processing executed by the capture device 2A will be described. FIG. 30 is a flow chart showing an overview of the process executed by the capture device 2A. In FIG. 30, the capture device 2A further executes steps S6 and S7 in addition to steps S1 to S5 in FIG. 12 described above. Therefore, steps S6 and S7 will be described below.

In step S6, if the capture device 2A is set to the reproduction mode (step S6: Yes), the capture device 2A executes reproduction mode processing for reproducing the image data recorded in the recording section 27 (step S7). . Details of the reproduction mode processing will be described later. After step S7, the capture device 2A proceeds to step S3. On the other hand, if the capture device 2A is not set to the reproduction mode (step S6: No), the capture device 2A proceeds to step S3.

[Playback mode processing]
Next, the reproduction mode processing described in step S7 of FIG. 30 will be described. FIG. 31 is a flow chart showing an outline of reproduction mode processing executed by the capture device 2A.

As shown in FIG. 31, the display control unit 292 performs reproduction to display an image corresponding to the image data recorded in the image file recording unit 271 on the display unit 28 (step S61).

Subsequently, when an instruction signal to be transmitted from the operation unit 25 to the image processing device 5 is input (step S62: Yes), the communication control unit 294 causes the designated image file designated according to the operation of the operation unit 25 to be displayed as an image. The communication unit 26 is caused to transmit to the processing device 5 (step S63).

Thereafter, when an instruction signal instructing termination is input from the operation unit 25 (step S64: Yes), the capture device 2A returns to the main routine of FIG. On the other hand, if an instruction signal instructing termination has not been input from the operation unit 25 (step S64: No), the capture device 2A returns to step S61.

In step S62, if an instruction signal instructing termination has not been input from the operation unit 25 (step S64: No), the capture device 2A proceeds to step S64.

[Processing of image processing device]
Next, processing executed by the image processing device 5 will be described. FIG. 32 outlines the processing executed by the image processing device 5 .

As shown in FIG. 32, first, when there is a connection from the mobile terminal 3 (step S71: Yes) and an image file is received from the mobile terminal 3 (step S72: Yes), an operation is performed from the mobile terminal 3 or the operation unit 52. When there is a signal (step S73: Yes), the control unit 56 edits the image corresponding to the image data stored in the image file and adjusts the image quality based on the operation signal input from the mobile terminal 3 or the operation unit 52. is adjusted (step S74).

Subsequently, when an instruction signal instructing printing by the special output service is input from the mobile terminal 3 or the operation unit 52 (step S75: Yes), the image processing device 5 prints the image corresponding to the image data as a special print. Then, a special print is performed (step S76). Specifically, the image processing device 5 prints and outputs the image P100 in FIG. 27 and the image P200 in FIG. 28 described above.

Thereafter, when an instruction signal instructing termination is input from the mobile terminal 3 or the operation unit 52 (step S77: Yes), the image processing device 5 terminates this process. On the other hand, if the instruction signal instructing the end is not input from the portable terminal 3 or the operation unit 52 (step S77: No), the image processing device 5 returns to step S71.

In step S71, if there is no connection from the mobile terminal 3 (step S71: No) and if the image file has not been received from the mobile terminal 3 (step S72: No), the image processing device 5 terminates this process.

In step S73, when there is no operation signal from the portable terminal 3 (step S73: No), the image processing device 5 proceeds to step S75.

In step S75, if the instruction signal for instructing printing by the special output service is not input from the portable terminal 3 (step S75: No), the image processing device 5 prints and outputs the image corresponding to the image data. (step S78). After step S78, the image processing apparatus 5 proceeds to step S77.

According to the second embodiment described above, the image processing apparatus 5 prints and outputs the first image on a paper medium in which text information as processing information is associated with the first image. can grasp. Although we have explained in examples such as printing on paper media that is easy to hand over to people and traditional evidence management, the technology of this application can of course be displayed on electronic panels such as liquid crystal panels, organic EL panels, and electronic paper. It also assumes an application example of time.

(Other embodiments)
Various inventions can be formed by appropriately combining the plurality of components disclosed in the image processing systems according to the first and second embodiments of the present disclosure. For example, some components may be deleted from all the components described in the image processing system according to the embodiment of the present disclosure. Furthermore, the components described in the image processing system according to the embodiment of the present disclosure described above may be combined as appropriate.

Further, in the image processing system according to the first and second embodiments of the present disclosure, the above-described "unit" can be read as "means" or "circuit". For example, the control unit can be read as control means or a control circuit.

Further, the program to be executed by the information providing system according to the first and second embodiments of the present disclosure is file data in an installable format or an executable format, and can be downloaded from CD-ROM, flexible disk (FD), CD-R, It is provided by being recorded on a computer-readable recording medium such as a DVD (Digital Versatile Disk), a USB medium, or a flash memory.

Also, the program to be executed by the information providing system according to the first and second embodiments of the present disclosure is stored on a computer connected to a network such as the Internet, and is configured to be provided by being downloaded via the network. may

In addition, in the description of the flowcharts in this specification, expressions such as "first", "after", and "following" were used to clarify the context of the processing between steps. The required order of processing is not uniquely defined by those representations. That is, the order of processing in the flow charts described herein may be changed within a consistent range. In addition, the program is not limited to such a simple branching process, and branching may be performed by comprehensively judging more judgment items. In that case, artificial intelligence technology may be used in combination, such as performing machine learning while urging the user to perform manual operations and repeating learning. Further, it is also possible to learn the operation patterns performed by many experts, and perform deep learning by incorporating more complex conditions.

As described above, some of the embodiments of the present application have been described in detail with reference to the drawings. It is possible to carry out the present invention in other forms with modifications and improvements.

1, 1A... image processing system; 2, 2A... capture device; 3... mobile terminal; 4... server; 5... image processing device; 21, 311... optical system; 24, 32 Clock; 25, 33, 52 Operation section; 26, 34, 41, 51 Communication section; 35, 43, 55... Recording section; 28, 36, 54... Display section; 29, 37, 44, 56... Control section; 42, 53... Output section; 100... Network; 271,351 Image file recording section; 272 Audio file recording section; 273,352 Auxiliary data recording section; 274,353,431,551 Program recording section; 292,376 display control unit; 293,377 recording control unit; 294,378,443,563 communication control unit; 312 image sensor; Detector; 372 Calculator; 373 Image processor; 374,441,561 Synthesizer; 442,562 Output controller

Claims (11)

a communication unit that receives first image data obtained by imaging a natural object by a first device;
a display control unit that displays a first image corresponding to the first image data on a display screen of a display unit;
an imaging unit that captures a second image including processing information for displaying the first image data on a paper medium;
a calculation unit that calculates a pasting area for pasting the first image on the paper medium;
an image processing unit that resizes the display area of the first image based on the pasting area calculated by the calculation unit;
with
The imaging unit is
Continuously capturing written information in which any one of characters, figures, and symbols written on the paper medium is written,
The calculation unit
calculating the pasting area based on the handwritten information appearing in a live view image sequentially generated by the imaging unit;
image processing system. a communication unit that receives first image data generated by an external device;
an imaging unit configured to capture a second image including processing information for displaying the first image corresponding to the first image data on a paper medium;
a calculation unit that calculates a pasting area for pasting the first image on the paper medium;
an image processing unit that resizes the display area of the first image based on the pasting area calculated by the calculation unit;
with
The imaging unit is
Continuously capturing written information in which any one of characters, figures, and symbols written on the paper medium is written,
The calculation unit
calculating the pasting area based on the handwritten information appearing in a live view image sequentially generated by the imaging unit;
Information processing equipment. The information processing device according to claim 2,
The processing information is
Character information synthesized when printing the first image data,
Information processing equipment. The information processing device according to claim 3,
The processing information is
Pasting information for pasting a first image corresponding to the first image data on the paper medium,
Information processing equipment. The information processing device according to claim 4,
The pasting information includes text information, position information indicating a position for pasting the first image on the paper medium, and size information.
Information processing equipment. The information processing device according to claim 5 ,
further comprising a recording control unit that generates an image file containing the first image, the second image, and the processing information and records it in a recording unit;
Information processing equipment. The information processing device according to claim 6 ,
further comprising a display control unit for displaying the live view image on a display unit;
The display control unit superimposes the first image resized by the image processing unit on the handwriting information in the live view image and causes the display unit to display the first image.
Information processing equipment. An image processing method executed by an image processing system,
a communication step of receiving first image data obtained by photographing a natural object by a first device;
a display control step of displaying a first image corresponding to the first image data on a display screen of a display unit;
an imaging step of capturing a second image including processing information for displaying the first image data on a paper medium;
a calculation step of calculating a pasting area for pasting the first image on the paper medium;
an image processing step of resizing the display area of the first image based on the pasting area;
including
The imaging step includes
Continuously capturing written information in which any one of characters, figures, and symbols written on the paper medium is written,
The calculating step includes:
calculating the pasting area based on the handwritten information appearing in the live view image, which is the second image sequentially generated by the imaging step;
Information processing methods. An information processing method executed by an information processing device,
a communication step of receiving first image data generated by an external device;
an imaging step of capturing a second image including processing information for displaying the first image corresponding to the first image data on a paper medium;
a calculation step of calculating a pasting area for pasting the first image on the paper medium;
an image processing step of resizing the display area of the first image based on the pasting area;
including
The imaging step includes
Continuously capturing written information in which any one of characters, figures, and symbols written on the paper medium is written,
The calculating step includes:
calculating the pasting area based on the handwritten information appearing in the live view image, which is the second image sequentially generated by the imaging step;
Information processing methods. A program to be executed by an image processing system,
a communication step of receiving first image data obtained by photographing a natural object by a first device;
a display control step of displaying a first image corresponding to the first image data on a display screen of a display unit;
an imaging step of capturing a second image including processing information for displaying the first image data on a paper medium;
a calculation step of calculating a pasting area for pasting the first image on the paper medium;
an image processing step of resizing the display area of the first image based on the pasting area;
including
The imaging step includes
Continuously capturing written information in which any one of characters, figures, and symbols written on the paper medium is written,
The calculating step includes:
calculating the pasting area based on the handwritten information appearing in the live view image, which is the second image sequentially generated by the imaging step;
program. A program to be executed by an information processing device,
a communication step of receiving first image data generated by an external device;
an imaging step of capturing a second image including processing information for displaying the first image corresponding to the first image data on a paper medium;
a calculation step of calculating a pasting area for pasting the first image on the paper medium;
an image processing step of resizing the display area of the first image based on the pasting area;
including
The imaging step includes
Continuously capturing written information in which any one of characters, figures, and symbols written on the paper medium is written,
The calculating step includes:
calculating the pasting area based on the handwritten information appearing in the live view image, which is the second image sequentially generated by the imaging step;
program.

Images