JP7638413B2 - IMAGE DISPLAY SYSTEM, IMAGE DISPLAY METHOD, AND GOGGLE-TYPE DEVICE - Google Patents

Description

The present invention relates to an image display system , an image display method, and a goggle-type device capable of displaying stereoscopic images.

Conventionally, there are stereoscopic image display devices that display a stereoscopic image by having the user view two images with parallax with each eye (see, for example, Patent Document 1). For example, Patent Document 1 discloses that a smartphone is stored in a goggle device that can be worn by the user, and the user can view a stereoscopic image displayed on the display screen of the smartphone through the goggle device.

JP 2018-198651 A

However, the stereoscopic image display device disclosed in Patent Document 1 does not take into consideration the presentation of a user interface image that accepts touch operations by the user when displaying a stereoscopic image on the display screen, so there is room for improvement in the convenience of the method of presenting the user interface image.

Therefore, an object of the present invention is to provide an image display system , an image display method, and a goggle-type device that can improve the convenience of presenting a user interface image.

In order to achieve the above object, the present invention may employ, for example, the following configuration. When interpreting the claims, it is understood that the scope should be interpreted only by the claims, and in the event of a contradiction between the claims and this section, the claims shall take precedence.

One configuration example of the image display system of the present invention includes a display device having a display screen for displaying an image, and a goggle device on which the display device can be worn. The image display system includes a mode setting means, a first display control means, and a second display control means. The mode setting means sets the display device to a first display mode or a second display mode different from the first display mode. The first display control means causes the display screen to display a first image including a content image, which is a non-stereoscopic image, and a first user interface image in the first display mode. The second display control means causes the display screen to display a second image including a content image consisting of a left-eye image and a right-eye image having parallax therebetween, and a second user interface image corresponding to the first user interface image in the second display mode. The second display control means causes the second user interface image to be displayed in a position on the display screen different from the position on the display screen where the first user interface image is displayed in the first display mode in the second display mode.

According to the above, when switching display modes, the user interface image is displayed in a different position, which makes it possible to improve the convenience of presenting the user interface image.

The image display system further includes a detection means. The detection means detects whether the display device is attached to the goggle device, or whether it is in the process of being attached. In this case, when the display device is set to the first display mode, the mode setting means may switch the display device to the second display mode based on the detection result of the detection means.

As described above, the display mode is switched based on the state in which the display device is attached to the goggle device or the state in which it is in the process of being attached, making it possible to switch display modes seamlessly.

The display device may also include an illuminance sensor. In this case, the goggle device may also include a light blocking section. The light blocking section blocks light from reaching the illuminance sensor of the display device when the display device is attached to the goggle device or in the process of being attached to the goggle device. The detection means may detect whether the display device is attached to the goggle device or in the process of being attached based on the detection result by the illuminance sensor.

As a result of the above, it is possible to easily detect whether the display device is in an attached state or in a partially attached state based on the light-blocking state of the display device.

The display device may also include a touch panel provided on the display screen. In this case, the first user interface image displayed on the display screen may be capable of providing operational instructions in response to a touch operation on the touch panel. The second user interface image displayed on the display screen may be capable of providing operational instructions in response to a touch operation on the touch panel when the display device is attached to the goggle device.

Based on the above, it is possible to improve the convenience of presenting user interface images that accept touch operations.

The second display control means may also display, on the display screen, an image corresponding to the content image displayed as a non-stereoscopic image by the first display control means, as a content image consisting of an image for the left eye and an image for the right eye.

As a result of the above, it is possible to seamlessly switch between stereoscopic and non-stereoscopic images of the same content image.

In addition, in the second display mode, the second display control means may convert the content image displayed in the first display mode immediately before the second display mode is set to the second display mode into a stereoscopic image consisting of a left-eye image and a right-eye image, and display the stereoscopic image on the display screen.

As a result of the above, it is possible to seamlessly switch between stereoscopic and non-stereoscopic images of the same content image.

The second display control means may, in the second display mode, display an image for the left eye in a first region of the display screen, display an image for the right eye in a second region of the display screen that is different from the first region, and display a second user interface image in a third region of the display screen that is different from the first and second regions.

As a result of the above, it is possible to prevent the first and second areas for displaying a stereoscopic image from being soiled by touch operations, and it is also possible to prevent a finger for touching the second user interface image from coming into view while viewing a stereoscopic image.

The second display control means may also display, as the second user interface image, a user interface image that is substantially the same in function as the first user interface image but has a different shape.

As a result of the above, it is possible to display a user interface image with a shape suitable for operation and stereoscopic image display.

In addition, the second display control means may adjust the shape of the second user interface image to fit the shape of a third area of the display screen that is different from the first area of the display screen that displays an image for the left eye and the second area of the display screen that displays an image for the right eye in the second display mode, and display the second user interface image in the third area.

According to the above, it is possible to display a user interface image of an appropriate shape.

The second display control means may also set a third area above or below the display screen, sandwiched between the first area and the second area on the display screen.

As a result of the above, it is possible to display a user interface image that does not interfere with the stereoscopic image display.

The second display control means may also set a third area below the display screen between the first and second areas on the display screen.

As a result of the above, it is possible to display a user interface image that does not interfere with the stereoscopic image display and is easy to operate.

The goggle device may also have an opening that exposes at least a third region, which is a part of the display screen, to the outside when the display device is attached to the goggle device.

As a result of the above, touch operations to touch the user interface image are possible while the goggle device is being worn.

The opening may also be formed at a position that corresponds to the user's nose when the user wears the goggle device.

As a result of the above, openings can be formed without compromising light-blocking properties.

The first display control means may also display the first user interface image superimposed on the content image displayed on the display screen in the first display mode.

As a result of the above, it is possible to display a relatively large content image without being affected by the display of the user interface image.

The second display control means may also display the second user interface image on the display screen as a non-stereoscopic image in the second display mode.

The above makes it easier to touch the user interface image.

The display device may further include a display device side connection section that can be electrically connected to another device. In this case, the goggle device may include a goggle device side connection section that can be electrically connected to the display device side connection section. The detection means may detect whether the display device is attached to the goggle device, or whether it is in the process of being attached, in response to the display device side connection section and the goggle device side connection section being connected.

As a result of the above, it is possible to reliably detect when a display device has been attached to a goggle device, or when the display device is in the process of being attached.

The present invention may also be embodied in the form of an image display program, an image display method, and a display device.

According to the present invention, when switching display modes, user interface images are displayed in different positions, thereby improving the convenience of presenting user interface images.

A diagram showing the left controller 3 and the right controller 4 attached to the main unit 2. FIG. 1 shows an example of a state in which the left controller 3 and the right controller 4 are detached from the main unit 2. FIG. 6 is a six-sided view showing an example of the main unit 2. Six-sided diagram showing an example of the left controller 3 Six-sided diagram showing an example of the right controller 4 FIG. 2 is a block diagram showing an example of the internal configuration of the main unit 2. A block diagram showing an example of the internal configuration of the main unit 2, the left controller 3, and the right controller 4. FIG. 2 is a perspective view showing an example of the appearance of the goggle device 150. FIG. 1 is a front view showing an example of a state in which the main body device 2 is attached to the goggle device 150. FIG. 1 is a front view showing an example of a state of the main unit 2 attached to the goggle device 150. FIG. 13 is a diagram showing an example of the shape of a front contact portion 151b that comes into contact with a part of the front surface of the main unit 2; FIG. 1 is a diagram showing an example of the internal structure of a goggle device 150. FIG. 1 is a side view showing an example of a state of the main unit 2 attached to the goggle device 150. FIG. 1 is a diagram showing an example of a user viewing an image displayed on an image display system. FIG. 1 is a diagram showing an example of a user holding an image display system; FIG. 13 shows an example of an image displayed on the main unit 2 in a stereoscopic display mode and a non-stereoscopic display mode. FIG. 2 is a diagram showing an example of a data area set in the DRAM 85 of the main unit 2. A flowchart showing an example of a game process executed by the game system 1.

The image display system according to an example of this embodiment will be described below. The example of the image display system in this embodiment is configured by a game system 1 (the main unit 2 included in the game system 1 as a minimum configuration) and a goggle device 150. The example of the game system 1 includes a main unit (information processing device; in this embodiment, it functions as a game device main unit) 2, a left controller 3, and a right controller 4. The left controller 3 and the right controller 4 are each detachable from the main unit 2. In other words, the game system 1 can be used as an integrated device by attaching the left controller 3 and the right controller 4 to the main unit 2. The game system 1 can also be used by using the main unit 2, the left controller 3, and the right controller 4 separately (see FIG. 2). The hardware configuration of the game system 1 of this embodiment will be described below, and then the control of the game system 1 of this embodiment will be described.

Figure 1 is a diagram showing an example of a state in which a left controller 3 and a right controller 4 are attached to a main unit 2. As shown in Figure 1, the left controller 3 and the right controller 4 are each attached to the main unit 2 and integrated together. The main unit 2 is a device that executes various processes (e.g., game processes) in the game system 1. The main unit 2 includes a display 12. The left controller 3 and the right controller 4 are devices that include an operation unit that allows the user to perform input.

Figure 2 is a diagram showing an example of the state in which the left controller 3 and right controller 4 have been removed from the main unit 2. As shown in Figures 1 and 2, the left controller 3 and right controller 4 are detachable from the main unit 2. In the following, the left controller 3 and right controller 4 may be collectively referred to as "controller."

Figure 3 is a six-sided view showing an example of the main unit 2. As shown in Figure 3, the main unit 2 includes a generally plate-shaped housing 11. In this embodiment, the main surface of the housing 11 (in other words, the front surface, i.e., the surface on which the display 12 is provided) is generally rectangular.

The shape and size of the housing 11 are arbitrary. As an example, the housing 11 may be of a size that is portable. Furthermore, the main unit 2 alone or an integrated device in which the left controller 3 and right controller 4 are attached to the main unit 2 may be a portable device. Furthermore, the main unit 2 or the integrated device may be a handheld device. Furthermore, the main unit 2 or the integrated device may be a portable device.

As shown in FIG. 3, the main unit 2 includes a display 12 provided on the main surface of the housing 11. The display 12 displays images generated by the main unit 2. In this embodiment, the display 12 is a liquid crystal display (LCD). However, the display 12 may be any type of display device.

The main unit 2 also includes a touch panel 13 on the screen of the display 12. In this embodiment, the touch panel 13 is of a type that allows multi-touch input (e.g., a capacitive type). However, the touch panel 13 may be of any type, and may be of a type that allows single-touch input (e.g., a resistive film type).

The main unit 2 is provided with a speaker (i.e., speaker 88 shown in FIG. 6) inside the housing 11. As shown in FIG. 3, speaker holes 11a and 11b are formed on the main surface of the housing 11. The output sound of the speaker 88 is output from these speaker holes 11a and 11b, respectively.

The main unit 2 also includes a left side terminal 17, which is a terminal for the main unit 2 to communicate with the left controller 3 via a wired connection, and a right side terminal 21, which is a terminal for the main unit 2 to communicate with the right controller 4 via a wired connection.

As shown in FIG. 3, the main unit 2 includes a slot 23. The slot 23 is provided on the upper side of the housing 11. The slot 23 has a shape that allows a predetermined type of storage medium to be attached thereto. The predetermined type of storage medium is, for example, a storage medium (e.g., a dedicated memory card) dedicated to the game system 1 and the same type of information processing device. The predetermined type of storage medium is used, for example, to store data used by the main unit 2 (e.g., application save data, etc.) and/or programs executed by the main unit 2 (e.g., application programs, etc.). The main unit 2 also includes a power button 28.

The main unit 2 has a lower terminal 27. The lower terminal 27 is a terminal through which the main unit 2 communicates with the cradle. In this embodiment, the lower terminal 27 is a USB connector (more specifically, a female connector). When the all-in-one device or the main unit 2 alone is placed on the cradle, the game system 1 can display images generated and output by the main unit 2 on a stationary monitor. In this embodiment, the cradle also has a function of charging the all-in-one device or the main unit 2 alone that is placed on it. The cradle also has a function of a hub device (more specifically, a USB hub).

The main unit 2 includes an illuminance sensor 29. In this embodiment, the illuminance sensor 29 is provided on the lower part of the main surface of the housing 11, and detects the illuminance (brightness) of light incident from the main surface side of the housing 11. Depending on the illuminance of the light detected by the illuminance sensor 29, the display 12 can be adjusted to an appropriate brightness to display an image. In this embodiment, the detected illuminance is used to determine whether the main unit 2 is attached to the goggle device 150 described below.

Figure 4 is a six-sided view showing an example of the left controller 3. As shown in Figure 4, the left controller 3 includes a housing 31. In this embodiment, the housing 31 has a vertically long shape, that is, a shape that is long in the up-down direction (i.e., the y-axis direction shown in Figures 1 and 4). The left controller 3 can also be held in a vertically long orientation when removed from the main unit 2. The housing 31 has a shape and size that allows it to be held in one hand, particularly the left hand, when held in a vertically long orientation. The left controller 3 can also be held in a horizontally long orientation. When the left controller 3 is held in a horizontally long orientation, it may be held with both hands.

The left controller 3 is equipped with an analog stick 32. As shown in FIG. 4, the analog stick 32 is provided on the main surface of the housing 31. The analog stick 32 can be used as a direction input unit capable of inputting a direction. By tilting the analog stick 32, the user can input a direction according to the tilt direction (and input a magnitude according to the tilt angle). Note that instead of an analog stick, the left controller 3 may be equipped with a cross key or a slide stick capable of slide input as the direction input unit. Also, in this embodiment, input is possible by pressing the analog stick 32.

The left controller 3 is equipped with various operation buttons. The left controller 3 is equipped with four operation buttons 33 to 36 (specifically, a right button 33, a down button 34, an up button 35, and a left button 36) on the main surface of the housing 31. Furthermore, the left controller 3 is equipped with a record button 37 and a - (minus) button 47. The left controller 3 is equipped with a first L button 38 and a ZL button 39 on the upper left of the side of the housing 31. The left controller 3 is also equipped with a second L button 43 and a second R button 44 on the side of the housing 31 that is attached when the left controller 3 is attached to the main unit 2. These operation buttons are used to give instructions according to various programs (for example, an OS program or an application program) executed on the main unit 2.

The left controller 3 also includes a terminal 42 that enables the left controller 3 to communicate with the main unit 2 via a wired connection.

Figure 5 is a six-sided view showing an example of the right controller 4. As shown in Figure 5, the right controller 4 includes a housing 51. In this embodiment, the housing 51 has a vertically long shape, that is, a shape that is long in the up-down direction. The right controller 4 can also be held in a vertical orientation when removed from the main unit 2. When held in a vertical orientation, the housing 51 has a shape and size that allows it to be held in one hand, particularly the right hand. The right controller 4 can also be held in a horizontal orientation. When the right controller 4 is held in a horizontal orientation, it may be held with both hands.

The right controller 4, like the left controller 3, has an analog stick 52 as a direction input unit. In this embodiment, the analog stick 52 has the same configuration as the analog stick 32 of the left controller 3. The right controller 4 may also have a cross key or a slide stick capable of slide input, instead of an analog stick. The right controller 4, like the left controller 3, has four operation buttons 53 to 56 (specifically, an A button 53, a B button 54, an X button 55, and a Y button 56) on the main surface of the housing 51. The right controller 4 further has a + (plus) button 57 and a home button 58. The right controller 4 also has a first R button 60 and a ZR button 61 on the upper right of the side of the housing 51. The right controller 4 also has a second L button 65 and a second R button 66, like the left controller 3.

The right controller 4 also includes a terminal 64 for wired communication between the right controller 4 and the main unit 2.

Figure 6 is a block diagram showing an example of the internal configuration of the main unit 2. In addition to the configuration shown in Figure 3, the main unit 2 includes components 81-91, 97, and 98 shown in Figure 6. Some of these components 81-91, 97, and 98 may be mounted on an electronic circuit board as electronic components and stored in the housing 11.

The main unit 2 includes a processor 81. The processor 81 is an information processing unit that executes various information processing executed in the main unit 2, and may be composed of only a CPU (Central Processing Unit), for example, or may be composed of a SoC (System-on-a-chip) that includes multiple functions such as a CPU function and a GPU (Graphics Processing Unit) function. The processor 81 executes various information processing by executing an information processing program (for example, a game program) stored in a storage unit (specifically, an internal storage medium such as a flash memory 84, or an external storage medium inserted in the slot 23, etc.).

The main unit 2 includes a flash memory 84 and a dynamic random access memory (DRAM) 85 as examples of internal storage media built into the main unit 2. The flash memory 84 and the DRAM 85 are connected to the processor 81. The flash memory 84 is a memory used primarily to store various data (which may be programs) saved in the main unit 2. The DRAM 85 is a memory used to temporarily store various data used in information processing.

The main unit 2 includes a slot interface (hereinafter abbreviated as "I/F") 91. The slot I/F 91 is connected to the processor 81. The slot I/F 91 is connected to the slot 23, and reads and writes data from a specific type of storage medium (e.g., a dedicated memory card) inserted in the slot 23 in response to instructions from the processor 81.

The processor 81 reads and writes data from and to the flash memory 84, DRAM 85, and each of the above storage media as appropriate to perform the above information processing.

The main unit 2 includes a network communication unit 82. The network communication unit 82 is connected to the processor 81. The network communication unit 82 communicates with an external device via a network (specifically, wireless communication). In this embodiment, the network communication unit 82 connects to a wireless LAN and communicates with an external device using a method conforming to the Wi-Fi standard as a first communication mode. The network communication unit 82 also performs wireless communication with other main units 2 of the same type using a predetermined communication method (e.g., communication using a proprietary protocol or infrared communication) as a second communication mode. Note that the wireless communication using the second communication mode enables wireless communication with other main units 2 located within a closed local network area, and realizes a function that enables so-called "local communication" in which data is transmitted and received by directly communicating between multiple main units 2.

The main unit 2 includes a controller communication unit 83. The controller communication unit 83 is connected to the processor 81. The controller communication unit 83 performs wireless communication with the left controller 3 and/or right controller 4. Any communication method may be used between the main unit 2 and the left controller 3 and right controller 4, but in this embodiment, the controller communication unit 83 performs communication with the left controller 3 and right controller 4 according to the Bluetooth (registered trademark) standard.

The processor 81 is connected to the left terminal 17, the right terminal 21, and the lower terminal 27. When the processor 81 performs wired communication with the left controller 3, it transmits data to the left controller 3 via the left terminal 17 and receives operation data from the left controller 3 via the left terminal 17. When the processor 81 performs wired communication with the right controller 4, it transmits data to the right controller 4 via the right terminal 21 and receives operation data from the right controller 4 via the right terminal 21. When the processor 81 performs communication with the cradle, it transmits data to the cradle via the lower terminal 27. Thus, in this embodiment, the main unit 2 can perform both wired communication and wireless communication with the left controller 3 and the right controller 4. When the integrated device with the left controller 3 and the right controller 4 attached to the main unit 2 or the main unit 2 alone is attached to the cradle, the main unit 2 can output data (e.g., image data and audio data) to a stationary monitor or the like via the cradle.

Here, the main unit 2 can communicate with multiple left controllers 3 simultaneously (in other words, in parallel). The main unit 2 can also communicate with multiple right controllers 4 simultaneously (in other words, in parallel). Therefore, multiple users can simultaneously input to the main unit 2 using each set of left controllers 3 and right controllers 4. As an example, a first user can input to the main unit 2 using a first set of left controllers 3 and right controllers 4, while a second user can input to the main unit 2 using a second set of left controllers 3 and right controllers 4.

The main unit 2 includes a touch panel controller 86, which is a circuit that controls the touch panel 13. The touch panel controller 86 is connected between the touch panel 13 and the processor 81. Based on a signal from the touch panel 13, the touch panel controller 86 generates data indicating, for example, the position where a touch input was performed, and outputs the data to the processor 81.

The display 12 is also connected to the processor 81. The processor 81 displays images generated (e.g., by executing the above-mentioned information processing) and/or images acquired from the outside on the display 12.

The main unit 2 includes a codec circuit 87 and speakers (specifically, a left speaker and a right speaker) 88. The codec circuit 87 is connected to the speaker 88 and the audio input/output terminal 25, and is also connected to the processor 81. The codec circuit 87 is a circuit that controls the input and output of audio data to the speaker 88 and the audio input/output terminal 25.

The main unit 2 also includes an acceleration sensor 89. In this embodiment, the acceleration sensor 89 detects the magnitude of acceleration along three predetermined axes (e.g., the x, y and z axes shown in FIG. 1). The acceleration sensor 89 may also detect acceleration along one axis or two axes.

The main unit 2 also includes an angular velocity sensor 90. In this embodiment, the angular velocity sensor 90 detects angular velocity around three predetermined axes (e.g., the x, y and z axes shown in FIG. 1). Note that the angular velocity sensor 90 may also detect angular velocity around one axis or two axes.

The acceleration sensor 89 and the angular velocity sensor 90 are connected to the processor 81, and the detection results of the acceleration sensor 89 and the angular velocity sensor 90 are output to the processor 81. The processor 81 is capable of calculating information regarding the movement and/or attitude of the main unit 2 based on the detection results of the acceleration sensor 89 and the angular velocity sensor 90.

The illuminance sensor 29 is connected to the processor 81, and the detection result of the illuminance sensor 29 is output to the processor 81. The processor 81 is able to calculate information regarding the brightness around the main unit 2 based on the detection result of the illuminance sensor 29.

The main unit 2 includes a power control unit 97 and a battery 98. The power control unit 97 is connected to the battery 98 and the processor 81. Although not shown, the power control unit 97 is also connected to each part of the main unit 2 (specifically, each part that receives power from the battery 98, the left terminal 17, and the right terminal 21). The power control unit 97 controls the power supply from the battery 98 to each of the above-mentioned parts based on instructions from the processor 81.

The battery 98 is also connected to the lower terminal 27. When an external charging device (e.g., a cradle) is connected to the lower terminal 27 and power is supplied to the main unit 2 via the lower terminal 27, the supplied power is charged into the battery 98.

Figure 7 is a block diagram showing an example of the internal configuration of the main unit 2, the left controller 3, and the right controller 4. Note that details of the internal configuration of the main unit 2 are omitted in Figure 7 because they are shown in Figure 6.

The left controller 3 is equipped with a communication control unit 101 that communicates with the main unit 2. As shown in FIG. 7, the communication control unit 101 is connected to each component including the terminal 42. In this embodiment, the communication control unit 101 can communicate with the main unit 2 both by wired communication via the terminal 42 and by wireless communication not via the terminal 42. The communication control unit 101 controls the communication method that the left controller 3 uses with the main unit 2. That is, when the left controller 3 is attached to the main unit 2, the communication control unit 101 communicates with the main unit 2 via the terminal 42. Also, when the left controller 3 is removed from the main unit 2, the communication control unit 101 performs wireless communication with the main unit 2 (specifically, the controller communication unit 83). The wireless communication between the controller communication unit 83 and the communication control unit 101 is performed according to, for example, the Bluetooth (registered trademark) standard.

The left controller 3 also includes a memory 102, such as a flash memory. The communication control unit 101 is configured, for example, by a microcomputer (also called a microprocessor), and executes firmware stored in the memory 102 to perform various processes.

The left controller 3 is equipped with buttons 103 (specifically, buttons 33 to 39, 43, 44, and 47). The left controller 3 also has an analog stick (written as "stick" in FIG. 7) 32. Each button 103 and analog stick 32 repeatedly outputs information relating to operations performed on them to the communication control unit 101 at appropriate timing.

The communication control unit 101 acquires information related to the input (specifically, information related to the operation, or the detection results by the sensors) from each input unit (specifically, each button 103, analog stick 32, each sensor 104 and 105). The communication control unit 101 transmits operation data including the acquired information (or information obtained by performing a specified process on the acquired information) to the main unit 2. Note that the operation data is repeatedly transmitted once every specified time. Note that the interval at which the information related to the input is transmitted to the main unit 2 may or may not be the same for each input unit.

By transmitting the above operation data to the main unit 2, the main unit 2 can obtain the input made to the left controller 3. In other words, the main unit 2 can determine the operation of each button 103 and analog stick 32 based on the operation data.

The left controller 3 is equipped with a power supply unit 108. In this embodiment, the power supply unit 108 has a battery and a power control circuit. Although not shown, the power control circuit is connected to the battery and to each part of the left controller 3 (specifically, each part that receives power from the battery).

As shown in FIG. 7, the right controller 4 includes a communication control unit 111 that communicates with the main unit 2. The right controller 4 also includes a memory 112 that is connected to the communication control unit 111. The communication control unit 111 is connected to each component including the terminal 64. The communication control unit 111 and the memory 112 have the same functions as the communication control unit 101 and the memory 102 of the left controller 3. Therefore, the communication control unit 111 can communicate with the main unit 2 both by wired communication via the terminal 64 and by wireless communication (specifically, communication according to the Bluetooth (registered trademark) standard) that does not go through the terminal 64, and controls the method of communication that the right controller 4 uses with the main unit 2.

The right controller 4 has input sections similar to those of the left controller 3. Specifically, it has buttons 113 and an analog stick 52. Each of these input sections has the same function as the input sections of the left controller 3, and operates in the same way.

The right controller 4 is equipped with a power supply unit 118. The power supply unit 118 has the same functions as the power supply unit 108 of the left controller 3 and operates in the same manner.

Next, with reference to Figs. 8 to 15, the goggle device 150, which is an example of a device that constitutes an image display system by wearing the game system 1 (specifically, the main unit 2), will be described. Note that Fig. 8 is a perspective view showing an example of the appearance of the goggle device 150. Fig. 9 is a front view showing an example of the state in which the main unit 2 is worn on the goggle device 150. Fig. 10 is a front view showing an example of the state of the main unit 2 worn on the goggle device 150. Fig. 11 is a diagram showing an example of the shape of the front abutment portion 151b that comes into contact with a part of the front surface of the main unit 2. Fig. 12 is a diagram showing an example of the internal structure of the goggle device 150. Fig. 13 is a side view showing an example of the state of the main unit 2 worn on the goggle device 150. Fig. 14 is a diagram showing an example of the state of a user viewing an image displayed on the image display system. Fig. 15 is a diagram showing an example of the state of a user holding the image display system. Note that Figure 11 is a view from the same direction as Figure 10, with parts of the goggle device 150 (part of the main body 151, the lens frame member 152, the lens 153, and the plate-like member 154) removed (see-through) to clearly show the front contact portion 151b.

8 to 13, the goggle device 150 has a main body 151, a lens frame member 152, a lens 153, and a plate-shaped member 154. Here, the goggle device, which is an example of a device constituting an image display system, is not limited to the configuration described below as long as it is worn by fitting the user's face so as to cover the user's left and right eyes, and has a function of blocking at least a part of external light and a function of supporting the user's stereoscopic vision with a pair of lenses. For example, the goggle device may be used in various states, such as a type that fits the user's face by being held by the user, a type that fits the user's face by being fixed to the user's head, or a type in which the user looks into the device in a placed state. In addition, the goggle device may function as a so-called head-mounted display by being worn on the user's head with the main body device 2 attached, and may have a helmet-type shape in addition to a goggle-type shape. In the following description of the goggle device 150, a goggle-type goggle device that is held by the user and worn to fit the user's face is used.

The main body 151 has an attachment section that detachably fixes the main body 2 by contacting the front, back, top, and bottom of the main body 2. The attachment section has a front contact section that contacts a part of the front of the main body 2 (the surface on which the display 12 is provided), a back contact section that contacts the back of the main body 2, an upper contact section that contacts the top surface of the main body 2, and a lower contact section that contacts the bottom surface of the main body 2, and is formed in a square tube shape with both left and right sides open and with a gap formed between the front contact section, the back contact section, the upper contact section, and the lower contact section. The attachment section has both sides (the side on the x-axis positive direction side and the side on the x-axis negative direction side) open so that it can be attached from the left side or right side of the main body 2. When the goggle device 150 is attached to the main unit 2 through the opening on the right side as shown in FIG. 9, the front contact portion contacts the front of the main unit 2, the rear contact portion contacts the rear of the main unit 2, the upper contact portion contacts the upper surface of the main unit 2, and the lower contact portion contacts the lower surface of the main unit 2. As shown in FIG. 11, the front contact portion 151b of the main unit 151 has an opening formed therein so as not to obstruct at least the view of the images (left eye image and right eye image) displayed on the display 12 when the main unit 2 is attached.

As shown in Figures 9 and 13, the main body device 2 is attached to the goggle device 150 by sliding it from the left or right side of the main body device 2 into the gap in the mounting portion of the main body 151 along the front abutment portion, rear abutment portion, upper abutment portion, and lower abutment portion of the mounting portion. The main body device 2 can be removed from the goggle device 150 by sliding it left or right along the front abutment portion, rear abutment portion, upper abutment portion, and lower abutment portion of the mounting portion from the state where it is attached to the goggle device 150. In this way, the goggle device 150 can detachably mount the main body device 2.

The lens frame member 152 is fixed to the opening formed on the front part of the main body 151. The lens frame member 152 has a pair of lens frames that are opened so as not to obstruct the view of the display images (left eye image IML and right eye image IMR) displayed on the display 12 of the main body device 2 attached to the main body 151. In addition, the outer edges formed on the top, bottom, left and right of the lens frame member 152 are formed with bonding surfaces for bonding to the main body device 2, and a V-shaped recess is formed in the center of the outer edge formed on the lower side for contacting the nose of the user wearing the goggle device 150.

The lens 153 is composed of a pair of left-eye lenses 153L and right-eye lenses 153R, for example a pair of Fresnel lenses. The left-eye lenses 153L and right-eye lenses 153R are fitted into the lens frames of the lens frame member 152. Specifically, the left-eye lens 153L is fitted into one of the lens frames that is opened so as not to obstruct the field of view of the left-eye image IML displayed on the display 12 of the main unit device 2 attached to the main body 151, and when the user looks into the left-eye lens 153L with his left eye, the left-eye image IML can be seen. The right-eye lens 153R is fitted into the other lens frame that is opened so as not to obstruct the field of view of the right-eye image IMR displayed on the display 12 of the main unit device 2 attached to the main body 151, and when the user looks into the right-eye lens 153R with his right eye, the right-eye image IMR can be seen. In addition, the left eye lens 153L and the right eye lens 153R may be typically circular or elliptical magnifying lenses, and may be lenses that distort an image to allow a user to view it. For example, the left eye lens 153L may distort a left eye image IML (described later) displayed in a circular or elliptical shape in a direction opposite to the distortion of the image to allow the user to view it, and the right eye lens 153R may distort a right eye image IMR (described later) displayed in a circular or elliptical shape in a direction opposite to the distortion of the image to allow the user to view it, thereby providing a stereoscopic view of the image. In addition, the left eye lens 153L and the right eye lens 153R may be configured to be integrally formed.

The main body 151 has an abutment portion that is provided so as to protrude outward from the front side of the main body 151 so as to surround the outer edge of the lens frame member 152 in a rectangular cylindrical shape. The end face of the abutment portion that protrudes outward from the front side is disposed on the near side of the lens 153 when the lens 153 is viewed from outside the goggle device 150, and the end face is disposed on the near side (z-axis negative direction side) of the goggle device 150 when the main body device 2 is worn. The abutment portion of the main body 151 has an end face shape that fits the user's face (typically the area around the user's eyes) when the user looks into the goggle device 150 with the main body device 2 worn, and has the function of fixing the positional relationship between the user's eyes and the lens 153 by abutting the end face against the user's face.

In addition, the contact portion can block external light to the left eye lens 153L and the right eye lens 153R when viewing a stereoscopic image displayed on the display 12 using an image display system. This can improve the immersive feeling of the user viewing the stereoscopic image displayed on the display 12. Note that the light blocking by the contact portion does not have to completely block external light. For example, as shown in FIG. 15, a recess may be formed in a part of the contact portion configured in a cylindrical shape. Note that the recess of the contact portion illustrated in FIG. 15 is formed at a position below the midpoint of the left eye lens 153L and the right eye lens 153R, which is the position where the nose of the user viewing the stereoscopic image displayed on the display 12 will contact. In other words, the recess of the contact portion can prevent the contact portion from contacting the user's nose too strongly, and even if the light blocking ability is somewhat deteriorated, the discomfort caused by the contact portion contacting the nose can be reduced.

10, the plate-like member 154 is fixed inside the main body 151 between the lens frame member 152 and the display 12 when the main body device 2 is attached to the attachment portion of the main body 151. For example, the plate-like member 154 is arranged like a wall (hereinafter referred to as a first wall portion) that connects the V-shaped recess of the lens frame member 152 and the display 12 of the attached main body device 2. The space surrounded by the first wall portion exposes a part of the display 12 of the main body device 2 attached to the main body 151 to the outside, and functions as an operation window through which the user can touch and operate the part. Note that the first wall portion of the plate-like member 154 may be partially open as shown in FIG. 12.

12, the plate-like member 154 is, for example, vertically erected between the left-eye lens 153L and the right-eye lens 153R, and is arranged as a wall (hereinafter, referred to as the second wall) connecting the recess and the display 12 of the attached main unit 2. When the main unit 2 is attached to the main unit 151, the second wall is arranged to separate the image between the left-eye image IML and the right-eye image IMR displayed on the display 12, and functions as a partition wall provided between the left-eye image IML and the right-eye image IMR. The plate-like member 154 is provided by extending the first wall to the second wall, and the first wall and the second wall are formed by an integrated member.

10, 13, 14, and 15, the image display system is configured by mounting the main unit 2 to the goggle device 150. Here, in this embodiment, the main unit 2 is mounted so that the entire main unit 2 is covered by the goggle device 150. When the main unit 2 is mounted on the goggle device 150, it is possible to view only the left-eye image IML displayed in the left region of the display 12 through the left-eye lens 153L, and only the right-eye image IMR displayed in the right region of the display 12 through the right-eye lens 153R. Therefore, a user of the image display system can view the left-eye image IML and the right-eye image IMR by viewing the left-eye lens 153L with the left eye and the right-eye lens 153R with the right eye, so that a stereoscopic image with a sense of stereoscopic effect can be displayed to the user by displaying the left-eye image IML and the right-eye image IMR with parallax on the display 12.

As shown in Figures 14 and 15, when a user holds an image display system in which the main unit 2 is attached to the goggle device 150 and views a stereoscopic image displayed on the display 12, the user can hold the left side of the goggle device 150 to which the main unit 2 is attached with the left hand and the right side of the goggle device 150 with the right hand. By holding the left and right sides of the goggle device 150 in this way, the user can keep the main unit 2 stably attached.

In addition, even when the main body device 2 is attached to the goggle device 150, the image display system can touch and operate a part of the touch panel 13 provided on the screen of the display 12 (a third area of the display 12 described later) through the opening 154h formed by being surrounded by the first wall portion of the plate-like member 154. In addition, the image display system can calculate information regarding the movement and/or attitude of the main body device 2, i.e., the movement and/or attitude of the image display system including the goggle device 150, based on the detection results of the acceleration sensor 89 and/or the angular velocity sensor 90 provided in the main body device 2. Therefore, the image display system can calculate the attitude of the user's head looking into the goggle device 150 on which the main body device 2 is attached, based on the gravity direction. In addition, when the attitude or direction of the head of the user looking into the goggle device 150 on which the main body device 2 is attached changes, the image display system can calculate the changed direction and angle. Furthermore, when a user looking into the goggle device 150 on which the main body device 2 is attached applies a vibration like hitting the image display system, the image display system can detect the vibration. Therefore, when the main device 2 is attached to the goggle device 150 and the stereoscopic image displayed on the display 12 is viewed through the left eye lens 153L and the right eye lens 153R, a play style is realized that allows touch operations through the opening 154h, operations based on the attitude of the image display system based on the gravity direction, operations that change the attitude of the image display system, and operations that give vibration to the image display system.

When using the image display system in this embodiment, at least one of the left controller 3 and the right controller 4 detached from the main unit 2 may be used for operation. For example, when operating the image display system using the left controller 3, the user holds the goggle device 150 to which the main unit 2 is attached with the right hand while viewing the stereoscopic image displayed on the display 12, and holds the detached left controller 3 alone with the left hand to operate it. In this case, operation information operated on the left controller 3 and/or the right controller 4 detached from the main unit 2 is transmitted to the main unit 2 by wireless communication with the main unit 2. Specifically, operation information operated on the left controller 3 is wirelessly transmitted from the communication control unit 101 of the left controller 3 and received by the controller communication unit 83 of the main unit 2. Operation information operated on the right controller 4 is wirelessly transmitted from the communication control unit 111 of the right controller 4 and received by the controller communication unit 83 of the main unit 2.

In this way, in this embodiment, by attaching the main unit 2 to the goggle device 150, a portable image display system can be configured that the user can hold and view a stereoscopic image. In addition, in the image display system of this embodiment, the user views a stereoscopic image displayed on the display 12 of the main unit 2 while placing the user's face against the goggle device 150, so the positional relationship between the stereo speakers (left speaker 88L and right speaker 88R) provided in the main unit 2 and the user's ears is fixed, and the left and right speakers are placed near the user's ears. Therefore, the main unit 2 can output sound based on the positional relationship between the sound output device and the viewer's ears without forcing the use of earphones or speakers. For example, the main unit 2 can control a sound source using so-called stereophonic technology based on the positional relationship between the sound output device and the viewer's ears.

Next, images displayed on the main unit 2 will be described with reference to Figures 9, 10, and 16. Note that Figure 16 shows an example of an image displayed on the main unit 2 in the stereoscopic display mode and the non-stereoscopic display mode.

The image display system in this embodiment is set to either a stereoscopic display mode used when the main device 2 is attached to the goggle device 150 and an image displayed on the display 12 is stereoscopically viewed, or a non-stereoscopic display mode used when the main device 2 is detached from the goggle device 150 and an image displayed on the display 12 is viewed directly for non-stereoscopic viewing. The image display system then displays an image corresponding to the set mode on the display 12 of the main device 2. Here, the stereoscopic image for stereoscopic viewing may be one that the user views stereoscopically by viewing a right-eye image and a left-eye image with a parallax between them with the right and left eyes. In this case, the non-stereoscopic image that is viewed non-stereoscopically is an image other than the above-mentioned two-image display (stereoscopic display), and may typically be one in which the user views a single image with the right and left eyes. In this embodiment, the non-stereoscopic display mode is used as an example of the first display mode. In this embodiment, the stereoscopic display mode is used as an example of the second display mode.

In the stereoscopic display mode, the image display system configures the content image to be displayed (for example, an image for displaying a part of a virtual space or a real space) with a left-eye image IML and a right-eye image IMR having a mutual parallax, displays the left-eye image IML in the left region of the display 12, and displays the right-eye image IML in the right region of the display 12. Specifically, as shown in FIG. 9, in the stereoscopic display mode, the left-eye image IML is a substantially elliptical region viewable by the left-eye lens 153L when the main unit 2 is attached to the goggle device 150, and is displayed in a first region that is a part of the left region of the display 12. In addition, in the stereoscopic display mode, the right-eye image IMR is a substantially elliptical region viewable by the right-eye lens 153R when the main unit 2 is attached to the goggle device 150, and is displayed in a second region that is a part of the right region of the display 12.

As described above, when the main device 2 is attached to the goggle device 150, the second wall of the plate-like member 154 is disposed between the left-eye image IML displayed in the first area of the display 12 and the right-eye image IMR displayed in the second area. Therefore, the left-eye image IML and the right-eye image IMR are separated by the second wall of the plate-like member 154 as a partition, so that it is possible to prevent the right-eye image IMR from being viewed through the left-eye lens 153L, or the left-eye image IML from being viewed through the right-eye lens 153R.

As an example, images of the virtual space seen from a pair of virtual cameras (left virtual camera and right virtual camera) arranged in the virtual space with a parallax between them are generated as a left eye image IML and a right eye image IMR, respectively. The pair of virtual cameras are arranged in the virtual space so as to correspond to the attitude of the main unit 2 based on the direction of gravity in the real space. The pair of virtual cameras change their attitude in the virtual space so as to correspond to the change in attitude of the main unit 2 in the real space, and the line of sight of the virtual camera is controlled according to the attitude of the main unit 2. As a result, a user wearing the image display system can change the display range of the virtual space viewed stereoscopically by changing the attitude of the main unit 2 (image display system) as if looking around, and can look around the virtual space viewed stereoscopically, so that the user can experience being in the location of the virtual camera. In this embodiment, the main unit 2 matches the direction of gravitational acceleration acting on the main unit 2 with the direction of gravity in the virtual space acting on the virtual camera, and matches the amount of change in the attitude of the main unit 2 with the amount of change in the line of sight of the virtual camera. This increases the realism of the operation of looking around a stereoscopic virtual space depending on the orientation of the main unit 2.

The image display system also displays on the display 12 a user interface image IMU for receiving touch operations on the touch panel 13 of the main unit 2. For example, the user interface image IMUa displayed in the stereoscopic display mode is displayed in a display area of the display 12 that can be touched through the opening 154h of the goggle device 150. For example, as described above, the opening 154h is formed surrounded by the first wall of the plate-like member 154, and enables touch operations to a part of the display 12 of the main unit 2 attached to the goggle device 150 (specifically, an area near the center of the lower part of the display 12) from a V-shaped recess in the lens frame member 152 that abuts against the user's nose. As an example, as shown in FIG. 9, the opening 154h enables touch operations to a third area set below the display 12 between the first area and the second area of the display 12, even when the main unit 2 is attached to the goggle device 150.

For example, in the stereoscopic display mode, two user interface images IMUa1 and IMUa2 are displayed in the third area of the display 12. As an example, the user interface image IMUa1 is an operation icon that, when the display position is touched via the touch panel 13, is an operation icon that is an operation instruction to retry the game from the beginning. The user interface image IMUa2 is an operation icon that, when the display position is touched via the touch panel 13, is an operation icon that is an operation instruction to end the game. The two user interface images IMUa1 and IMUa2 are displayed in the third area of the display 12 side by side in a size that fits the shape of the third area. This allows the user to touch one of the two user interface images IMUa1 and IMUa2 through the opening 154h, thereby enabling multiple operation instructions by touch operation even when the main unit 2 is attached to the goggle device 150. The two user interface images IMUa1 and IMUa2 may be displayed near the third area that allows touch operation by exposing a portion of the display 12 to the outside. That is, a portion of the two user interface images IMUa1 and/or IMUa2 may be displayed outside the third area. In this embodiment, the user interface image IMUa is used as an example of the second user interface image.

The two user interface images IMUa1 and IMUa2 are images displayed in the stereoscopic display mode, but are displayed in a third area on the display 12 of the main unit 2 attached to the goggle device 150, which is outside the first area visible to the user's left eye and the second area visible to the user's right eye. Therefore, the user interface images IMUa1 and IMUa2 displayed in the third area are displayed outside the field of view of the user viewing through the goggle device 150, and are not composed of two images with parallax between them, so they are displayed as non-stereoscopic images that cannot be viewed stereoscopically. In addition, since the user interface images IMUa1 and IMUa2 that are the subject of touch operations are displayed outside the first and second areas for displaying stereoscopic images, the first and second areas are less likely to be touched. Therefore, it is possible to prevent the first and second areas for displaying stereoscopic images from being soiled by touch operations on the display 12, and it is possible to prevent fingers for touch operations from entering the field of view when viewing stereoscopic images.

In another embodiment, the user interface images IMUa1 and IMUa2 may be displayed on the display 12 as stereoscopic images that can be viewed stereoscopically in the stereoscopic display mode. In this case, the user interface images IMUa1 and IMUa2 are displayed on the display 12 as stereoscopic images by being composed of two images each having a parallax between them, and typically, one image for stereoscopic viewing is displayed in a portion of the first area, and the other image for stereoscopic viewing is displayed in a portion of the second area.

As shown in FIG. 16, in the non-stereoscopic display mode, the image display system configures the content image to be displayed as a single image IMS, which is a non-stereoscopic image, and displays the single image IMS over the entire display area of the display 12, as an example.

As an example, an image of the virtual space seen from a single virtual camera arranged in the virtual space is generated as a single image IMS. The single virtual camera is arranged in the virtual space so as to correspond to the attitude of the main unit 2 based on the direction of gravity in the real space. The single virtual camera changes its attitude in the virtual space so as to correspond to the change in attitude of the main unit 2 in the real space, and controls the line of sight of the virtual camera according to the attitude of the main unit 2. As a result, a user holding the main unit 2 detached from the goggle device 150 can change the display range of the virtual space displayed on the display 12 by changing the attitude of the main unit 2 as if looking around, and can look around the virtual space, so that the user can have an experience as if they were actually at the location of the virtual camera. Note that in this embodiment, even in the non-stereoscopic display mode, the main unit 2 matches the direction of the gravitational acceleration acting on the main unit 2 with the direction of gravity in the virtual space acting on the virtual camera, and matches the amount of change in the attitude of the main unit 2 with the amount of change in the line of sight of the virtual camera. As a result, the reality is increased even in the operation of looking around the non-stereoscopic virtual space depending on the attitude of the main unit 2. In this embodiment, a single image IMS is used as an example of a non-stereoscopic image.

The user interface image IMUb displayed in the non-stereoscopic display mode is displayed on the display 12, for example, by superimposing it on the content image (single image IMS) displayed on the display 12. For example, as shown in FIG. 16, even in the non-stereoscopic display mode, two user interface images IMUb1 and IMUb2 are displayed on the display 12. As an example, the user interface image IMUb1 is an image corresponding to the user interface image IMUa1, and is an operation icon that becomes an operation instruction to retry the game from the beginning when the display position is touched via the touch panel 13. The user interface image IMUb2 is an image corresponding to the user interface image IMUa2, and is an operation icon that becomes an operation instruction to end the game when the display position is touched via the touch panel 13. Here, the image corresponding to the user interface image IMUa indicates that the design and/or size are different but the function is substantially the same as the user interface image IMUa (for example, the operation instruction content by touch operation is the same). Note that the image corresponding to the user interface image IMUa may not only be substantially identical in function to the user interface image IMUa displayed in the non-stereoscopic display mode, but may also be the same in design and size, i.e., completely identical. Note that in this embodiment, the user interface image IMUb is used as an example of the first user interface image.

The two user interface images IMUb1 and IMUb2 are displayed in corner regions (e.g., the upper left corner region and the upper right corner region) of the display 12 that are different from the third region. Note that the two user interface images IMUb1 and IMUb2 are not limited by the region in which touch operations are possible, and therefore can be enlarged and displayed more than the user interface image IMUa displayed in the stereoscopic display mode, and can be displayed in a size and shape that is easy for the user to touch, and in a position where the visibility of the content image (single image IMS) is not likely to be impaired by touch operations.

The image display system in this embodiment can automatically switch between the stereoscopic display mode and the non-stereoscopic display mode based on the detection result of whether the main device 2 is attached to the goggle device 150 or is in the middle of being attached. For example, the main device 2 is provided with an illuminance sensor 29 that detects the illuminance (brightness) of light incident from the main surface side of the housing 11, and based on the illuminance detection result by the illuminance sensor 29, it can detect whether the main device 2 is attached to the goggle device 150 or is in the middle of being attached. Specifically, when the main device 2 is attached to the goggle device 150 or is in the middle of being attached, the illuminance detected by the illuminance sensor 29 becomes dark, so that it is possible to detect whether the main device 2 is attached to the goggle device 150 or is in the middle of being attached by setting a threshold value that can detect the darkened illuminance and detecting whether it is equal to or greater than the threshold value. Here, the "attached state" detected by the main device 2 based on the illuminance detection result by the illuminance sensor 29 means that the main device 2 is completely attached to the goggle device 150. Furthermore, the "partially attached state" detected by the main unit 2 based on the illuminance detection result by the illuminance sensor 29 means that the main unit 2 is in a state prior to being completely attached to the goggle device 150.

There are several possible ways in which the illuminance sensor 29 can detect whether the main device 2 is in a worn state or in a partially worn state. As a first example, as shown in FIG. 11, in the mounting portion of the goggle device 150 that detachably fixes the main device 2, if the front contact portion 151b of the main body 151 that comes into contact with a part of the front surface (the surface on which the display 12 is provided) of the main device 2 in the worn state is formed to cover the light receiving surface and light receiving hole of the illuminance sensor 29, the front contact portion 151b covers the light receiving surface and light receiving hole of the illuminance sensor 29, thereby detecting whether the main device 2 is in a worn state or in a partially worn state on the goggle device 150. As an example, if the light receiving surface or light receiving hole of the illuminance sensor 29 is provided at a corner of the main unit 2, when the main unit 2 is inserted into the mounting portion of the goggle device 150 from the corner side, the light receiving surface or light receiving hole is covered by the front abutment portion 151b at the beginning of the mounting operation, so that it becomes possible to detect that the main unit 2 is "in the middle of being mounted" based on the illuminance detection result by the illuminance sensor 29. Also, when the main unit 2 is inserted into the mounting portion of the goggle device 150 from the side opposite to the corner where the light receiving surface or light receiving hole of the illuminance sensor 29 is provided, the light receiving surface or light receiving hole is covered by the front abutment portion 151b at the end of the mounting operation, so that it becomes possible to detect that the main unit 2 is "in the mounted state" based on the illuminance detection result by the illuminance sensor 29. As another example, if the light receiving surface or light receiving hole of the illuminance sensor 29 is provided on the lower center side of the main unit 2, when the main unit 2 is inserted into the mounting portion of the goggle device 150, the light receiving surface or light receiving hole will be covered by the front contact portion 151b in the early to middle stages of the mounting operation regardless of the direction from which the main unit 2 is inserted, so it is possible to detect that the main unit 2 is "in the middle of being mounted" based on the illuminance detection result by the illuminance sensor 29. Note that in the first example above, the front contact portion 151b of the main body 151 is used as an example of a light blocking portion.

As a second example, even if the front contact portion 151b is not formed to cover the light receiving surface or light receiving hole of the illuminance sensor 29, the relative darkness inside the main body 151 is utilized to detect whether the main body device 2 is in a state of being attached to the goggle device 150 or in a state of being in the middle of being attached. For example, a state in which external light is incident on the main body 151 of the goggle device 150 through the left eye lens 153L and the right eye lens 153R is considered, and even in this state, the wearing state of the main body device 2 is detected by detecting that the illuminance is darker than the outside of the main body 151. As an example, if the light receiving surface or light receiving hole of the illuminance sensor 29 is provided at a corner of the main body device 2, when the main body device 2 is inserted into the mounting portion of the goggle device 150 from the corner side, the light receiving surface or light receiving hole is inserted inside the main body 151 at the beginning of the mounting operation, so that it is possible to detect that the main body device 2 is in a "state of being in the middle of being attached" based on the illuminance detection result by the illuminance sensor 29. In addition, when the main unit 2 is inserted into the goggle device 150 from the side opposite to the corner where the light receiving surface and light receiving hole of the illuminance sensor 29 are provided, the light receiving surface and light receiving hole are inserted into the main unit 150 at the end of the wearing operation, so that it is possible to detect that the main unit 2 is "in a wearing state" based on the illuminance detection result by the illuminance sensor 29. As another example, if the light receiving surface and light receiving hole of the illuminance sensor 29 are provided on the lower center side of the main unit 2, when the main unit 2 is inserted into the goggle device 150, the light receiving surface and light receiving hole are inserted into the main unit 151 at the beginning to middle of the wearing operation regardless of the direction from which the main unit 2 is inserted, so that it is possible to detect that the main unit 2 is "in the middle of wearing" based on the illuminance detection result by the illuminance sensor 29. In the second example, the main unit 151 is used as another example of a light blocking part.

As a third example, even if the front contact portion 151b is not formed to cover the light receiving surface or light receiving hole of the illuminance sensor 29, the fact that the main unit 2 is attached to the goggle device 150 is detected by utilizing the fact that the inside of the main unit 151 becomes darker when the user wears the goggle device 150 to which the main unit 2 is attached. For example, when the user is not wearing the goggle device 150, it is considered that external light enters the main unit 151 of the goggle device 150 through the left eye lens 153L and the right eye lens 153R. In this case, when the user wears the goggle device 150, the external light entering through the left eye lens 153L and the right eye lens 153R is blocked, and by detecting that the illuminance inside the main unit 151 becomes darker, it is possible to detect the wearing state of the main unit 2. As an example, when light incident on the light receiving surface or light receiving hole of the illuminance sensor 29 disposed in the main body 151 of the goggle device 150 is blocked by the main body 151 and the user's face, it is possible to detect that the main body device 2 is "in a worn state" based on the illuminance detection result by the illuminance sensor 29. Furthermore, in this example, it is possible to detect not only whether the main body device 2 is in a worn state on the goggle device 150, but also whether the goggle device 150 to which the main body device 2 is attached is being worn by the user. Note that in the above third example, the main body 151 is used as another example of a light blocking section.

In the image display system of the present embodiment, when it is determined that the main body device 2 is not attached to the goggle device 150, the display mode of the main body device 2 is set to the non-stereoscopic display mode. On the other hand, when it is determined that the main body device 2 has changed from a non-attached state to an attached state on the goggle device 150 in the main body device 2 set to the non-stereoscopic display mode, the main body device 2 changes the content image of the non-stereoscopic image being displayed to a stereoscopic image, thereby displaying the same content image on the display 12. As an example, when a single virtual camera is set in the virtual space to display a single image IMS in the non-stereoscopic display mode and a virtual space image is generated, the main body device 2 changes the position and line of sight of the single virtual camera to a pair of virtual cameras (left virtual camera and right virtual camera) having parallax with each other without changing the position and line of sight of the single virtual camera, and sets the virtual cameras for displaying the left eye image IML and the right eye image IMR, thereby switching to the generation of the virtual space image in the stereoscopic display mode. Furthermore, when it is determined that the main device 2 has changed from a state where the main device 2 is attached to the goggle device 150 to a state where the main device 2 is not attached to the goggle device 150 in the stereoscopic display mode, the main device 2 changes the content image of the displayed stereoscopic image to a non-stereoscopic image, thereby displaying the same content image corresponding to the content image of the stereoscopic image as a non-stereoscopic image on the display 12. As an example, when a pair of virtual cameras is set in the virtual space to display the left eye image IML and the right eye image IMR in the stereoscopic display mode and a virtual space image is generated, the main device 2 changes the pair of virtual cameras to a single virtual camera without changing the position and line of sight direction of the pair of virtual cameras, and sets a virtual camera for displaying the single image IMS, thereby switching to the generation of a virtual space image in the non-stereoscopic display mode. In this way, the content image of a non-stereoscopic image (e.g., a virtual space image) corresponding to the content image of a stereoscopic image (e.g., a virtual space image), or the content image of a stereoscopic image (e.g., a virtual space image) corresponding to the content image of a non-stereoscopic image (e.g., a virtual space image) indicates that the only difference in either case is whether the image is a stereoscopic image or a non-stereoscopic image. However, the display range of a non-stereoscopic image content image corresponding to a stereoscopic image content image, or a stereoscopic image content image corresponding to a non-stereoscopic image content image, may be different, and typically the display range of a stereoscopic image content image may be narrower than that of a non-stereoscopic image content image.

Furthermore, when the display mode is switched, the image display system in this embodiment changes the size, shape, and position of the user interface image IMU and displays it on the display 12. For example, in a case where the main body device 2 is set to a non-stereoscopic display mode, when it is determined that the main body device 2 has changed from a non-attached state to an attached state on the goggle device 150, the main body device 2 changes the shape of the user interface images IMUb1 and IMUb2, which are displayed overlapping with the content image in the corner area of the display 12, to user interface images IMUa1 and IMUa2, and moves the display position into the third area of the display 12, thereby displaying a user interface image IMU with the same function. Furthermore, when the main unit 2 is set to the stereoscopic display mode and it is determined that the main unit 2 has changed from a state in which it is attached to the goggle device 150 to a state in which it is not attached to the goggle device 150, the main unit 2 changes the shape of the user interface images IMUa1 and IMUa2 displayed in the third area of the display 12 to user interface images IMUb1 and IMUb2, and moves the display position so that it overlaps with the content image in the corner area of the display 12, thereby displaying the user interface image IMU with the same function.

In the above embodiment, an example was used in which the main device 2 is detected as being attached to the goggle device 150 or is in the middle of being attached based on the illuminance detection result by the illuminance sensor 29, but the main device 2 may be detected as being attached to the goggle device 150 or is in the middle of being attached based on other detection results. As an example, the main device 2 may be detected as being attached to the goggle device 150 or is in the middle of being attached based on a detection result of a connection terminal provided on the main device 2 and a connection terminal provided on the goggle device 150 being electrically connected by being in the attached state or the middle of being attached, or a detection result of a predetermined switch mechanism provided on the main device 2 being turned on or off by being in the attached state or the middle of being attached. As another example, the main device 2 may be detected as being attached to the goggle device 150 or is in the middle of being attached by determining whether a predetermined image is captured or not, or by determining whether the captured luminance is equal to or greater than a threshold value, based on the imaging result by the imaging means (imaging element) provided on the main device 2. As another example, when the main device 2 is attached to the goggle device 150 or is in the process of being attached to the goggle device 150, the user may be prompted to perform a specific operation, and based on the execution of the specific operation, it may be possible to detect whether the main device 2 is attached to the goggle device 150 or is in the process of being attached to the goggle device 150.

In the above embodiment, the third area is set below the center of the display 12 sandwiched between the first area and the second area on the display 12, and touch operation on the third area is enabled, but the third area may be set in another area on the display 12. As a first example, the third area may be set above the display 12, above the center of the display 12 sandwiched between the first area and the second area on the display 12. As a second example, the third area may be set above (i.e., the upper left corner area of the display 12) or below (i.e., the lower left corner area of the display 12) between the first area and the left edge of the display 12. As a third example, the third area may be set above (i.e., the upper right corner area of the display 12) or below (i.e., the lower right corner area of the display 12) between the second area and the right edge of the display 12 on the display 12. Regardless of the region in which the third region is set, a user interface image IMUa that matches the shape of the third region is displayed in the third region, and an opening 154h that allows touch operation within the third region is formed in the goggle device 150, thereby enabling the same operation as described above. Note that when the third region is set between the first region and the second region that are sandwiched by the first region and the second region, the third region may be shifted to the left or right from the midpoint between the first region and the second region.

In the above-described embodiment, the function of the user interface image is, as an example, what is displayed on the display 12 to accept touch operations on the touch panel 13, but it may be something that does not accept touch operations. For example, the user interface image may be text information or icons that do not allow touch operations but present some information to the user. As an example, the user interface image may be an image showing instructions corresponding to button operations, stick operations, operations to move the controller body, etc. on the left controller 3 or right controller 4.

The left eye image IML and the right eye image IMR may also be displayed outside the display area of the display 12 visible through the left eye lens 153L and the right eye lens 153R (typically outside the first area and/or outside the second area), and a portion of them may also be displayed within the third area where touch operations are possible. The left eye image IML and the right eye image IMR may also be displayed in an area smaller than the display area of the display 12 visible through the left eye lens 153L and the right eye lens 153R (typically the first area and/or the second area).

Next, an example of a specific process executed by the game system 1 in this embodiment will be described with reference to Figures 17 and 18. Figure 17 is a diagram showing an example of a data area set in the DRAM 85 of the main unit 2 in this embodiment. In addition to the data shown in Figure 17, the DRAM 85 also stores data used in other processes, but detailed descriptions will be omitted.

The program storage area of the DRAM 85 stores various programs Pa executed by the game system 1. In this embodiment, the various programs Pa include a communication program for wireless communication between the left controller 3 and the right controller 4 described above, and an application program for performing information processing (e.g., game processing) based on data acquired from the operation unit (left controller 3, right controller 4, touch panel 13, acceleration sensor 89, angular velocity sensor 90) and the illuminance sensor 29. The various programs Pa may be stored in advance in the flash memory 84, or may be acquired from a storage medium (e.g., a predetermined type of storage medium attached to the slot 23) that is detachable from the game system 1 and stored in the DRAM 85, or may be acquired from another device via a network such as the Internet and stored in the DRAM 85. The processor 81 executes the various programs Pa stored in the DRAM 85.

The data storage area of the DRAM 85 also stores various data used in communication processing, information processing, and other processing executed in the game system 1. In this embodiment, the DRAM 85 stores operation data Da, angular velocity data Db, acceleration data Dc, illuminance data Dd, attitude data De, operation object data Df, virtual camera data Dg, left eye virtual space image data Dh, right eye virtual space image data Di, stereoscopic UI (user interface) image data Dj, non-stereoscopic virtual space image data Dk, non-stereoscopic UI (user interface) image data Dm, and image data Dn, etc.

The operation data Da is operation data acquired appropriately from the left controller 3 and/or the right controller 4 and the touch panel 13. As described above, the operation data transmitted from the left controller 3 and/or the right controller 4 includes information on input from each input unit (specifically, each button, analog stick, each sensor) (specifically, information on the operation and detection results by each sensor). In this embodiment, the operation data is transmitted from the left controller 3 and/or the right controller 4 at a predetermined cycle by wireless communication, and the operation data Da is updated appropriately using the received operation data. The update cycle of the operation data Da may be updated every frame, which is the cycle of the process executed by the game system 1 described later, or may be updated every cycle in which the operation data is transmitted by the wireless communication. Furthermore, the operation data indicating that the touch panel 13 has been operated is acquired every cycle of the process, and is stored in the operation data Da and updated in response to the acquisition.

The angular velocity data Db is data indicating the angular velocity occurring in the main unit 2 detected by the angular velocity sensor 90. For example, the angular velocity data Db includes data indicating the angular velocity occurring in the main unit 2 around the x, y and z axes.

The acceleration data Dc is data indicating the acceleration occurring in the main unit 2 detected by the acceleration sensor 89. For example, the acceleration data Dc includes data indicating the acceleration occurring in the main unit 2 in the x, y and z axis directions.

The illuminance data Dd is data indicating the illuminance around the main unit 2 detected by the illuminance sensor 29.

The attitude data De is data that indicates the attitude of the main unit 2 in real space. As an example, the attitude data De includes data that indicates an attitude that is used as a reference for a gravity vector that indicates the gravitational acceleration occurring in the main unit 2, and data that indicates a change in attitude of the main unit 2.

Operation object data Df is data that indicates the position, direction, posture, and movement of the object operated by the user in virtual space.

The virtual camera data Dg is data that indicates the position, direction, viewing angle, magnification, etc. of a virtual camera (a pair of left and right virtual cameras in stereoscopic display mode, or a single virtual camera in non-stereoscopic display mode) set in the virtual space.

The left-eye virtual space image data Dh is data for generating the left-eye image IML in the stereoscopic display mode. The right-eye virtual space image data Di is data for generating the right-eye image IMR in the stereoscopic display mode. The stereoscopic UI image data Dj is data that indicates the position, shape, size, etc. of the user interface image IMUa in the stereoscopic display mode.

Non-stereoscopic virtual space image data Dk is data for generating a single image IMS in a non-stereoscopic display mode. Non-stereoscopic UI image data Dm is data that indicates the position, shape, size, etc. of the user interface image IMUb in a non-stereoscopic display mode.

Image data Dn is data for displaying images (e.g., images of virtual objects, user interface images, information images, field images, background images, etc.) on the display screen during the game.

Next, a detailed example of information processing (game processing) in this embodiment will be described with reference to FIG. 18. FIG. 18 is a flow chart showing an example of game processing executed by the game system 1. In this embodiment, the series of processes shown in FIG. 18 are performed by the processor 81 executing a communication program and a predetermined application program (game program) included in the various programs Pa. In addition, the timing at which the game processing shown in FIG. 18 is started is arbitrary.

Note that the processing of each step in the flowchart shown in FIG. 18 is merely an example, and the order of processing each step may be changed, or another process may be executed in addition to (or instead of) the processing of each step, as long as the same result is obtained. In addition, in this embodiment, the processing of each step in the above flowchart is described as being executed by the processor 81, but the processing of some steps in the above flowchart may be executed by a processor other than the processor 81 or a dedicated circuit. In addition, some of the processing executed in the main unit 2 may be executed by another information processing device that can communicate with the main unit 2 (for example, a server that can communicate with the main unit 2 via a network). In other words, each process shown in FIG. 18 may be executed by multiple information processing devices including the main unit 2 working together.

18, the processor 81 performs initial settings in the game processing (step S200) and proceeds to the next step. For example, in the initial settings, the processor 81 initializes parameters for performing the processing described below. As an example, the processor 81 uses the acceleration data stored in the acceleration data Dc to calculate the direction of the gravity vector of the gravitational acceleration acting on the main unit 2, sets the initial attitude of the main unit 2 based on the direction of the gravity vector, and updates the attitude data De. The processor 81 also sets the initial attitude of the virtual camera in the virtual space to be the same as the relationship between the direction of the gravity vector of the gravitational acceleration acting on the main unit 2 and the x, y, and z-axis directions of the main unit 2, and updates the virtual camera data Dg. Here, being in the same direction as the x, y, and z-axes of the main unit 2 means that the positive z-axis direction (screen depth direction) based on the gravitational acceleration direction in the real space is the same as the line of sight direction of the virtual camera based on the gravitational acceleration direction in the virtual space, and the positive x-axis direction (screen left direction) based on the gravitational acceleration direction in the real space is the same as the left direction of the virtual camera based on the gravitational acceleration direction in the virtual space.

Next, the processor 81 acquires various data, updates the operation data Da, angular velocity data Db, acceleration data Dc, and illuminance data Dd (step S201), and proceeds to the next step. For example, the processor 81 acquires operation data from the left controller 3 and/or the right controller 4, and updates the operation data Da. The processor 81 also acquires touch operation data from the touch panel 13, and updates the operation data Da. The processor 81 also acquires inertia data (acceleration data and angular velocity data) from the inertial sensors (acceleration sensor 89 and angular velocity sensor 90) provided in the main unit 2, and updates the acceleration data Dc and angular velocity data Db. The processor 81 also acquires illuminance data from the illuminance sensor 29, and updates the illuminance data Dd.

Next, the processor 81 calculates the attitude of the main unit 2 (step S202) and proceeds to the next step. For example, the processor 81 uses the acceleration data and angular velocity data stored in the angular velocity data Db and the acceleration data Dc to calculate the direction of the gravity vector of the gravitational acceleration acting on the main unit 2, and updates the attitude data De. The processor 81 also calculates the direction of rotation and the amount of rotation from the initial attitude of the main unit 2, and updates the attitude data De. For example, the processor 81 calculates the direction of rotation and the amount of rotation about the x, y, and z-axis directions of the main unit 2 in the initial attitude, and updates the attitude data De. Note that the direction of rotation can be expressed by the positive and negative values of the amount of rotation, so only data indicating the amount of rotation may be stored in the attitude data De. For example, the processor 81 may add the amount of rotation based on the angular velocity data acquired in this step S202 to the amount of rotation calculated in the previous process, and calculate a new amount of rotation.

Next, the processor 81 performs a process to determine whether the main device 2 is attached to the goggle device 150 (step S203), and proceeds to the next step. For example, if the illuminance indicated by the illuminance data Dd indicates a dark illuminance less than a threshold value for detecting the illuminance when the main device 2 is attached to the goggle device 150, the processor 81 determines in step S203 that the main device 2 is attached to the goggle device 150. The detection means detects whether the display device is attached to the goggle device or is in the process of being attached, and corresponds to the processor 81 that performs the process of step S203, for example.

Next, the processor 81 determines whether or not the display mode is stereoscopic (step S204). For example, if the processor 81 determines that the main device 2 is attached to the goggle device 150 in the wearing determination process in the above step S203, the processor 81 makes a positive determination in the above step S204 and performs processing in the stereoscopic display mode. On the other hand, if the processor 81 determines that the main device 2 is not attached to the goggle device 150 in the wearing determination process in the above step S203, the processor 81 makes a negative determination in the above step S204 and performs processing in the non-stereoscopic display mode. Then, if the processor 81 performs processing in the stereoscopic display mode, the process proceeds to step S205. On the other hand, if the processor 81 performs processing in the non-stereoscopic display mode, the process proceeds to step S213. The mode setting means executes processing to set the display device to the first display mode or a second display mode different from the first display mode, and corresponds to the processor 81 that performs the processing in step S204, for example.

In step S205, the processor 81 performs object movement processing and proceeds to the next step. For example, when an operation is being performed to move a control object in virtual space, the processor 81 refers to the operation data Da and sets the movement of the control object according to the operation. Then, based on the set movement of the control object, the processor 81 sets the position, direction, attitude, movement, etc. of the control object in the virtual space, and updates the control object data Df.

In the object operation process in step S205, the following object operation control is possible. As a first example, a predetermined operation object is set as an operation target and the operation object is operated. In this case, the predetermined operation object is moved, operated, or deformed based on an input to each input unit in the left controller 3 and/or the right controller 4. As a second example, an operation object is selected as an operation target based on an operation, and the operation object is operated. In this case, an operation object arranged at a predetermined display position (for example, an operation object overlapping with a sign displayed in the center of the display screen) is set as an operation target, and the operation object selected as the operation target is moved, operated, or deformed based on an input to each input unit in the left controller 3 and/or the right controller 4. As a third example, the operation object set as the operation target is operated based on vibration to the goggle device 150 to which the main unit 2 is attached. For example, the gravitational acceleration components are removed from the acceleration in the x, y and z axes directions in the main unit 2 indicated by the acceleration data Dc, and if the acceleration after the removal indicates that a vibration of a predetermined magnitude or greater is being applied to the main unit 2, the operation object that is the subject of the operation is moved in response to the vibration. Note that any method may be used to extract the gravitational acceleration, and for example, the acceleration components occurring on average in the main unit 2 may be calculated and extracted as the gravitational acceleration.

Next, the processor 81 performs a process of operating the pair of left and right virtual cameras (step S206) and proceeds to the next step. For example, the processor 81 sets the orientation of the pair of left and right virtual cameras in the virtual space by rotating the pair of left and right virtual cameras by the rotation amount calculated in step S202 from the initial orientation, and updates the virtual camera data Dg. For example, the processor 81 rotates the virtual camera about the left and right direction of the virtual camera as an axis by the same amount of rotation about the left and right axis direction (x-axis direction) of the main unit 2 calculated in step S202, rotates the virtual camera about the up and down direction of the virtual camera as an axis by the same amount of rotation about the up and down axis direction (y-axis direction) of the main unit 2 calculated in step S202, and rotates the virtual camera about the line of sight direction of the virtual camera as the same amount of rotation about the screen depth axis direction (z-axis direction) of the main unit 2 calculated in step S202, while fixing the positional relationship of the pair of left and right virtual cameras from the initial orientation, thereby setting the orientation of the pair of left and right virtual cameras in the virtual space.

Next, the processor 81 performs processing to generate a virtual space image for the left eye (step S207), and proceeds to the next step. For example, the processor 81 places an operation object in the virtual space based on the operation object data Df. Then, the processor 81 generates a virtual space image as seen from the left virtual camera of the pair of left and right virtual cameras set by the virtual camera data Dg, as a virtual space image for the left eye, and updates the virtual space image data for the left eye Dh.

Next, the processor 81 performs processing to generate a virtual space image for the right eye (step S208), and proceeds to the next step. For example, the processor 81 generates a virtual space image as seen from the right virtual camera of the pair of left and right virtual cameras set by the virtual camera data Dg, as a virtual space image for the right eye, and updates the virtual space image data Di for the right eye.

Next, the processor 81 performs processing to generate a user interface image for stereoscopic vision (step S209), and proceeds to the next step. For example, the processor 81 generates a user interface image for stereoscopic vision that matches the shape of the third area (see FIG. 9) of the display 12, and updates the UI image data Dj for stereoscopic vision.

Next, the processor 81 performs a display control process to display the left-eye virtual space image in the first area of the display 12 (step S210), and proceeds to the next step. For example, the processor 81 displays the left-eye virtual space image set in the left-eye virtual space image data Dh as the left-eye image IML in the entire first area of the display 12 (see FIG. 9).

Next, the processor 81 performs a display control process to display the right-eye virtual space image in the second area of the display 12 (step S211), and proceeds to the next step. For example, the processor 81 displays the right-eye virtual space image set in the right-eye virtual space image data Di as the right-eye image IMR in the entire second area of the display 12 (see FIG. 9).

Next, the processor 81 performs a display control process to display the stereoscopic UI image in the third area of the display 12 (step S212), and proceeds to step S219. For example, the processor 81 displays the user interface image set in the stereoscopic UI image data Dj as the user interface image IMUa (e.g., two user interface images IMUa1 and IMUa2) at a predetermined position in the third area of the display 12 (see FIG. 9). Note that the second display control means executes a process to display, in the second display mode, a second image including a content image consisting of a left-eye image and a right-eye image having parallax therebetween and a second user interface image corresponding to the first user interface image on the display screen, and corresponds to the processor 81 that performs the processes of steps S205 to S212, for example.

On the other hand, if it is determined in step S204 that the display mode is non-stereoscopic, then in step S213, the processor 81 performs object movement processing and proceeds to the next step. For example, when an operation is performed to move a control object in virtual space with reference to the operation data Da, the processor 81 sets the movement of the control object according to the operation. Then, based on the set movement of the control object, the processor 81 sets the position, direction, attitude, movement, etc. of the control object in virtual space, and updates the control object data Df. Note that the object movement processing in step S213 is similar to the object movement processing in step S205 described above, and therefore a detailed description thereof will be omitted.

Next, the processor 81 performs a process to operate the single virtual camera (step S214) and proceeds to the next step. For example, the processor 81 sets the orientation of the single virtual camera in the virtual space by rotating it from the initial orientation by the amount of rotation calculated in step S202, and updates the virtual camera data Dg. For example, the processor 81 sets the orientation of the single virtual camera in the virtual space by rotating the single virtual camera from the initial orientation by the same amount of rotation about the left-right axis direction (x-axis direction) of the main unit 2 calculated in step S202, rotating the single virtual camera about the up-down direction (y-axis direction) of the main unit 2 calculated in step S202, and rotating the single virtual camera about the line of sight of the virtual camera by the same amount of rotation about the screen depth axis direction (z-axis direction) of the main unit 2 calculated in step S202.

Next, the processor 81 performs processing to generate a virtual space image (step S215), and proceeds to the next step. For example, the processor 81 generates a virtual space image viewed from the virtual camera set by the virtual camera data Dg, and updates the non-stereoscopic virtual space image data Dk.

Next, the processor 81 performs processing to generate a non-stereoscopic user interface image (step S216), and proceeds to the next step. For example, the processor 81 generates a non-stereoscopic user interface image to be superimposed on the single image IMS and displayed on the display 12 (see FIG. 16), and updates the non-stereoscopic UI image data Dm.

Next, the processor 81 performs a display control process to display the virtual space image in the entire area of the display 12 (step S217), and proceeds to the next step. For example, the processor 81 displays the virtual space image set in the non-stereoscopic virtual space image data Dk as a single image IMS in the entire area of the display 12 (see FIG. 16).

Next, the processor 81 performs a display control process for superimposing the non-stereoscopic UI image on the single image IMS (step S218), and proceeds to step S219. For example, the processor 81 superimposes the user interface image set in the non-stereoscopic UI image data Dm as a user interface image IMUb (e.g., two user interface images IMUb1 and IMUb2) on the single image IMS in the upper left corner area and upper right corner area (see FIG. 16) of the display 12. The first display control means executes a process for displaying a first image including a content image, which is a non-stereoscopic image, and a first user interface image on the display screen in the first display mode, and corresponds to the processor 81 that performs the processes of steps S213 to S218, for example.

In step S219, the processor 81 performs a user interface operation process and proceeds to the next step. For example, when a touch operation is being performed on the touch panel 13, the processor 81 refers to the operation data Da and sets a user operation instruction corresponding to the touch operation according to a user interface image displayed on the display 12 that overlaps with the position where the touch operation is being performed. Then, the processor 81 performs processing according to the set user operation instruction.

Next, processor 81 determines whether or not to end the game (step S220). Conditions for ending the game in step S220 include, for example, the result of the game being finalized or the user performing an operation to end the game. If processor 81 does not end the game, it returns to step S201 and repeats the process, and if it does end the game, it ends the process according to this flowchart. Thereafter, the series of processes from step S201 to step S220 are repeatedly executed until it is determined in step S220 that the game is to be ended.

In this way, in this embodiment, when switching between the normal display mode (non-stereoscopic display mode) and the stereoscopic display mode, the user interface image is displayed in a different position according to the set display mode, thereby improving the convenience of presenting the user interface image. In addition, the display mode is automatically switched based on the state in which the main unit 2 is attached to the goggle device 150, so that seamless switching of the display mode is possible. Furthermore, in this embodiment, in the stereoscopic display mode, the user interface image is displayed in a third area different from the first area and the second area of the display 12, which respectively displays the image for the right eye and the image for the left eye, so that a user interface image that does not interfere with the stereoscopic image display can be displayed.

In the above embodiment, the image display system is configured by mounting the main body device 2 having the information processing function to the goggle device 150, but the image display system may be configured in other ways. As a first example, a control unit that performs the above information processing (game processing) to generate an image may be provided in the goggle device 150, and a display device having a function of displaying an image may be mounted on the goggle device 150 to configure an image display system that displays a stereoscopic image. In this case, the stereoscopic image and the user interface image are displayed on the display device by outputting image data from the control unit provided in the goggle device 150 to the display device for displaying the stereoscopic image and the user interface image on the display device. In the above first example, the mechanism for detecting the mounting status of the display device, the mechanism for detecting the attitude of the image display system, the mechanism for accepting user operations, etc. may be provided in either the display device or the goggle device 150. As a second example, a control device that is wirelessly or wired connected to the display device may be provided separately from the goggle device 150 to which the display device is mounted, and the image display system may be configured by the display device, the goggle device 150, and the control device. In this case, operation data, acceleration data, angular velocity data, illuminance data, etc. are output from the display device to the control device, and a content image and a user interface image in a display mode based on the illuminance data are output from the control device to the display device. Note that, even in the above-mentioned second example, the mechanism for detecting the wearing status of the display device, the mechanism for detecting the attitude of the image display system, the mechanism for accepting user operations, etc. may be provided in either the display device or the goggle device 150. The control device controls the display range of the content image to be displayed according to the operation data, acceleration data, and angular velocity data acquired from the display device and/or the goggle device 150, and outputs the same to the display device. Note that, in this embodiment, the main unit device 2 is used as an example of a display device.

Note that in the above-described embodiment, the method for detecting the attitude of the main unit 2 is merely an example, and the attitude of the main unit 2 may be detected using other methods or other data. As an example, the main unit 2 and/or the goggle device 150 to which the main unit 2 is attached may be imaged from the outside, and the attitude of the main unit 2 and/or the goggle device 150 may be detected using the image thus captured. Furthermore, the controller for controlling the operation of the operation object may be not only the left controller 3 or the right controller 4, but also another controller.

In addition, the game system 1 and/or the main unit 2 may be any type of device, such as a portable game device, any portable electronic device (PDA (Personal Digital Assistant), mobile phone, personal computer, camera, tablet, etc.), etc.

In the above-described embodiment, a stereoscopic image is displayed by displaying a left-eye image and a right-eye image, which have parallax between them, on the left and right screens of the display 12, respectively. However, the left-eye image and the right-eye image may be displayed on different screens. For example, if the display 12 provided on the main unit 2 is composed of multiple display screens, the left-eye image is displayed on one of the multiple display screens, and the right-eye image is displayed on another of the multiple display screens. In this case, the user can view a stereoscopic image through the goggle device 150 by viewing with his left eye the left-eye image displayed on one of the multiple display screens through the left-eye lens 153L, and viewing with his right eye the right-eye image displayed on the other of the multiple display screens through the right-eye lens 153R.

In addition, the stereoscopic image and non-stereoscopic image displayed on the main unit 2 are displayed as game images by the processor 81 executing information processing (game processing) in response to a user operation, or as moving images and still images by the processor 81 performing video playback and still image playback in response to a user operation. In other words, the stereoscopic image and non-stereoscopic image displayed on the main unit 2 are generated by the processor 81 of the main unit 2 performing information processing (e.g., game processing, video playback processing, still image playback processing), but at least a part of the processing for generating the stereoscopic image and non-stereoscopic image may be performed by another device. For example, if the main unit 2 is further configured to be able to communicate with another device (e.g., a server, another image display device, another game device, another mobile terminal, another information processing device), the above processing may be further performed by the other device working in cooperation. In this way, by performing at least a part of the above processing by another device, processing similar to the above processing is possible. In addition, the above information processing can be performed by one processor or multiple processors working in cooperation with each other in an information processing system composed of at least one information processing device. In addition, in the above embodiment, the processor 81 of the main unit 2 can perform information processing by executing a specific program, but some or all of the above processing may be performed by a dedicated circuit provided in the main unit 2.

Here, according to the above-mentioned modified example, it is possible to realize the present invention in a so-called cloud computing system configuration, or in a distributed wide area network or local network system configuration. For example, in a distributed local network system configuration, it is possible for the above-mentioned processing to be performed in cooperation between a stationary information processing device (stationary game device) and a portable information processing device (portable game device). Note that in these system configurations, there is no particular limitation on which device performs the above-mentioned processing, and it goes without saying that the present invention can be realized regardless of the division of processing load.

Furthermore, the processing order, setting values, conditions used for judgment, etc. used in the above-mentioned information processing are merely examples, and it goes without saying that this embodiment can be realized even with other orders, values, and conditions.

The above program may be supplied to the game system 1 not only through an external storage medium such as an external memory, but also through a wired or wireless communication line to the device. The above program may be pre-recorded in a non-volatile storage device inside the device. Note that, in addition to non-volatile memory, the information storage medium for storing the above program may be a CD-ROM, a DVD, or similar optical disk-shaped storage medium, a flexible disk, a hard disk, a magneto-optical disk, a magnetic tape, or the like. The information storage medium for storing the above program may be a volatile memory for storing the above program. Such a storage medium may be a recording medium that can be read by a computer or the like. For example, the various functions described above can be provided by having a computer or the like read and execute the program from these recording media.

Although the present invention has been described in detail above, the above description is merely illustrative of the present invention in all respects and is not intended to limit its scope. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. It is understood that the scope of the present invention should be interpreted only by the claims. It is also understood that a person skilled in the art can implement an equivalent scope based on the description of the present invention and technical common sense from the description of specific examples of the present invention. It is also understood that the terms used in this specification are used in the sense commonly used in the field unless otherwise specified. Therefore, unless otherwise defined, all technical terms and technical terms used in this specification have the same meaning as commonly understood by a person skilled in the art to which this invention belongs. In the event of a conflict, this specification (including definitions) shall prevail.

As described above, the present invention can be used as an image display system, an image display program, an image display method, a display device, etc., which can improve the convenience of presenting user interface images.

1... Game system 2... Main unit 3... Left controller 4... Right controller 11... Housing 12... Display 29... Illuminance sensor 81... Processor 83... Controller communication unit 85... DRAM
150: Goggle device 151: Main body 152: Lens frame member 153: Lens 154: Plate-shaped member

Claims (13)

An image display system including a display device having a display screen for displaying an image and a goggle-type device on which the display device can be worn,
a mode setting means for setting the display device to a first display mode or a second display mode different from the first display mode;
a first display control means for causing the display screen to display a first image including a content image, which is a non-stereoscopic image, and a first user interface image in the first display mode;
a second display control means for displaying, on the display screen in the second display mode, a second image including a content image including a left-eye image and a right-eye image having parallax therebetween, and a second user interface image which is a non-stereoscopic image at least in a position different from that of the first user interface image;
the second display control means, in the second display mode, causes the second user interface image to be displayed in a third area of the display screen excluding an area constituted by a first area on the display screen for displaying the left eye image and a second area on the display screen for displaying the right eye image;
the goggle-type device has an opening for exposing to the outside the second user interface image included in the third area that is a part of the display screen when the display device is attached to the goggle-type device;
The opening is formed at a position corresponding to a user's nose when the user wears the goggle-type device. The present invention further includes a detection means for detecting whether the display device is attached to the goggle-type device or is in the process of being attached to the goggle-type device,
2. The image display system according to claim 1, wherein the mode setting means switches the display device to the second display mode based on a detection result of the detection means when the display device is set to the first display mode. The display device includes an illuminance sensor,
the goggle-type device includes a light blocking unit that blocks light from reaching the illuminance sensor of the display device when the display device is in a state of being worn or in the process of being worn on the goggle-type device;
3. The image display system according to claim 2, wherein the detection means detects whether the display device is attached to the goggle-type device or is in the process of being attached based on a detection result by the illuminance sensor. the display device includes a touch panel provided on the display screen,
the first user interface image displayed on the display screen can provide an operation instruction in response to a touch operation on the touch panel;
4. The image display system according to claim 1, wherein the second user interface image displayed on the display screen is capable of providing operational instructions in response to a touch operation on the touch panel when the display device is attached to the goggle-type device. The image display system according to any one of claims 1 to 4, wherein the second display control means causes the display screen to display an image corresponding to the content image displayed as a non-stereoscopic image by the first display control means as a content image consisting of a left-eye image and a right-eye image. The image display system according to any one of claims 1 to 5, wherein the second display control means, in the second display mode, converts the content image displayed in the first display mode immediately before the second display mode is set to a stereoscopic image consisting of a left-eye image and a right-eye image, and displays the stereoscopic image on the display screen. The image display system according to any one of claims 1 to 6, wherein the second display control means displays, as the second user interface image, a user interface image that is substantially the same in function as the first user interface image but has a different display mode. The image display system according to claim 7, wherein the second display control means adjusts the shape of the second user interface image to fit the shape of the third area and displays the image in the third area. 9. The image display system according to claim 1 , wherein the second display control means sets the third area below the display screen between the first area and the second area on the display screen. 10. The image display system according to claim 1, wherein the first display control means, in the first display mode, displays the first user interface image so as to be superimposed on a content image displayed on the display screen. The display device further includes a display device side connection section that can be electrically connected to another device,
the goggle-type device includes a goggle-type device-side connection part that is electrically connectable to the display device-side connection part,
The image display system of claim 2, wherein the detection means detects whether the display device is attached to the goggle-type device or is in the process of being attached in response to the connection between the display device side connection unit and the goggle-type device side connection unit. 1. An image display method for displaying an image on a display device that has a display screen for displaying an image and is attachable to a goggle-type device, comprising:
a mode setting step of setting the display device to a first display mode or a second display mode different from the first display mode;
a first display control step of displaying a first image including a content image, which is a non-stereoscopic image, and a first user interface image on the display screen in the first display mode;
a second display control step of displaying, on the display screen in the second display mode, a second image including a content image including a left-eye image and a right-eye image having parallax therebetween, and a second user interface image which is a non-stereoscopic image at least in a position different from that of the first user interface image;
In the second display control step, in the second display mode, the second user interface image is displayed in a third area of the display screen excluding an area constituted by a first area on the display screen for displaying the left eye image and a second area on the display screen for displaying the right eye image,
the third region is configured such that, when the display device is attached to the goggle-type device, the second user interface image is exposed to the outside through an opening of the goggle-type device;
The image display method, wherein the opening is formed at a position corresponding to a user's nose when the user wears the goggle-type device. A goggle-type device on which a display device can be attached,
the display device has a display screen on a front surface for displaying a content image including a left-eye image and a right-eye image having a parallax therebetween, and a user interface image;
The goggle-type device comprises:
a left-eye lens located in front of the left-eye image when the display device is worn;
a right-eye lens located in front of the right-eye image when the display device is worn;
an opening portion that is located on a front side of the user interface image when the display device is attached and exposes the user interface image to the outside;
A goggle-type device, wherein the opening is formed at a position corresponding to a user's nose when the user wears the goggle-type device.

Images