JP7851339B2 - Image processing program, image processing system, image processing method, and image processing apparatus. - Google Patents

Description

This invention relates to an image processing program, an image processing system, an image processing method, and an image processing apparatus capable of representing reflections based on a drawn image.

Conventionally, there are methods for representing reflection by processing the drawn image (for example, Non-Patent Document 1).

“Screen Space Reflection”, [online], 2023, Unity Technologies, [Accessed December 20, 2023], Internet <URL: https://docs.unity3d.com/ja/2019.4/Manual/PostProcessing-ScreenSpaceReflection.html>

The conventional techniques described above may not be able to generate reflections from objects located in the background of the screen.

Therefore, the object of the present invention is to provide an image processing program, an image processing system, an image processing method, and an image processing apparatus capable of generating reflections from objects located behind an object.

To solve the above problems, the present invention employs the following configuration.

(First structure)
The first configuration image processing program causes the computer of the information processing device to perform a first depth test and update the first depth buffer using the first depth buffer for objects in the virtual space, excluding objects of the first type, to draw to the frame buffer based on the results of the first depth test, and for objects of the first type, to place a planar object with the rendered image set as a texture without drawing to the frame buffer at the position of the first object in the virtual space. The image processing program causes the computer to, for each pixel in the frame buffer that has been drawn, to calculate, based on the depth of the first depth buffer, the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest, as the incident direction, and the direction of the ray reflected with that position as the reflection position, as the ray trace direction, trace the ray along the ray trace direction, determine the collision position where the ray collides with an object in the virtual space based on the first depth buffer, and if the collision position is determined within a range where the trace distance of the ray is within a first distance, determine a reflection color based on the color of the pixel in the frame buffer corresponding to the collision position, to be added to the color of the pixel of interest, and if the ray intersects with the planar object, determine a reflection color based on the color of the texture at the intersection position, to be added to the color of the pixel of interest. The image processing program causes the computer to reflect the reflection color in the frame buffer, and after reflecting the reflection color, draw the first type of object in the frame buffer.

According to the above, objects other than the first type are first drawn to the frame buffer, ray tracing is performed, and the color of the pixel at the ray collision point is determined as the reflection color to be added to the color of the pixel of interest. On the other hand, when a planar object with the image of the first type object set as its texture intersects with the ray, the color of the texture at the intersection point is determined as the reflection color to be added to the color of the pixel of interest. This makes it possible to generate reflections of both the first type object and the other object even when another object exists behind the first type object.

(Second structure)
In the second configuration, in the first configuration, the first type of object may be a flat-shaped object.

According to the above, even when generating reflections using a planar object with an image of the first type of object set as its texture, the image can be made to look natural because the first type of object itself has a flat shape.

(The third structure)
In the third configuration, in the first or second configuration described above, the first type of object may be a player character object controlled based on an operation input.

According to the above, even when the first type of object is controlled based on an input, it is possible to generate reflections of the first type of object and objects located behind it.

Furthermore, other embodiments may include an image processing system, an image processing apparatus, or an image processing method that performs the image processing described above.

According to the present invention, even when another object exists behind the first type of object, it is possible to generate reflections of both the first type of object and the other object.

This diagram shows an example of the main unit 2 with the left controller 3 and right controller 4 attached. Block diagram showing an example of the internal configuration of the main unit 2. This figure shows an example of multiple objects placed in the virtual space during the execution of the game of this embodiment. A diagram showing an example of a game image displayed on a display device. This diagram shows an overview of image processing related to the reflection of objects other than character objects. A diagram showing an example of the first depth buffer. This figure shows an example of an image drawn to the frame buffer after rendering in step S2. A diagram showing an example of a second depth buffer. This diagram illustrates how ray tracing is performed, showing how a ray reflected from reflection position RPa collides with the cylindrical object 34. This figure shows how collision detection for rays reflected from reflection position RPa is performed using the first depth buffer. This diagram illustrates how ray tracing is performed, showing how a ray reflected from reflection position RPb collides with tree object 32. This diagram illustrates how ray tracing works, showing how rays reflected from the reflection point RPc do not collide with the object. This diagram shows how ray collision detection is performed using the second depth buffer. This figure shows an example of an image drawn to the frame buffer when the color reflected in each pixel is added. A diagram showing an example of a ray tracing direction with the reflection direction corrected. A diagram illustrating the details of the correction for the direction of reflection. This diagram shows an example of a game image when the character object 41 is not located in front of the cylindrical object 34. This diagram shows an example of a game image when the character object 41 moves and is now in front of the cylindrical object 34. This diagram outlines the image processing steps used to generate reflections of character objects and objects behind them. This figure shows an example of a character plane created in step S11. A diagram illustrating the calculation of the color of the character's reflection in step S13. Figure 19 shows an example of a game image that will be displayed after the image processing shown in Figure 19 has been performed. This diagram shows an example of data stored in the memory of the main unit 2. A flowchart showing an example of the main processing performed by the processor 81 of the main unit 2. A flowchart showing an example of the SSR process in step S105. A flowchart showing an example of ray tracing processing in step S122. A flowchart showing an example of the process for calculating the color of the character's reflection in step S141.

(System configuration)
The following describes a game system according to an example of this embodiment. The example of the game system 1 in this embodiment includes a main unit (information processing device; functioning as the main unit of the game device in this embodiment) 2, a left controller 3, and a right controller 4. The left controller 3 and the right controller 4 are detachable from the main unit 2. In other words, the game system 1 can be used as an integrated device by attaching the left controller 3 and the right controller 4 to the main unit 2. Alternatively, the game system 1 can be used with the main unit 2 and the left controller 3 and right controller 4 as separate units. The following describes the hardware configuration of the game system 1 of this embodiment, followed by a description of the control of the game system 1 of this embodiment.

Figure 1 shows an example of the main unit 2 with the left controller 3 and right controller 4 attached. As shown in Figure 1, the left controller 3 and right controller 4 are each attached to the main unit 2 and integrated together. The main unit 2 is a device that executes various processes (e.g., game processing) in the game system 1. The main unit 2 is equipped with a display 12. The left controller 3 and right controller 4 are devices equipped with operation parts for user input.

The left controller 3 and the right controller 4 are detachable from the main unit 2. In the following, the left controller 3 and the right controller 4 will be collectively referred to as "controllers."

The main unit 2 alone, or the integrated unit with the left controller 3 and right controller 4 attached to the main unit 2, may be a portable device. Furthermore, the main unit 2 or the integrated unit may be a handheld device. Also, the main unit 2 or the integrated unit may be a portable device.

Furthermore, the main unit 2 includes a touch panel 13 on the screen of the display 12. In this embodiment, the touch panel 13 is of a multi-touch input type (for example, a capacitive touch panel). However, the touch panel 13 may be of any type; for example, it may be of a single-touch input type (for example, a resistive touch panel).

Figure 2 is a block diagram showing an example of the internal configuration of the main unit 2.

The main unit 2 includes a processor 81. The processor 81 is an information processing unit that performs various information processing operations in the main unit 2, and includes one or more CPUs (Central Processing Units) and one or more GPUs (Graphics Processing Units). The processor 81 may be composed of a System-on-a-chip (SoC) that includes multiple functions such as CPU and GPU functions. Note that the CPU and GPU may be configured as separate processors. Furthermore, one or more memories for temporarily storing data are provided inside the processor 81. The processor 81 performs various information processing operations by executing information processing programs (e.g., game programs) stored in a storage unit (specifically, an internal storage medium such as flash memory 84, or an external storage medium installed in slot 23).

The main unit 2 includes, as an example of an internal storage medium, a flash memory 84 and a DRAM (Dynamic Random Access Memory) 85. The flash memory 84 and DRAM 85 are connected to the processor 81. The flash memory 84 is primarily used to store various types of data (which may include programs) stored in the main unit 2. The DRAM 85 is used to temporarily store various types of data used in information processing.

The main unit 2 is equipped with a slot interface (hereinafter abbreviated as "I/F") 91. The slot I/F 91 is connected to the processor 81. The slot I/F 91 is connected to slot 23 and performs data reading and writing to a predetermined type of storage medium (e.g., a dedicated memory card) installed in slot 23, according to instructions from the processor 81.

The processor 81 performs the above-mentioned information processing by appropriately reading and writing data to and from the flash memory 84 and DRAM 85, as well as to each of the storage media.

The main unit 2 includes a network communication unit 82. The network communication unit 82 is connected to the processor 81. The network communication unit 82 communicates with external devices via a network (specifically, wirelessly). In this embodiment, the network communication unit 82 communicates with external devices by connecting to a wireless LAN using a method compliant with the Wi-Fi standard as a first communication mode. Furthermore, the network communication unit 82 performs wireless communication with other main units 2 of the same type using a predetermined communication method (for example, communication using a proprietary protocol or infrared communication) as a second communication mode.

The main unit 2 includes a controller communication unit 83. The controller communication unit 83 is connected to the processor 81. The controller communication unit 83 communicates wirelessly with the left controller 3 and/or the right controller 4. While the communication method between the main unit 2 and the left controller 3 and right controller 4 is arbitrary, in this embodiment, the controller communication unit 83 communicates with the left controller 3 and the right controller 4 in accordance with the Bluetooth® standard.

The processor 81 is connected to the left terminal 17, the right terminal 21, and the lower terminal 27. When the processor 81 communicates with the left controller 3 via a wired connection, it transmits data to the left controller 3 via the left terminal 17 and receives operation data from the left controller 3 via the left terminal 17. Similarly, when the processor 81 communicates with the right controller 4 via a wired connection, it transmits data to the right controller 4 via the right terminal 21 and receives operation data from the right controller 4 via the right terminal 21. Furthermore, when the processor 81 communicates with the cradle, it transmits data to the cradle via the lower terminal 27. Thus, in this embodiment, the main unit 2 can perform both wired and wireless communication with the left controller 3 and the right controller 4, respectively. Also, when the left controller 3 and the right controller 4 are mounted on the main unit 2 as an integrated unit, or when the main unit 2 alone is mounted on the cradle, the main unit 2 can output data (e.g., image data or audio data) to a stationary monitor or the like via the cradle.

The main unit 2 includes a touch panel controller 86, which is a circuit for controlling the touch panel 13. The touch panel controller 86 is connected between the touch panel 13 and the processor 81. Based on signals from the touch panel 13, the touch panel controller 86 generates data indicating, for example, the location where a touch input occurred, and outputs it to the processor 81.

The main unit 2 comprises a power control unit 97 and a battery 98. The power control unit 97 is connected to the battery 98 and the processor 81. Although not shown, the power control unit 97 is also connected to various parts of the main unit 2 (specifically, the parts that receive power from the battery 98, the left terminal 17, and the right terminal 21). Based on commands from the processor 81, the power control unit 97 controls the power supply from the battery 98 to these parts.

Furthermore, the battery 98 is connected to the lower terminal 27. When an external charging device (e.g., a cradle) is connected to the lower terminal 27 and power is supplied to the main unit 2 via the lower terminal 27, the supplied power charges the battery 98.

(Overview of image processing)
Next, the image processing of this embodiment will be described. In the game system 1 of this embodiment (an example of an image processing system), multiple objects are placed in a three-dimensional virtual space, and the game is played.

Figure 3 shows an example of multiple objects placed in the virtual space during the execution of the game of this embodiment. When the game of this embodiment starts, a three-dimensional virtual space defined by the xyz Cartesian coordinate system is established. For example, the y-axis is the upward axis of the virtual space, the x-axis is the rightward axis, and the z-axis is the backward axis.

As shown in Figure 3, multiple objects, including a ground object 30, are placed in the virtual space. The ground object 30 is an object that mimics the ground, for example, a plane parallel to the xz plane. The ground object 30 may have irregularities or slopes. A tree object 32 and a cylinder object 34 are placed on the ground object 30. Furthermore, for example, a mountain object 36 is placed at a predetermined position in the z-axis direction of the virtual space. These ground objects 30, tree objects 32, and cylinder objects 34 are fixed in the virtual space.

Furthermore, character objects 41 and 43 are placed in the virtual space. Character objects 41 and 43 are flat objects, specifically plate-like objects with a thickness shorter than their length in the vertical, horizontal, and vertical directions. For example, character objects 41 and 43 are 3D objects, each containing a planar mesh forming the front and a planar mesh forming the back. Character object 41 is a player character controlled by the player. Character object 41 performs actions in the virtual space in response to controller input. For example, character object 41 may move on the ground object 30, move its limbs in the virtual space, jump, or change the direction of its face. When character object 41 moves horizontally, the screen scrolls horizontally. Character object 43 is a non-player character that moves in response to the movement of character object 41 and is automatically controlled by processor 81. Character object 43 may also be controlled by the player.

A virtual camera is placed within the virtual space, and a game image is generated from the virtual camera's perspective of the virtual space. This generated game image is then displayed on the display 12 or a stationary monitor (hereinafter referred to as the "display device").

Figure 4 shows an example of a game image displayed on a display device. The display device shows images of each object (30, 32, 34, 36, 41, 43). For example, the ground object 30 is a reflective surface, and each object is reflected on the ground object 30. In this embodiment, the reflections of each object are generated using a method called SSR (Screen Space Reflection).

For example, as shown in Figure 4, the reflections 52 of the tree object 32 and 54 of the cylinder object 34 are displayed on the ground object 30. Additionally, the reflections 61 of the character object 41 and 63 of the character object 43 are displayed on the ground object 30. However, the reflection of the mountain object 36 is not displayed on the ground object 30.

In this embodiment, the tree object 32 and the cylinder object 34 are pre-configured as specific objects and are set to appear on the ground object even when they are a certain distance away from the virtual camera. On the other hand, the mountain object 36 is not configured as a specific object.

A specific object is defined at the mesh (a collection of polygons) level. For example, each of the multiple meshes that make up the tree object 32 may be defined as a specific object. Alternatively, any of the multiple meshes that make up the tree object 32 may be defined as a specific object. For example, if the tree object 32 is composed of a first mesh that makes up the trunk and a second mesh that makes up the leafy part, only the first mesh may be defined as a specific object. In other words, a "specific object" does not necessarily have to be an object that appears as a single object visually; it may be a mesh that makes up part of an object that appears as a single object visually. For example, a terrain object in which a ground surface parallel to the xz plane and a wall surface perpendicular to the xz plane are continuously formed may appear to form a single terrain with steps, but the ground surface and the wall surface may be composed of different meshes. In such a terrain object, the wall surface may be defined as a specific object, while the ground surface does not necessarily have to be defined as a specific object.

In this embodiment, the tree object 32 and the cylindrical object 34, which are set as specific objects, are reflected on the ground object. The image processing related to the reflection of these objects will be described below.

Figure 5 shows an overview of image processing related to reflections of objects other than character objects.

As shown in Figure 5, the processor 81 first performs a first depth test (step S1). Here, a depth test is performed on a pixel-by-pixel basis for all objects other than character objects (30, 32, 34, 36), and the results of this depth test are stored in the first depth buffer.

Figure 6 shows an example of the first depth buffer. As shown in Figure 6, the first depth buffer stores the pixel-by-pixel depth values for all objects other than character objects (30, 32, 34, 36). Note that Figure 6 displays images of each object, which conceptually represent the depth values of each pixel stored in the first depth buffer. The position of each pixel is represented by coordinate values in the screen coordinate system. For example, the Sx axis of the screen coordinate system is the rightward axis in the image (screen), and the Sy axis is the upward axis in the image. The origin of the screen coordinate system may be set, for example, at the center of the image, or at the bottom left of the image.

Returning to Figure 5, after step S1, the processor 81 renders to the frame buffer based on the results of the first depth test (step S2). In step S2, the image is drawn to the frame buffer based on the depth stored in the first depth buffer. In this embodiment, deferred rendering (also called deferred shading) is used as the rendering method. In step S2, the normal buffer, which stores the normal information for each pixel, is updated, and the image is drawn to the frame buffer. Note that forward rendering may be used instead of deferred rendering as the rendering method.

Figure 7 shows an example of an image drawn to the frame buffer after rendering in step S2. As shown in Figure 7, in step S2, images of all objects other than character objects (30, 32, 34, 36) are drawn to the frame buffer. Color information is stored for each pixel. The color information includes, for example, RGB values representing three colors and an alpha value representing transparency (or opacity). Note that shading is omitted in Figure 7, but in step S2, an image with shading is generated.

Returning to Figure 5, after step S2, the processor 81 performs a second depth test on specific objects among the multiple objects placed in the virtual space (step S3). Specifically, the second depth test is performed on the tree object 32 and the cylinder object 34, which are set as specific objects among the multiple objects (30, 32, 34, 36), and the results of this second depth test are stored in the second depth buffer. As a result, the second depth buffer stores the pixel-by-pixel depth values only for the tree object 32 and the cylinder object 34 among the multiple objects (30, 32, 34, 36) included in the imaging range of the virtual camera.

Figure 8 shows an example of a second depth buffer. As shown in Figure 8, the second depth buffer stores the depth values for each pixel of the tree object 32 and the cylinder object 34. On the other hand, the depth value of the mountain object 36 is not stored in the second depth buffer. Note that Figure 8 conceptually represents the depth values of each pixel stored in the second depth buffer.

Returning to Figure 5, after step S3, the processor 81 performs ray tracing using the first or second depth buffer (step S4). Here, processing is performed for each pixel (hereinafter referred to as the "focus pixel"). Specifically, the processor 81 sets the focus pixel, calculates the reflection position of a ray (a virtual light ray) cast from the virtual camera's position toward the focus pixel, and performs a collision determination to determine whether the ray reflected from that reflection position collides with an object.

Next, the processor 81 calculates the color of the pixel in the frame buffer corresponding to the ray collision position as the color reflected on the pixel of interest (step S5). The processing in step S5 is also performed for each pixel.

The following describes in detail the processing in steps S4 and S5 when each of the pixels PIXa to PIXc shown in Figure 7 is set as the pixel of interest.

For example, the processor 81 sets pixel PIXa, shown in Figure 7, as the pixel of interest. Pixel PIXa is a pixel located closer to the virtual camera than the cylindrical object 34. Based on the position of the pixel of interest PIXa in the screen coordinate system and the depth value of that position stored in the first depth buffer, the processor 81 calculates the position of the pixel of interest PIXa in virtual space as the ray reflection position RPa.

Figure 9 illustrates the ray tracing process, showing how a ray reflected from the reflection point RPa collides with the cylindrical object 34. The rightward direction in Figure 9 represents the depth of the screen when the game image, generated based on the virtual camera VC, is displayed on the screen.

As shown in Figure 9, the processor 81 defines the direction from the virtual camera VC towards the reflection position RPa as the incident direction, and calculates the direction of the ray reflected from the reflection position RPa as the ray tracing direction RTDa. Specifically, the processor 81 calculates the reflection direction of the ray based on the incident direction and the normal to the reflection position RPa. The angle of incidence in the incident direction and the angle of reflection in the reflection direction are equal. Then, the processor 81 calculates the reflection direction, or a direction corrected from the reflection direction, as the ray tracing direction RTDa. The correction of the reflection direction will be explained later.

The processor 81 propagates a ray from the reflection position RPa in the ray tracing direction RTDa and performs a collision determination to determine whether the ray collides with an object. If the ray's trace distance is less than La, the processor 81 uses the first depth buffer to perform the collision determination. Here, the trace distance is the distance from the reflection position along the ray tracing direction.

Figure 10 shows how collision detection of a ray reflected from the reflection position RPa is performed using the first depth buffer. The processor 81 first calculates the position in the virtual space by traveling a maximum distance Lb in the ray tracing direction RTDa from the reflection position RPa in the virtual space, and then converts this calculated position to the screen coordinate system to calculate the ray tracing end position in the screen coordinate system. In the screen coordinate system, the processor 81 uses the reflection position RPa (the pixel of interest RPa) as the ray tracing start position, extends the ray from the ray tracing start position to the ray tracing end position, and determines whether the tip of the ray collides with an object. Specifically, the processor 81 travels the ray a predetermined distance and determines whether the ray collides with an object based on the depth of the ray tip and the depth of each position stored in the first depth buffer. In the example shown in Figures 9 and 10, it is determined that the ray reflected at the reflection position RPa collides at the collision position CPa.

If it is determined that the ray collides with an object at collision position CPa, in step S5, the color of the pixel in the frame buffer corresponding to the collision position CPa is calculated as the color reflected in the pixel of interest RPa.

Next, we will explain the cases where pixel PIXb, shown in Figure 7, is set as the pixel of interest, and where pixel PIXc is set as the pixel of interest. Pixel PIXb is a pixel located closer to the virtual camera than the tree object 32. Similarly, pixel PIXc is a pixel located closer to the virtual camera than the mountain object 36.

Figure 11 shows how ray tracing is performed, illustrating how a ray reflected from reflection position RPb collides with the tree object 32. Figure 12 shows how ray tracing is performed, illustrating how a ray reflected from reflection position RPc does not collide with the object. Figure 13 shows how ray collision detection is performed using the second depth buffer.

As shown in Figure 11, when the processor 81 sets pixel PIXb as the pixel of interest, it calculates the reflection position RPb corresponding to the pixel of interest PIXb, as described above, and calculates the direction of the ray reflected from the reflection position RPb as the ray tracing direction RTDb. The processor 81 propagates the ray in the ray tracing direction RTDb and performs a collision determination to determine whether the ray collides with an object. When the trace distance is less than La, the processor 81 performs the ray collision determination using the first depth buffer. However, when the trace distance is less than La, the ray does not collide with an object. Therefore, the processor 81 propagates the ray further and performs the ray collision determination using the second depth buffer (Figure 13). As shown in Figures 11 and 13, when the trace distance is greater than or equal to La and less than or equal to the maximum distance Lb, it is determined that the ray collides with the tree object 32 at the collision position CPb.

On the other hand, as shown in Figure 12, when the processor 81 sets pixel PIXc as the pixel of interest, it calculates the reflection position RPc corresponding to the pixel of interest PIXc, calculates the direction of the ray reflected from the reflection position RPc as the ray tracing direction RTDc, and propagates the ray in the ray tracing direction RTDb. When the trace distance is less than La, the processor 81 performs ray collision detection using the first depth buffer. When the trace distance is La or greater and less than or equal to the maximum distance Lb, it performs ray collision detection using the second depth buffer. Here, as shown in Figure 13, the depth of the mountain object 36 is not stored in the second depth buffer. Therefore, as shown in Figures 12 and 13, the processor 81 determines that the ray will not collide with the object even when propagated to the maximum distance Lb.

Furthermore, the parameters used to add the color of the pixel at the collision point to the color of the pixel of interest may differ depending on whether the ray collision point is calculated based on the first depth buffer or the second depth buffer. For example, when the collision point is calculated based on the second depth buffer, the parameters may be changed so that the reflected color becomes darker than when the collision point is calculated based on the first depth buffer.

The processing described in steps S4 and S5 above is performed for each pixel of interest, thereby determining the color reflected in each pixel. Then, by adding the determined reflected color to the color of each pixel of interest, the reflection of each object is drawn to the frame buffer.

Figure 14 shows an example of an image drawn to the frame buffer when the color reflected in each pixel is added.

As shown in Figure 14, the color of the pixel at the collision position CPa calculated as described above is added to the pixel RPa. The same process is performed on each pixel in the area in front of the cylinder object 34 (the area on the virtual camera side), so that the reflection 54 of the cylinder object 34 is drawn to the frame buffer. Similarly, the color of the pixel at the collision position CPb is added to the pixel RPb. The same process is performed on each pixel in the area in front of the tree object 32, so that the reflection 52 of the tree object 32 is drawn to the frame buffer. On the other hand, the collision position is not calculated for the pixel RPc. The same applies when processing is performed on each pixel in the area in front of the mountain object 36. Therefore, the reflection of the mountain object 36 is not drawn to the frame buffer.

In this embodiment, the results of a first depth test performed on multiple objects are stored in a first depth buffer, and the results of a second depth test performed on a specific object among the multiple objects are stored in a second depth buffer. When the trace distance is less than La, ray collision detection is performed based on the first depth buffer. When the trace distance is La or greater and less than or equal to the maximum distance Lb, ray collision detection is performed based on the second depth buffer. This allows SSR to display reflections of a specific object among multiple objects, even when it is far from the virtual camera, while preventing reflections from being displayed for other objects. By designating the object to be reflected as a specific object, the desired object can be reflected, for example, on the ground. Conversely, by not designating an object that should not be reflected as a specific object, the desired object can be prevented from being reflected.

Furthermore, the parameters used to reflect the color of the pixel at the collision point onto the target pixel differ depending on whether the ray collision point is calculated based on the first depth buffer or the second depth buffer. This allows, for example, a specific object to be reflected more intensely and clearly.

(Correction of reflection direction)
Next, the correction of the reflection direction will be explained. As described above, the direction from the virtual camera VC towards the reflection position is calculated as the incident direction, and the reflection direction is calculated based on this incident direction and the normal direction of the reflection position. In this embodiment, if the correction conditions are met, the direction in which the reflection direction has been corrected is set as the ray tracing direction. If the correction conditions are not met, the reflection direction is set as the ray tracing direction. The correction conditions for correcting the reflection direction and the method of correction will be explained in detail below.

Figure 15 shows an example of a ray tracing direction after correction of the reflection direction. As shown in Figure 15, for example, when ray tracing is performed on a pixel of interest near the left edge of the screen (reflection position RP1), the reflection direction RD1, calculated based on the incident direction and normal direction, is corrected, and the corrected direction is set as the ray tracing direction RTD1. Similarly, when ray tracing is performed on a pixel of interest located to the right of reflection position RP1 (reflection position RP2), the reflection direction RD2 is corrected, and the corrected direction is set as the ray tracing direction RTD2.

The correction conditions are met when both the conditions regarding the reflection position and the reflection direction are satisfied. The condition regarding the reflection position is that the reflection position is within a predetermined range from the edge of the screen. The condition regarding the reflection direction is that the reflection direction is directed outwards from the screen. If either the reflection position condition or the reflection direction condition is not met, the correction conditions are not satisfied, and the reflection direction is not corrected. For example, if the reflection position is within a range from the left edge of the screen to 1/4 of the screen's horizontal length, and the reflection direction is directed to the left of the screen, the correction conditions are satisfied. Similarly, if the reflection position is within a range from the right edge of the screen to 1/4 of the screen's length, and the reflection direction is directed to the right of the screen, the correction conditions are satisfied.

The degree of correction is greater at reflection position RP1, which is closer to the left edge of the screen, than at reflection position RP2. Specifically, the degree of correction CR is determined based on the position of the reflection position in the Sx-axis direction of the screen coordinate system. The closer the reflection position is to the left or right edge of the screen in the Sx-axis direction, the greater the degree of correction CR. Also, the larger the angle between the reflection direction and the Sy-axis direction in the screen coordinate system, the smaller the degree of correction CR.

Figure 16 illustrates the details of the correction for the reflection direction. As shown in Figure 16, for example, if the angle between the reflection direction RD and the upward direction of the screen (Sy axis direction) is 45 degrees or less, the correction degree CR is set to a value CR(Sx) corresponding to the Sx coordinate value of the reflection position RP. CR(Sx) is set, for example, in the range of 0 to 1, and increases as the reflection position approaches the left or right edge of the screen. If the angle between the reflection direction RD and the upward direction of the screen exceeds 45 degrees, the value CR(Sx) corresponding to the Sx coordinate value of the reflection position RP decreases according to the angle exceeding 45 degrees. If the angle between the reflection direction RD and the upward direction of the screen exceeds a threshold (for example, 60 to 65 degrees), the correction degree CR is set to "0".

Depending on the degree of correction CR, a correction is applied in the direction of reflection, towards the inside of the screen. Specifically, the position obtained by converting the position obtained by moving the maximum distance Lb from the reflection position RP in the reflection direction RD in the virtual space to the screen coordinate system is calculated as the ray tracing end position EP. The Sx coordinate value of this ray tracing end position EP is corrected to move inward towards the screen, depending on the degree of correction CR. For example, linear interpolation is used to correct the Sx coordinate value of the ray tracing end position EP to approach the Sx coordinate value of the reflection position RP. For example, if the reflection position RP is located within a predetermined range from the left edge of the screen, the ray tracing end position EP is moved to the right (positive Sx axis direction) depending on the degree of correction CR. The Sx coordinate value of the ray tracing end position EP is not moved beyond the Sx coordinate value of the reflection position RP. Also, the Sy coordinate value of the ray tracing end position EP does not change before and after the correction. Then, the direction from the reflection position EP to the ray tracing end position EP' after movement is calculated as the ray tracing direction RTD.

Thus, in areas near the edges of the screen, the direction corrected for the reflection direction is calculated as the ray tracing direction. This allows for the generation of SSR reflections at the edges of the screen. For example, as shown in Figure 15, when ray tracing is performed in the reflection direction RD1 for a pixel at position RP1 on the left edge of the screen, the ray collision point may be outside the screen, and the color of the pixel at the collision point cannot be added to the pixel at position RP1. Therefore, if no correction is performed for the reflection direction, the edges of the screen may have a different color than other parts, resulting in an unnatural-looking image.

In this embodiment, the reflection direction is corrected based on the incident direction and the normal direction, and the corrected direction is set as the ray tracing direction. This makes it easier to keep the ray collision position within the screen, and allows for the generation of reflections based on the rendered image. The Sx coordinate value of the ray tracing end position EP is moved to approach the Sx coordinate value of the reflection position RP. Therefore, even with maximum correction, the reflection direction pointing outwards from the screen will not point inwards from the screen, but rather upwards. Consequently, for example, the color of pixels to the right of the left edge of the screen will not be reflected in pixels at that position. This prevents unnatural distortion of objects when reflected on the ground.

Furthermore, in this embodiment, the degree of correction increases as the reflection position approaches the edge of the screen. Therefore, even near the edge of the screen, reflections can be generated based on pixels within the screen.

Furthermore, for example, if the reflection direction is directed inward, correcting the reflection direction can result in a reflection that is far removed from the actual reflection. In this embodiment, since no correction is performed when the reflection direction is not directed outward, it is possible to prevent the generation of reflections that are far removed from the actual reflection.

Furthermore, when the angle between the reflection direction and the top of the screen is relatively large (for example, exceeding 45 degrees), the degree of correction (CR) decreases. When the angle between the reflection direction and the top of the screen exceeds a threshold, the degree of correction becomes 0. This prevents unnatural reflections from being generated due to excessive correction, for example, when the ground is sloped. It also suppresses correction for reflections on walls where the reflection direction is horizontal.

Furthermore, in this embodiment, the correction of the reflection direction is performed only for the left-right component in the screen coordinate system. Therefore, it is suitable, for example, for games where the virtual space expands horizontally and is scrolled horizontally.

(Generating reflections of character objects)
Next, we will explain how to generate reflections of character objects 41 and 43. Figure 17 shows an example of a game image when character object 41 is not in front of cylindrical object 34. Figure 18 shows an example of a game image when character object 41 has moved and is in front of cylindrical object 34.

As shown in Figure 17, if there is no character object 41 in front of the cylindrical object 34, for example, ray tracing is performed from the reflection position RPd on the front side, the collision position CPd is calculated, and the color of the collision position CPd is added to the color of the reflection position RPd. By performing the same process for each pixel, the reflection 54 of the cylindrical object 34 is displayed.

On the other hand, in Figure 18, the character object 41 is located in front of the cylindrical object 34, and the head of the character object 41 is at the same position CPd as in Figure 17. In this case, the pixel at the reflection position RPd cannot have the color of the pixel at the collision position CPd of the cylindrical object 34 shown in Figure 17, and part of the cylindrical object 34 is not reflected at the reflection position RPd. Thus, for example, when the character object 41 is located in front of the cylindrical object 34, it may not be possible to represent a natural reflection that corresponds to the actual positional relationship of each object.

In this embodiment, image processing is performed using the method shown in Figure 19 to display the reflections of the character object and the object behind it. Figure 19 is a diagram illustrating the overview of the image processing for generating the reflections of the character object and the object behind it.

As shown in Figure 19, the processor 81 first renders objects other than character objects (step S10). Here, the process of drawing multiple objects 30, 32, 34, and 36 placed in the virtual space to the frame buffer is performed. The process in step S10 corresponds to the processes in steps S1 and S2 of Figure 5.

Next, the processor 81 renders the character object into a separate buffer and creates a character plane with the rendered image set as a texture (step S11).

Figure 20 shows an example of a character plane created in step S11. As shown in Figure 20, the character object 41 is rendered to a separate buffer, and the rendered image of the character object 41 is set as a texture on a two-dimensional plane object to create the character plane 65. Similarly, a character plane 66 is created by setting the rendered image of the character object 43 as a texture. In this embodiment, an image of each character object is generated every frame. Note that the rendered images of each character object may be prepared in advance. For example, images corresponding to each action of the character object 41 may be stored in advance on a storage medium, and the character plane 65 may be created by setting these pre-stored images as textures.

Next, the processor 81 calculates the reflection colors of objects other than character objects (step S12). Here, the reflection colors are calculated for multiple objects 30, 32, 34, and 36 placed in the virtual space. Specifically, the processor 81 performs the processing shown in steps S3 to S5 of Figure 5 above. At this point, since the character object has not yet been drawn to the frame buffer, the reflection color of that object can be calculated even if the character object is located in front of the virtual camera VC in the virtual space. The calculated reflection colors of objects other than character objects (hereinafter referred to as "first reflection colors") are stored in the reflection buffer. For example, the reflection buffer stores RGB values and alpha values as color information indicating the first reflection color.

Next, the processor 81 calculates the color of the character's reflection (step S13). Here, the processor 81 calculates the color of the character object's reflection when the character plane created in step S11 is placed in the virtual space.

Figure 21 is a diagram illustrating the calculation of the color of the character reflection in step S13. As shown in Figure 21, the processor 81 calculates the intersection position of the character plane 65 and the ray when the character plane 65 is placed in the virtual space according to the position and orientation of the character object 41 in the virtual space. Specifically, the processor 81 calculates the depth when the character plane 65 is placed in the virtual space and calculates the intersection position CRPe of the ray traveling from the reflection position RPe toward the ray tracing direction RTDe and the character plane 65. The color of the pixel at the intersection position CRPe is calculated as the color of the character reflection (hereinafter referred to as the "second reflection color"). Here, the part of the character plane 65 other than the image of the character object 41 is set to transparent and does not intersect with the ray. For example, a ray traveling from the reflection position RPf toward the ray tracing direction RTDf does not intersect with the image of the character object 41 in the character plane 65. In this case, the intersection position is not calculated. The calculated second reflection color is stored in the reflection buffer. For example, the reflection buffer stores RGB values and alpha values as color information indicating the color of the second reflection.

Returning to Figure 19, the processor 81 then calculates the color reflected in the pixel of interest based on the results of steps S12 and S13 (step S14). For example, the color reflected in the pixel of interest is calculated by adding the second reflected color to the first reflected color. Here, "adding the second color to the first color" may also mean mixing the two colors. For example, "adding the second color to the first color" may mean performing alpha blending with the first color as the background color and the second color as the foreground color. Alternatively, "adding the second color to the first color" may mean performing alpha blending with the second color as the background color and the first color as the foreground color. Furthermore, "adding the second color to the first color" may mean mixing the first and second colors based on their respective alpha values. Also, "adding the second color to the first color" may mean that the second color takes precedence over the first color, and the second color is overpainted onto the first color.

Next, the processor 81 renders the result calculated in step S14 into the frame buffer (step S15). Here, the processor 81 adds the reflected color calculated in step S14 to the color of the pixel of interest stored in the frame buffer. For example, the processor 81 performs alpha blending using the color of the pixel of interest as the background color and the reflected color calculated in step S14 as the foreground color. As a result, the reflections of the character object and other objects are drawn into the frame buffer.

After step S15 is completed, the processor 81 renders the character objects 41 and 43 into the frame buffer (step S16). This generates a game image, as shown in Figure 22, which is then displayed on the display device.

Figure 22 shows an example of a game image displayed after the image processing shown in Figure 19 has been performed. As shown in Figure 22, a tree object 32 and its reflection 52 are displayed. A character object 43 and its reflection 63 are also displayed. Furthermore, a cylinder object 34 and its reflection 54 are displayed. A character object 41 is located in front of the cylinder object 34, and the character object 41 and its reflection 61 are displayed. While the reflection 54 of the cylinder object 34 and the reflection 61 of the character object 41 partially overlap, their shapes correspond to their actual shapes and relative positions, resulting in a natural-looking reflection.

In this embodiment, the character object is a flat object. As described above, when generating a reflection of the character object based on a character plane where the character object's image is set as a texture, the reflection of the character object appears flat. If the character object itself is a three-dimensional object with a certain thickness, its reflection appearing flat may result in an unnatural image. However, in this embodiment, since the character object itself is a flat object, an image without unnaturalness can be achieved.

(Details of image processing)
Next, we will explain the details of the image processing described above. First, we will explain the data stored in the memory of the main unit 2 (memory in the processor 81, DRAM 85, flash memory 84, or external storage medium, etc.). Figure 23 is a diagram showing an example of the data stored in the memory of the main unit 2.

As shown in Figure 23, the memory of the main unit 2 stores the program, operation data, object data, character data, and character plane data. The memory of the main unit 2 also stores the first depth buffer, the second depth buffer, the normal buffer, the reflection buffer, and the frame buffer.

The program is for executing the main processing described later, and includes an image processing program for performing the image processing related to reflections mentioned above. The program is pre-stored in an external storage medium or flash memory 84 installed in slot 23 and is loaded into DRAM 85 when the game is executed. The program may also be obtained from other devices via a network (e.g., the Internet).

The operation data consists of data related to operations acquired from the left controller 3 and the right controller 4. For example, the operation data is transmitted from the left controller 3 and the right controller 4 to the main unit 2 at predetermined time intervals (e.g., 1/200 second intervals) and stored in memory.

Object data pertains to each object (30, 32, 34, 36) placed in the virtual space, excluding character objects. Object data includes data on each object's position and orientation in the virtual space, as well as data on its shape and texture. Furthermore, object data includes information on whether each object is designated as a specific object. For example, tree object 32 and cylinder object 34 are designated as specific objects. On the other hand, mountain object 36 is not designated as a specific object.

The character data pertains to character objects 41 and 43. The character data includes data relating to the position and orientation of each character in virtual space, data relating to its shape, and data relating to its texture. In this embodiment, each character object is formed as a 3D model, but it is formed as a planar object with a thickness that is thinner in the thickness direction compared to the up, down, left, and right directions.

Character plane data is a two-dimensional object, and it is data relating to a plane object for which the rendered image of a character object has been set as a texture. The character plane data includes data relating to character plane 65, for which the image of character object 41 has been set as a texture, and data relating to character plane 66, for which the image of character object 43 has been set as a texture.

The first depth buffer is a buffer that stores the depth values for each pixel, which are generated and updated based on the results of the first depth test. The first depth buffer stores the depth values of all objects except character objects that are included in the imaging range of the virtual camera VC.

The second depth buffer is a buffer that stores the depth values for each pixel, which are generated and updated based on the results of the second depth test. The second depth buffer stores the depth values of specific objects included within the imaging range of the virtual camera VC.

A normal buffer is a buffer that stores the normal information of each pixel.

The reflection buffer is a buffer for storing the first reflection color (the reflection color of objects other than character objects) and the second reflection color (the reflection color of character objects). In the ray tracing process described later, the reflection buffer stores the first reflection color when a ray collides with an object other than a character object, and the second reflection color when the ray intersects with the character plane. As will be explained in detail later, the reflection buffer may store multiple reflection colors.

The frame buffer is a buffer that stores the game images to be displayed. The images stored in the frame buffer are output to the display device at predetermined timings and displayed on the display device.

(Details of game processing in main unit 2)
Next, with reference to Figures 24 to 27, the details of the processing performed in the main unit 2 will be described. Figure 24 is a flowchart showing an example of the main processing performed by the processor 81 of the main unit 2. The processing shown in Figures 24 to 27 is performed by the CPU or GPU of the processor 81.

As shown in Figure 24, the processor 81 first performs initial processing (step S100). Specifically, the processor 81 sets up a three-dimensional virtual space and places various objects used in the game, such as objects (30, 32, 34, 36), character objects (41, 43), a virtual camera VC, a light source, and other objects, into the virtual space. After performing the initial processing, the processor 81 repeatedly executes the following steps S101 to S108 at predetermined frame time intervals (for example, 1/60 second intervals).

In step S101, the processor 81 acquires operation data from the controller.

Next, the processor 81 performs game processing based on the acquired operation data (step S102). For example, the processor 81 moves the character object 41 in the virtual space or causes the character object 41 to perform a predetermined action (e.g., a jump action, an attack action, etc.) based on the operation data. The processor 81 also moves the character object 43 in the virtual space or causes the character object 43 to perform a predetermined action according to a predetermined algorithm. Furthermore, the processor 81 controls enemy objects other than character objects in the virtual space and moves obstacle objects that hinder character objects in the virtual space.

Next, the processor 81 performs rendering processing on objects other than character objects (step S103). Specifically, the processor 81 performs a first depth test on each object other than character objects 41 and 43 that are included in the imaging range of the virtual camera VC, and updates the first depth buffer. The processor 81 then draws each object to the frame buffer based on the depth stored in the first depth buffer. In this embodiment, deferred rendering is used as the rendering method. The normal buffer is also updated in step S103. Note that forward rendering may also be used as the rendering method.

Next, the processor 81 renders the character object into a separate buffer to create a character plane (step S104). Specifically, the processor 81 renders the character object 41 into a separate buffer and creates a character plane 65 by setting the image of the rendered character object 41 as a texture to a plane object. Similarly, the processor 81 creates a character plane 66 by setting the image of the character object 43 as a texture to a plane object.

After step S104, the processor 81 performs SSR processing (step S105). SSR processing is the process of rendering reflections of all objects, including character objects, to the frame buffer. The details of SSR processing are described below.

(SSR processing)
Figure 25 is a flowchart showing an example of the SSR process in step S105.

As shown in Figure 25, the processor 81 performs a second depth test on a specific object and updates the second depth buffer (step S121). For example, the processor 81 performs a depth test on the tree object 32 and the cylinder object 34, which have been pre-configured as specific objects, and updates the second depth buffer. Note that the configuration of specific objects is done on a mesh-by-mesh basis.

Next, processor 81 performs ray tracing (step S122). Ray tracing is performed for each pixel of interest. Here, the color of the reflection on each pixel of interest is calculated. Note that ray tracing may be performed for all pixels, or only for a specific range of pixels. For example, the load may be reduced by excluding areas where the model is not rendered, areas where SSR is disabled at the model or mesh level, or areas where the normal direction is outside a specific range. The details of ray tracing are described below.

(Ray tracing)
Figure 26 is a flowchart showing an example of the ray tracing process in step S122.

As shown in Figure 26, the processor 81 calculates the position of the pixel of interest in virtual space as the reflection position and calculates the reflection direction (step S131). Specifically, the processor 81 calculates the position of the pixel of interest in virtual space as the reflection position based on the position of the pixel of interest in the screen coordinate system and the depth of the pixel of interest stored in the first depth buffer. Furthermore, the processor 81 defines the direction from the position of the virtual camera VC toward the reflection position as the incident direction, and calculates the reflection direction based on this incident direction and the normal direction of the reflection position.

Next, the processor 81 sets the calculated reflection direction, or the direction corrected from the reflection direction, as the ray tracing direction (step S132). Here, if the correction conditions are met, the direction corrected from the reflection direction is set as the ray tracing direction; if the correction conditions are not met, the reflection direction is set as the ray tracing direction. The ray tracing end position is also set. The correction conditions and correction method are as described above.

After step S132, the processor 81 sets a first depth buffer as a depth buffer for determination and starts ray tracing from the pixel of interest (step S133). Ray tracing is performed in the screen coordinate system. Next, the processor 81 advances the ray a predetermined distance (step S134).

Next, the processor 81 determines whether the trace distance has exceeded the maximum distance Lb (step S135). Specifically, the processor 81 determines whether the leading edge of the ray in the screen coordinate system has reached the ray trace end position.

If it is determined that the trace distance exceeds Lb (step S135: YES), the processor 81 then performs the process in step S141.

On the other hand, if the trace distance is less than or equal to Lb (step S135: NO), the processor 81 determines whether or not the ray has collided with an object (step S136). Specifically, the processor 81 uses a depth buffer for determination to determine whether or not the leading edge of the ray has collided with an object.

If it is determined that the ray has not collided with an object (step S136: NO), the processor 81 determines whether the trace distance is greater than or equal to La (step S137).

If the trace distance is greater than or equal to La (step S137: YES), the processor 81 changes the determination depth buffer to the second depth buffer (step S138) and executes the process in step S134 again. On the other hand, if the trace distance is less than La (step S137: NO), the processor 81 maintains the determination depth buffer and executes the process in step S134 again.

On the other hand, if it is determined that the ray has collided with an object (step S136: YES), the processor 81 calculates the color of the first reflection from the color of the ray's collision location (step S139). For example, the processor 81 may calculate the color of the pixel at the collision location stored in the frame buffer as the color of the first reflection, or it may calculate the color of the pixel at the collision location after applying a predetermined process as the color of the first reflection. The color of the first reflection differs depending on whether the collision is determined based on the first depth buffer or the second depth buffer. For example, if the first depth buffer is set as the depth buffer for determination, the color of the pixel at the collision location may be lightened (alpha value reduced) according to the trace distance, and the changed color may be calculated as the color of the first reflection. Also, if the second depth buffer is set as the depth buffer for determination, the color of the pixel at the collision location may be calculated as the color of the first reflection regardless of the trace distance. Furthermore, if a second depth buffer is set as the depth buffer for determination, the color of the pixel at the collision position is lightened according to the trace distance, but it is changed to be darker than when the first depth buffer is set as the depth buffer for determination, and this changed color may be calculated as the color of the first reflection. The processor 81 then stores the calculated color of the first reflection in the reflection buffer. Here, the color of the first reflection stored in the reflection buffer when a ray collision is determined based on the first depth buffer is denoted as "color of the first reflection (first depth buffer)". Similarly, the color of the first reflection stored in the reflection buffer when a ray collision is determined based on the second depth buffer is denoted as "color of the first reflection (second depth buffer)".

Next, the processor 81 determines whether the depth buffer for determination is the first depth buffer and whether the color of the first reflection calculated in step S139 is semi-transparent (step S140). For example, if the trace distance is less than or equal to La and the color of the first reflection is semi-transparent, ray tracing is performed further into the background. Semi-transparent means that the alpha value of the reflection is less than 1.0, but the alpha value is calculated independently based on the distance La, collision position, and various other parameters.

If the result in step S140 is NO, the processor 81 proceeds to step S141. If the result in step S140 is YES, the processor 81 proceeds to step S138.

In step S141, the processor 81 calculates the color of the reflection of the character object (the color of the second reflection). The details of the process in step S141 are described below.

Figure 27 is a flowchart showing an example of the color calculation process for character reflections in step S141.

As shown in Figure 27, the processor 81 calculates the intersection position of the ray and the character plane (step S151). Specifically, the processor 81 calculates the depth when the character plane is positioned according to the position and orientation of the character object in virtual space, and determines whether the ray used in the ray tracing intersects with the character plane. If the ray intersects with the character plane, the processor 81 calculates the intersection position.

Next, the processor 81 stores the color of the intersection point as the color of the second reflection in the reflection buffer (step S152). Specifically, the processor 81 stores the color based on the pixel color of the character object's texture image corresponding to the intersection point as the color of the second reflection. The color of the second reflection may be the pixel color of the character object's texture image, or it may be a color calculated by applying a predetermined process to the pixel color. After performing the process in step S152, the processor 81 terminates the process shown in Figure 27 and returns to the process shown in Figure 26. If the ray intersects with multiple character planes, multiple second reflection colors are stored in the reflection buffer. In this case, the multiple second reflection colors may be stored in the reflection buffer in order of proximity of the intersection point to the virtual camera.

As a result of the processing in steps S139 and S152, the reflection buffer stores zero to multiple reflection colors. For example, the reflection buffer may store reflection colors corresponding to the ray's collision or intersection positions, in order of proximity to the virtual camera. For example, if it is determined that a ray has collided at a first position where the trace distance is less than La, and also determined that a ray has collided at a second position where the trace distance is between La and Lb, and the ray intersects the character plane between the first and second positions, the reflection buffer stores the first reflection color (first depth buffer), the second reflection color, and the first reflection color (second depth buffer) in order of proximity to the virtual camera (see Figure 23). If the ray does not collide with any object and does not intersect the character plane, no reflection color is stored in the reflection buffer.

Returning to Figure 26, after processing in step S141, the processor 81 calculates the color reflected in the pixel of interest based on the reflected colors stored in the reflection buffer (step S142). If multiple reflected colors are stored in the reflection buffer, the processor 81 may calculate the color reflected in the pixel of interest by alpha blending the multiple reflected colors in order of distance from the virtual camera. For example, as shown in Figure 23, if the reflection buffer stores the first reflected color (first depth buffer), the second reflected color, and the first reflected color (second depth buffer) in order of proximity to the virtual camera, the processor 81 may calculate a color by alpha blending the second reflected color with the first reflected color (second depth buffer), and then alpha blend the first reflected color (first depth buffer) with the calculated color to calculate the color reflected in the pixel of interest. Furthermore, for example, if the reflection buffer stores the colors of the second reflection and the first reflection in order of proximity to the virtual camera (i.e., when there are no objects in front of the character from the virtual camera's perspective, and there are objects behind the character), the processor 81 may calculate the color reflected in the pixel of interest by alpha blending the color of the second reflection with the color of the first reflection. For example, if the color of the second reflection on the foreground is opaque, the color of the second reflection is set as the color reflected in the pixel of interest. Also, if the color of the second reflection on the foreground is semi-transparent, a color is calculated by blending the colors of the first and second reflections as the color reflected in the pixel of interest. The processor 81 stores the calculated reflection color in memory.

If the process in step S142 is completed, the processor 81 terminates the process shown in Figure 26 and returns to the process shown in Figure 25.

Returning to Figure 25, after step S122, the processor 81 renders to the frame buffer based on the ray tracing results (step S123). For example, the processor 81 adds the reflected color calculated in step S142 to the color of the pixel of interest stored in the frame buffer. This ensures that the reflected color calculated in step S142 is reflected in the color of the pixel of interest.

Next, the processor 81 determines whether or not the processes in steps S122 and S123 have been performed for all pixels (step S124). If the processor 81 determines NO in step S124, it repeats the process in step S122. By performing the processes in steps S122 and S123 for all pixels, reflections of objects other than character objects placed in the virtual space (52, 54) and reflections of character objects (61, 63) are drawn to the frame buffer. If the processor 81 determines YES in step S124, it terminates the process in Figure 25 and returns to the process in Figure 24.

Returning to Figure 24, after step S105, the processor 81 further renders the character objects 41 and 43 into the frame buffer (step S106).

Next, the processor 81 outputs the image stored in the frame buffer to the display device (step S107). This displays the game image.

Next, the processor 81 determines whether or not to terminate the game (step S108). For example, if the player instructs the game to end, the processor 81 terminates the game processing shown in Figure 24. If the processor 81 determines that the game should not be terminated (step S108: NO), it executes the process in step S101 again. This concludes the explanation of the main processing shown in Figure 24.

The order of processes, content, and values used for determination in the flowchart above are merely examples and may be modified as appropriate.

As described above, in this embodiment, if the trace distance is less than La, ray collision detection is performed based on the first depth buffer, and if the trace distance is La or greater, ray collision detection is performed based on the second depth buffer (steps S135 to S140). This allows reflections to be displayed even when the trace distance is long for a specific object. Furthermore, reflections can be hidden from objects other than the specific object, allowing for the display or exclusion of desired objects. Additionally, by generating reflections based on the first and second depth buffers, and by lightening the color of the reflection according to the trace distance when based on the first depth buffer, specific objects can be reflected large and clearly, while other objects can be reflected small.

Furthermore, in this embodiment, when the pixel of interest (reflection position) is near the edge of the screen, the direction obtained by correcting the reflection direction based on the incident direction and the normal direction at the reflection position is set as the ray tracing direction (step S132). This allows the color of the reflection to be calculated based on the pixels drawn in the frame buffer, enabling the generation of reflections even at the edge of the screen.

Furthermore, in this embodiment, objects other than character objects are first drawn to the frame buffer, their reflections are drawn, then the reflections of character objects are drawn, and finally the character objects are drawn. This allows for the display of reflections of objects behind the character object from the virtual camera's perspective, as well as the reflection of the character object itself.

Furthermore, in this embodiment, the color of the collision position calculated based on the first depth buffer, the color of the collision position calculated based on the second depth buffer, and the color of the intersection position with the character plane are calculated, and these colors are alpha-blended in order from the back. This allows the reflection of each object to be displayed even when the objects overlap as viewed from the virtual camera.

(Variant)
The image processing of this embodiment has been described above, but the above embodiment is merely an example, and modifications such as the following may be made.

For example, in the above embodiment, deferred rendering was used as the rendering method, but in other embodiments, forward rendering may be used. When forward rendering is used, in the SSR processing described above, for example, the normal direction of the reflection position corresponding to the pixel of interest may be estimated based on the depth of multiple pixels surrounding the pixel of interest, and the reflection direction may be calculated based on the incident direction and the said normal direction. Alternatively, the pose of the mesh in virtual space may be calculated, the normal direction of the reflection position corresponding to the pixel of interest may be calculated, and the reflection direction may be calculated based on the said normal direction.

Furthermore, in the above embodiment, the reflection direction was corrected for pixels in a predetermined range from the left or right edge of the screen. However, in other embodiments, the reflection direction may also be corrected for pixels in a predetermined range from the top or bottom edge of the screen. In the above embodiment, in correcting the reflection direction, the position of the ray tracing end position in the Sx axis direction in the screen coordinate system was brought closer to the position of the reflection position in the Sx axis direction. In other embodiments, the reflection direction may also be corrected by moving the position of the ray tracing end position in the Sy axis direction in the positive or negative direction of the Sy axis. The reflection direction may also be corrected by moving the ray tracing end position horizontally and/or vertically so that the ray collision position fits within the screen.

Furthermore, while ray tracing was performed in the screen coordinate system in the above embodiment, in other embodiments, ray tracing may be performed in the xyz coordinate system of a virtual space. That is, rays may be extended in the xyz coordinate system of a virtual space, the collision positions of the rays may be calculated, and the color of the pixel at the position corresponding to the collision position may be calculated as the reflected color.

Furthermore, in the above embodiment, a character plane was created by setting the image of the character object as a texture on a planar object, and the color of the reflection of the character object was calculated using this character plane. In other embodiments, the image of the character object may be set as a texture on a three-dimensional object, and the color of the reflection of the character object may be calculated using this three-dimensional object. Also, in the above embodiment, the character object was assumed to be a flat object, but in other embodiments, the character object may be a three-dimensional object with thickness.

Furthermore, while the above embodiment described the case where the ground is used as a reflective surface and an object is reflected on the ground, it is also possible to use a surface other than the ground (for example, a wall or ceiling) as a reflective surface and reflect the object using the process described above.

Furthermore, while the above embodiment was used to generate game images, the image processing described above may be used to generate any image, not just game images.

Furthermore, the above hardware configuration is merely an example, and the image processing may be performed on any other hardware. For example, the processing may be executed on any information processing device, such as a personal computer, tablet terminal, smartphone, or server on the internet. Also, the image processing may be performed in an information processing system including multiple devices. Multiple devices may perform the image processing in a distributed manner. Furthermore, the device performing the image processing and the device displaying the image may be different. For example, a first device (e.g., a server) may generate an image by performing part or all of the image processing, transmit the generated image to a second device via a network (e.g., the internet or a LAN), and the second device may display the image.

Furthermore, the configurations of the above embodiments and their modified forms can be combined in any way, as long as they do not contradict each other. Also, the above are merely illustrative examples of the present invention, and various other improvements and modifications may be made.

1. Game System 2. Main Unit 81. Processor 32. Tree Object 34. Cylinder Object 36. Mountain Object 41, 43. Character Object

Claims (12)

In the computer of the information processing device,
Of the objects in the virtual space, all objects except for the first type of object ,
The first depth buffer is used to perform a first depth test and update the first depth buffer.
Based on the results of the first depth test, the frame buffer is used to draw.
For the first type of object, a planar object with a rendered image set as a texture without drawing to the frame buffer is placed at the position of the first type of object in the virtual space.
For each pixel of the frame buffer on which drawing has been performed, the pixel in question is designated as the pixel of interest.
Based on the depth of the first depth buffer, the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest is calculated as the incident direction, and the direction of the ray reflected with that position as the reflection position is calculated as the ray tracing direction.
The ray is traced along the ray tracing direction, and the collision position where the ray collides with an object in the virtual space is determined based on the first depth buffer.
When the collision position is determined within a range where the tracing distance of the ray is within a first distance, a color based on the color of the pixel in the frame buffer corresponding to the collision position is determined as the reflection color to be added to the color of the pixel of interest.
When the ray intersects the planar object, the color of the texture at the intersection point is determined as the reflected color to be added to the color of the pixel of interest.
The frame buffer is then configured to reflect the color of the reflection.
An image processing program that, after reflecting the colors of the reflections, draws the first type of object to the frame buffer. The image processing program according to claim 1, wherein the first type of object is a flat-shaped object. The image processing program according to claim 2, wherein the first type of object is a player character object controlled based on an operation input. An image processing system comprising a processor, wherein the processor is
Of the objects in the virtual space, all objects except for the first type of object ,
A first depth test and an update of the first depth buffer are performed using the first depth buffer.
Based on the results of the first depth test, the data is drawn to the frame buffer.
For the first type of object, a planar object with a rendered image (without drawing to the frame buffer) set as its texture is placed at the position of the first type of object in the virtual space.
For each pixel of the frame buffer on which drawing has been performed, the pixel in question is designated as the pixel of interest.
Based on the depth of the first depth buffer, the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest is calculated as the incident direction, and the direction of the ray reflected with that position as the reflection position is calculated as the ray tracing direction.
The ray is traced along the ray tracing direction, and the collision position where the ray collides with an object in the virtual space is determined based on the first depth buffer.
When the collision position is determined within a range where the tracing distance of the ray is within a first distance, a color based on the color of the pixel in the frame buffer corresponding to the collision position is determined as the reflection color to be added to the color of the pixel of interest.
When the ray intersects the planar object, the color of the texture at the intersection point is determined as the reflection color to be added to the color of the pixel of interest.
The frame buffer reflects the color of the reflection,
An image processing system that, after reflecting the colors of the reflections, draws the first type of object to a frame buffer. The image processing system according to claim 4, wherein the first type of object is a flat-shaped object. The image processing system according to claim 5, wherein the first type of object is a player character object controlled based on an operation input. An image processing method,
Of the objects in the virtual space, all objects except for the first type of object ,
Perform a first depth test and update the first depth buffer using the first depth buffer.
Based on the results of the first depth test, draw to the frame buffer.
With respect to the first type of object, a planar object with a rendered image set as a texture without drawing to the frame buffer is placed at the position of the first type of object in the virtual space.
For each pixel of the frame buffer on which drawing has been performed, the pixel in question is designated as the pixel of interest.
Based on the depth of the first depth buffer, the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest is calculated as the incident direction, and the direction of the ray reflected with that position as the reflection position is calculated as the ray tracing direction.
The ray is traced along the ray tracing direction, and the collision position where the ray collides with an object in the virtual space is determined based on the first depth buffer.
When the collision position is determined within a range where the tracing distance of the ray is within a first distance, a color based on the color of the pixel in the frame buffer corresponding to the collision position is determined as the reflection color to be added to the color of the pixel of interest.
When the ray intersects the planar object, the color of the texture at the intersection point is determined as the reflection color to be added to the color of the pixel of interest.
The frame buffer reflects the color of the reflection.
An image processing method comprising: reflecting the color of the reflection, and then drawing the first type of object to the frame buffer. The image processing method according to claim 7, wherein the first type of object is a flat-shaped object. The image processing method according to claim 8, wherein the first type of object is a player character object controlled based on an operation input. An image processing apparatus comprising a processor, wherein the processor is
Of the objects in the virtual space, all objects except for the first type of object ,
A first depth test and an update of the first depth buffer are performed using the first depth buffer.
Based on the results of the first depth test, the data is drawn to the frame buffer.
For the first type of object, a planar object with a rendered image (without drawing to the frame buffer) set as its texture is placed at the position of the first type of object in the virtual space.
For each pixel of the frame buffer on which drawing has been performed, the pixel in question is designated as the pixel of interest.
Based on the depth of the first depth buffer, the direction from the virtual camera toward the position in the virtual space corresponding to the pixel of interest is calculated as the incident direction, and the direction of the ray reflected with that position as the reflection position is calculated as the ray tracing direction.
The ray is traced along the ray tracing direction, and the collision position where the ray collides with an object in the virtual space is determined based on the first depth buffer.
When the collision position is determined within a range where the tracing distance of the ray is within a first distance, a color based on the color of the pixel in the frame buffer corresponding to the collision position is determined as the reflection color to be added to the color of the pixel of interest.
When the ray intersects the planar object, the color of the texture at the intersection point is determined as the reflection color to be added to the color of the pixel of interest.
The frame buffer reflects the color of the reflection,
An image processing device that, after reflecting the color of the reflection, draws the first type of object to the frame buffer. The image processing apparatus according to claim 10, wherein the first type of object is a flat-shaped object. The image processing apparatus according to claim 11, wherein the first type of object is a player character object controlled based on an operation input.