diff --git a/art/diagrams/ImageUploadReadPixels.png b/art/diagrams/ImageUploadReadPixels.png deleted file mode 100644 index ee0d5ced2f03..000000000000 Binary files a/art/diagrams/ImageUploadReadPixels.png and /dev/null differ diff --git a/art/diagrams/NDC.png b/art/diagrams/NDC.png deleted file mode 100644 index 78bfe87e6760..000000000000 Binary files a/art/diagrams/NDC.png and /dev/null differ diff --git a/art/diagrams/RenderingReadPixels.png b/art/diagrams/RenderingReadPixels.png deleted file mode 100644 index 530549c7e709..000000000000 Binary files a/art/diagrams/RenderingReadPixels.png and /dev/null differ diff --git a/docs_src/.gitignore b/docs_src/.gitignore index f3711d83e3f7..cf1f74e640a7 100644 --- a/docs_src/.gitignore +++ b/docs_src/.gitignore @@ -3,3 +3,7 @@ src_mdbook/book src_mdbook/src/dup/*.md src_mdbook/src/main/filament.md src_mdbook/src/main/materials.md +src_mdbook/src/samples/web/*.md +src_mdbook/src/web +src_mdbook/src/release/cocoapods.md +src_mdbook/src/remote \ No newline at end of file diff --git a/docs_src/src_mdbook/src/SUMMARY.md b/docs_src/src_mdbook/src/SUMMARY.md index 00485e733c16..f789b4545a84 100644 --- a/docs_src/src_mdbook/src/SUMMARY.md +++ b/docs_src/src_mdbook/src/SUMMARY.md @@ -23,23 +23,18 @@ - [skinning](./samples/web/skinning.md) - [sky](./samples/web/sky.md) - [Technical Notes](./notes/README.md) - - [Material Properties](./notes/material_properties.md) - - [Release](./release/README.md) - - [CocoaPods](./release/cocoapods.md) - - [Maven](./release/maven.md) - - [Github](./release/guide.md) - - [Versioning](./release/versioning.md) - - [Branching](./release/branching.md) - - [Documentation](./dup/docs.md) - [Debugging](./notes/debugging.md) - [Metal](./notes/metal_debugging.md) - [Vulkan](./notes/vulkan_debugging.md) - [SPIR-V](./notes/spirv_debugging.md) - - [Running with ASAN and UBSAN](./notes/asan_ubsan.md) - - [Using Instruments on macOS](./notes/instruments.md) - - [Code coverage analysis](./notes/coverage.md) - - [Performance analysis](./notes/performance_analysis.md) + - [ASAN and UBSAN](./notes/asan_ubsan.md) + - [Instruments on macOS](./notes/instruments.md) + - [Code coverage](./notes/coverage.md) + - [Performance](./notes/performance_analysis.md) + - [Documentation](./dup/docs.md) + - [Material Properties](./notes/material_properties.md) - [Framegraph](./notes/framegraph.md) + - [2D Coordinate Spaces](./notes/2d_coordinate_spaces.md) - [Tests](./notes/tests.md) - [backend](./dup/backend_test.md) - [CI: backend](./dup/test_ci_backend.md) @@ -66,3 +61,9 @@ - [specgen](./dup/specgen.md) - [specular-color](./dup/specular_color.md) - [zbloat](./dup/zbloat.md) + - [Release](./release/README.md) + - [CocoaPods](./release/cocoapods.md) + - [Maven](./release/maven.md) + - [Github](./release/guide.md) + - [Versioning](./release/versioning.md) + - [Branching](./release/branching.md) diff --git a/docs_src/src_mdbook/src/notes/2d_coordinate_spaces.md b/docs_src/src_mdbook/src/notes/2d_coordinate_spaces.md new file mode 100644 index 000000000000..51aad1db6865 --- /dev/null +++ b/docs_src/src_mdbook/src/notes/2d_coordinate_spaces.md @@ -0,0 +1,221 @@ +# 2D Coordinate Spaces + +This page describes Filament's interpretation of 2D coordinate spaces. We address the differences +between graphics APIs and provide explanations for custom logic and client APIs. + +## 1. Texture coordinate space + +To begin our discussion, we imagine looking at a rectangular image, like a photograph. To be able to +describe to another person where things are on the photograph (say, a **tree**), we need to first +define a consistent coordinate system that both parties understand. A natural choice is to pick one +of the corners as the origin of the coordinate system (that is, corners are easy to identify on a +rectangle, and do not require tools - like a ruler - for positioning). We can further refine this +coordinate system by assigning 2D numbers to the four corners so that we can accurately +describe the positions on the photograph. + +So for example, we assign the corners in clockwise order, starting from the top-left corner: +\\((0, 0)\\), \\((1, 0)\\), \\((1, 1)\\), \\((0, 1)\\). + +Assuming we are sampling from a texture in a shader, we sample by using a 2D coordinate (UV) +to identify a point on the rectangular space of the texture. We can make a statement such as, "the +**tree** is near the point of \\(((0.65, 0.35)\\))", to communicate the position of a tree in +the picture to another person. + +Given the description above, we conclude this section with the following statements + - Programmers communicate with computers using coordinate systems like the above to draw 2D images + onto a display. + - In 3D graphics, similar coordinate systems are used to texturize 3D models with 2D images. The + 2D images are called *textures*. + - Filament defines its coordinate system using the *top-left* of the image as \\((0, 0)\\) and + the *bottom-right* is \\((1, 1)\\). + +### 1.1. OpenGL (The Bottom-Left API) + +OpenGL has its texture coordinate origin \\((0, 0)\\) at the **bottom-left**, and +\\((1, 1)\\) is at the top-right. Note that this differs from Filament's assumed texture +coordinate system. + +The GL function for uploading texture is `glTexImage2D`. Even though GL interprets \\((0, 0)\\) as +the bottom-left of a 2D image, `glTexImage2D` uploads the lower-bits as the *bottom* of the image +as opposed to the *top*. (This behavior is opposite of the other backends' texture upload +APIs). The two differences cancel each other, meaning there is no difference between the different +backends in upload and sampling of textures. + +We can restate the above as + +| | +|---| +| *Filament's internal texture coordinate space defines the **top-left** corner of the image as the origin.* | + +### 1.2. Readback of rendertarget + +While we've established that the APIs are not any different in the *uploading* and *sampling* of +textures for Filament's backends, there are, unfortunately, cases where special handling of +texture coordinate is necessary. + +Reading the result of rendering into CPU memory can be very useful; for example, one might want to +take snapshots of the rendering and output to image formats such as PNG or JPEG. Another use is to +perform image analysis or processing over a rendered image. In Filament, the readback path +corresponds to the `readPixels`. In the OpenGL backend, this is implemented using the `glReadPixels` +function. GL's interpretation of texture coordinate space holds true for `glReadPixels`; this means +that it treats the bottom-left of the image as the origin and the readback output *starts* from the +origin. Hence, the first row of the image in CPU memory is the bottom-most row of the image (when +displayed). + +This creates two related inconsistencies: + - First, (not a real use case), if one were to upload a texture, attach it as an attachment, and + read the attachment, the GL backend would place the bottom of the image as the first row of + pixels in CPU memory. + - Second, the other backends consider the top-left corner as the origin, which means their + readbacks would place the top of the image as the first row of the image in CPU memory. + +To account for these inconsistencies, *the GL backend vertically flips the rows of the readback +image*. + +### 1.3. `flipUV` in Materials + +In order to work with textures and associated primitives from different sources, Filament provides +a simple, material-level client-set flag to flip the v-direction. + +For example, if a 3D modeler +exported a mesh with textures such that the bottom-left corner of the texture is \\(u=0, v=0\\) in +texture space, then this is not consistent with Filament's assumption. Luckily, modern modelers will +only choose either the top-left corner or the bottom-right corner as the origin. To address this +inconsistency, Filament provides `flipUV` in the material definition. Setting this flag to true will +flip the UV coordinate in the vertex attributes so that \\(v' = 1 - v\\), and the texture +coordinate interpretation is then consistent with Filament's internal coordinates. + +## 2. Clip-space + +The clip-space is the bounded 3D space that a primitive vertex is transformed into at the end of a +vertex shader. Abstractly, the XY-plane of the clip-space is parallel to the display, and its +bounds correspond to the bounds of the window, surface, or buffer that we are rendering into. (This +API-specific bounded space is also known as Normalized Device Coordinates or NDC). +In different graphics API, the bounds and orientation of the NDC is also inconsistent. For all of +the backends, the ranges of the XY-plane \\([-1, 1]\\) and the origin (\\((0, 0)\\)) are the +same. However, the depth range and the Y-axis directions are not the same. + +| Graphics API | Depth Range (Z) | Y-Axis Direction | Frame Buffer Origin | +| -------------- | --------------- | ---------------- | ------------- | +| OpenGL / WebGL | \\([−1, 1]\\) | Up | bottom-left | +| Metal | \\([0, 1]\\) | Up | top-left | +| WebGPU | \\([0, 1]\\) | Up | top-left | +| Vulkan | \\([0, 1]\\) | Down | top-left | + +(We will defer the discussion of GL's depth range difference to section 2.3.) + +### 2.1. Vulkan y-axis correction + +The XY ranges for all backends are the same, but Vulkan interprets the Y-Axis direction as +"Down", meaning that \\(y=1\\) is at the bottom of the image. To account for this difference, +Filament flips the projected \\(y\\) coordinate in the vertex shader for Vulkan + +```c++ +#if defined(TARGET_VULKAN_ENVIRONMENT) + // In Vulkan, clip space is Y-down. In OpenGL and Metal, clip space is Y-up. + position.y = -position.y; +#endif +``` + +### 2.2 Fix Frame buffer origin difference with `uvToRenderTargetUV` + +After the workaround in the above section, we can rewrite the table thus + +| Graphics API | Y-Axis Direction | Frame Buffer Origin | +| -------------- | ---------------- | ------------------- | +| OpenGL / WebGL | Up | bottom-left | +| Metal | Up | top-left | +| WebGPU | Up | top-left | +| Vulkan | Up (corrected) | top-left | + +We can rephrase the Y-Axis Direction column to say that + +| | +|---| +| *Filament's internal clip-space defines the Y-axis direction as **UP** (i.e. the physical world up direction pointing to \\(+1\\)).* | + +Now consider the "Frame Buffer Origin" column. This column identifies the origin of the rendered +image in the context of sampling by subsequent passes. + +Assume + - We have two passes \\(A\\) and \\(B\\) + - \\(A\\) renders to a render target that attaches texture \\(T\\) + - \\(B\\) renders a full-screen quad + - In \\(B\\), top-left and bottom-right corners of the quad are assigned texturing UV coordinates + \\((0, 0)\\) and \\((1, 1)\\), respectively. + - The quad in \\(B\\) samples texture \\(T\\). + +For GL, having the *bottom-left* corner as the origin means that rendering a pixel to clip-space +\\(x=-1, y=-1\\) in \\(A\\) will output to pixel at \\(u=0, v=0\\) of \\(T\\). Note that GL +clip-space's y-axis direction is consistent with the texture space v-axis direction. That is, in +pass \\(B\\), sampling from \\(v=0\\) to \\(v=1\\) (moving in the positive direction) maps to going +from \\(y=-1\\) to \\(y=1\\) in clip-space (also moving in positive direction). + +This behavior is opposite for all other backends; sampling from \\(v=0\\) to \\(v=1\\) will map to +clip-space movement from \\(y=1\\) to \\(y=-1\\) (going in the negative direction). + +To account for this, Filament provides the following shader method that clients can call to +correctly sample from a texture that was the rendertarget of a previous pass. + +```c++ +highp vec2 uvToRenderTargetUV(const highp vec2 uv) { +#if defined(TARGET_METAL_ENVIRONMENT) || defined(TARGET_VULKAN_ENVIRONMENT) || defined(TARGET_WEBGPU_ENVIRONMENT) + return vec2(uv.x, 1.0 - uv.y); +#else + return uv; +#endif +} +``` + +It's meant to be used like so + +```c++ +vec2 uv0 = uvToRenderTargetUV(getUV0()); +material.baseColor = texture(materialParams_baseColor, uv0); +``` + +### 2.3. Clip-space depth adjustments for OpenGL + +While the clip-space depth (or Z) dimension does not pertain to the 2D coordinate spaces that we have +been discussing, the concept in this section parallels the space adjustment logic in the other +sections. Hence, we have included it here. + +Consider again this table + +| Graphics API | Depth Range (Z) | +| -------------- | --------------- | +| OpenGL / WebGL | \\([−1, 1]\\) | +| Metal | \\([0, 1]\\) | +| WebGPU | \\([0, 1]\\) | +| Vulkan | \\([0, 1]\\) | + +For OpenGL, the NDC depth dimension ranges from \\(-1\\) to \\(1\\). Filament implements two +paths to adjust this so that the internal clip-space definition remains consistent: + +1. Use the `EXT_clip_control` extension to change the depth range to \\([0, 1]\\). + + ```c++ + #if !defined(__EMSCRIPTEN__) + if (ext.EXT_clip_control) { + #if defined(BACKEND_OPENGL_VERSION_GL) + glClipControl(GL_LOWER_LEFT, GL_ZERO_TO_ONE); + #elif defined(GL_EXT_clip_control) + glClipControlEXT(GL_LOWER_LEFT_EXT, GL_ZERO_TO_ONE_EXT); + #endif + } + #endif + ``` + +2. For cases where the extension is not available, the following shader code will convert the + internal (or noted as virtual in the code) coordinates to GL's expected depth coordinates. + + ```c++ + #if !defined(TARGET_VULKAN_ENVIRONMENT) && !defined(TARGET_METAL_ENVIRONMENT) && !defined(TARGET_WEBGPU_ENVIRONMENT) + // This is not needed in Vulkan, Metal or WebGPU because clipControl is always (1, 0) + position.z = position.z * frameUniforms.clipControl.x + + position.w * frameUniforms.clipControl.y; + #endif + ``` + + This code uses a vector in the uniform buffer to scale and translate the resulting depth. This + vector is written in runtime (so that we'd know whether the extension is present or not).