Skip to content

fix: honor EXIF orientation when decoding caller-supplied images #184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hm21 merged 3 commits into from
Jul 30, 2026
Merged

fix: honor EXIF orientation when decoding caller-supplied images #184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5ec8703
fix: honor EXIF orientation when decoding caller-supplied images
hm21 Jul 30, 2026
480abc3
fix: address review findings on EXIF orientation decoding
hm21 Jul 30, 2026
56662cb
fix: simplify video composition comments and remove unsupported features
hm21 Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
## 2.11.1
- **FIX**(android, iOS, macOS): A `ChromaKey.backgroundImage` or image `VideoLayer` fed a photo straight from the gallery no longer renders sideways — the EXIF `Orientation` tag is now honored on all three platforms.

## 2.11.0
- **FEAT**(android, iOS, macOS): Add `ChromaKey` — green-screen removal with a soft edge and spill suppression. The keyed area becomes a `backgroundColor`, a `backgroundImage`, or (in a `VideoComposition`) the layer below. Settable on `VideoRenderData`, `VideoLayer` and `VideoSegment`, resolved per clip as segment → layer → global. H.264/HEVC carry no alpha, so on the single-track `videoSegments` path a key without a background is flattened to black.
- **FEAT**: `ChromaKey.autoDetect(video)` measures the key color and `similarity` off the footage, for any saturated screen hue. `ChromaKey.detect` returns the raw measurement. Costs one thumbnail decode, no render.
Expand Down
7 changes: 6 additions & 1 deletion android/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,12 @@ android {
implementation("androidx.media3:media3-inspector-frame:$media3_version")
implementation("androidx.media3:media3-effect:$media3_version")
implementation("androidx.media3:media3-muxer:$media3_version")


// EXIF metadata. Preferred over android.media.ExifInterface: it covers
// HEIF/AVIF/WebP/PNG as well as JPEG, and it is plain Java, so the
// orientation parsing is reachable from the JVM unit tests.
implementation("androidx.exifinterface:exifinterface:1.4.2")

// Coroutines
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.7.3")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.7.3")
Expand Down
47 changes: 41 additions & 6 deletions ...d/src/main/kotlin/ch/waio/pro_video_editor/src/features/render/helpers/ApplyImageLayer.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package ch.waio.pro_video_editor.src.features.render.helpers

import RENDER_TAG
import android.graphics.Bitmap
import android.graphics.BitmapFactory
import android.graphics.Matrix
import androidx.media3.common.Effect
import androidx.media3.common.util.UnstableApi
Expand All @@ -15,6 +14,7 @@ import androidx.media3.effect.StaticOverlaySettings
import androidx.media3.effect.TimestampWrapper
import ch.waio.pro_video_editor.src.features.render.models.ImageLayer
import ch.waio.pro_video_editor.src.shared.logging.PluginLog as Log
import ch.waio.pro_video_editor.src.shared.media.ImageOrientation

/**
* Applies static image overlay on video.
Expand Down Expand Up @@ -148,12 +148,25 @@ fun applyTimedImageLayers(
"Layer: animated GIF with ${gifFrames.size} frame(s), loop=${layer.loop}"
)
} else {
val options = BitmapFactory.Options().apply {
inPreferredConfig = Bitmap.Config.ARGB_8888
}
val layerBitmap = BitmapFactory.decodeByteArray(
imageBytes, 0, imageBytes.size, options
// Decoded through ImageOrientation so a gallery photo carrying an
// EXIF orientation is laid in the way the user sees it, not as the
// sideways pixels it is stored as.
//
// `prepareOverlay` scales the result down straight away, so it is
// decoded no larger than it will end up: a full-resolution decode
// of a phone photo costs a second full-resolution copy when the
// orientation has to be turned, which is where an overlay OOMs.
val (reqWidth, reqHeight) = overlayDecodeSize(layer, videoWidth, videoHeight)
val layerBitmap = ImageOrientation.decode(
imageBytes,
reqWidth = reqWidth,
reqHeight = reqHeight,
config = Bitmap.Config.ARGB_8888,
)
if (layerBitmap == null) {
Log.e(RENDER_TAG, "Layer: image bytes did not decode; skipping layer")
continue
}
val prepared = prepareOverlay(layerBitmap, layer, videoWidth, videoHeight)

bitmapOverlay = if (hasAnimations) {
Expand Down Expand Up @@ -208,6 +221,28 @@ private data class PreparedOverlay(
val overlaySettings: StaticOverlaySettings,
)

/**
* The size [prepareOverlay] will first scale a decoded layer down to, in
* displayed pixels, or `0 × 0` when it keeps the image at its natural size.
*
* Mirrors [prepareOverlay]'s own branching, so decoding to this size cannot cost
* resolution the overlay would otherwise have kept. A layer with an explicit
* size is scaled to it; a layer with neither an explicit size nor a position is
* stretched over the whole frame; a positioned layer without an explicit size is
* laid out from its own pixel dimensions and so must not be sampled down.
*/
internal fun overlayDecodeSize(
layer: VideoSequenceBuilder.ImageLayerConfig,
videoWidth: Int,
videoHeight: Int,
): Pair<Int, Int> {
val width = layer.width
val height = layer.height
if (width != null && height != null) return Pair(width.toInt(), height.toInt())
if (layer.x == null && layer.y == null) return Pair(videoWidth, videoHeight)
return Pair(0, 0)
}

/**
* Scales, positions, unpremultiplies and rotates a single overlay [rawBitmap]
* according to [layer], returning the final bitmap plus its anchor/settings.
Expand Down
36 changes: 19 additions & 17 deletions ...d/src/main/kotlin/ch/waio/pro_video_editor/src/features/render/helpers/ChromaKeyEffect.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import androidx.media3.effect.GlEffect
import androidx.media3.effect.GlShaderProgram
import ch.waio.pro_video_editor.src.features.render.models.ChromaKeyConfig
import ch.waio.pro_video_editor.src.shared.logging.PluginLog as Log
import ch.waio.pro_video_editor.src.shared.media.ImageOrientation

/**
* Removes a solid-colored background ("green screen") from every frame.
Expand Down Expand Up @@ -70,17 +71,13 @@ class ChromaKeyEffect(private val config: ChromaKeyConfig) : GlEffect {
* Bounds-only probe of the background image.
*
* Decoding just the header tells us whether the bytes are usable — and
* how large they are — without holding a full-size bitmap from
* construction until the first draw. The real decode happens in
* [drawFrame], where there is a GL context to size it against.
* how large they are once their EXIF orientation is honored — without
* holding a full-size bitmap from construction until the first draw. The
* real decode happens in [drawFrame], where there is a GL context to
* size it against.
*/
private val backgroundBounds: BitmapFactory.Options? =
config.backgroundImageData?.let { bytes ->
BitmapFactory.Options().apply {
inJustDecodeBounds = true
BitmapFactory.decodeByteArray(bytes, 0, bytes.size, this)
}.takeIf { it.outWidth > 0 && it.outHeight > 0 }
}
private val backgroundProbe: ImageOrientation.Probe? =
config.backgroundImageData?.let { ImageOrientation.probe(it) }

/**
* Background texture id. A 1x1 placeholder is uploaded when there is no
Expand All @@ -90,7 +87,7 @@ class ChromaKeyEffect(private val config: ChromaKeyConfig) : GlEffect {
private var backgroundTexId: Int = -1

private val bgMode: Int = when {
backgroundBounds != null -> BG_IMAGE
backgroundProbe != null -> BG_IMAGE
config.backgroundColor != null -> BG_COLOR
// An image was asked for but its bytes will not decode. Falling
// through to BG_TRANSPARENT would quietly un-key the clip on the
Expand Down Expand Up @@ -180,7 +177,7 @@ class ChromaKeyEffect(private val config: ChromaKeyConfig) : GlEffect {
}

init {
if (backgroundBounds == null && config.backgroundImageData != null) {
if (backgroundProbe == null && config.backgroundImageData != null) {
Log.w(
RENDER_TAG,
"Chroma key: the background image could not be decoded; " +
Expand Down Expand Up @@ -287,21 +284,26 @@ class ChromaKeyEffect(private val config: ChromaKeyConfig) : GlEffect {
*/
private fun decodeBackgroundWithinTextureLimit(): Bitmap? {
val bytes = config.backgroundImageData ?: return null
val bounds = backgroundBounds ?: return null
val probe = backgroundProbe ?: return null

val limit = maxTextureSize()
var sampleSize = 1
while (bounds.outWidth / sampleSize > limit ||
bounds.outHeight / sampleSize > limit
) {
// `probe` is oriented and `inSampleSize` applies to the stored pixels,
// but an orientation only ever exchanges the two dimensions, so the
// pair is over the limit either way round.
while (probe.width / sampleSize > limit || probe.height / sampleSize > limit) {
sampleSize *= 2
}

val decoded = BitmapFactory.decodeByteArray(
val raw = BitmapFactory.decodeByteArray(
bytes, 0, bytes.size,
BitmapFactory.Options().apply { inSampleSize = sampleSize }
) ?: return null

// A portrait photo is stored as landscape pixels plus an EXIF tag;
// without this the background would be keyed in sideways.
val decoded = ImageOrientation.orient(raw, probe.orientation)

// inSampleSize only halves, so one more exact pass may be needed.
if (decoded.width <= limit && decoded.height <= limit) return decoded
val scale = minOf(
Expand Down
95 changes: 6 additions & 89 deletions ...d/src/main/kotlin/ch/waio/pro_video_editor/src/features/stopmotion/StopMotionGenerator.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,6 @@ package ch.waio.pro_video_editor.src.features.stopmotion
import RENDER_TAG
import android.content.Context
import android.graphics.Bitmap
import android.graphics.BitmapFactory
import android.graphics.Matrix
import android.media.ExifInterface
import android.net.Uri
import android.os.Handler
import android.os.Looper
Expand All @@ -30,7 +27,7 @@ import applyBitrate
import ch.waio.pro_video_editor.src.features.render.models.RenderJobHandle
import ch.waio.pro_video_editor.src.features.stopmotion.models.StopMotionConfig
import ch.waio.pro_video_editor.src.shared.logging.PluginLog as Log
import java.io.ByteArrayInputStream
import ch.waio.pro_video_editor.src.shared.media.ImageOrientation
import java.io.File
import java.util.concurrent.atomic.AtomicBoolean
import java.util.concurrent.atomic.AtomicReference
Expand Down Expand Up @@ -105,9 +102,9 @@ class StopMotionGenerator(private val context: Context) {
var targetWidth = config.width ?: 0
var targetHeight = config.height ?: 0
if (targetWidth <= 0 || targetHeight <= 0) {
val (w, h) = orientedBounds(config.frames[0].imageData)
targetWidth = w
targetHeight = h
val probe = ImageOrientation.probe(config.frames[0].imageData)
targetWidth = probe?.width ?: 2
targetHeight = probe?.height ?: 2
}
targetWidth = evenize(targetWidth)
targetHeight = evenize(targetHeight)
Expand All @@ -117,7 +114,8 @@ class StopMotionGenerator(private val context: Context) {
config.frames.forEachIndexed { index, frame ->
if (shouldStopPolling.get()) return@Thread

val bitmap = decodeOriented(frame.imageData, targetWidth, targetHeight)
val bitmap = ImageOrientation
.decode(frame.imageData, targetWidth, targetHeight)
?: throw IllegalStateException("Failed to decode frame $index")

val file = File(
Expand Down Expand Up @@ -298,87 +296,6 @@ class StopMotionGenerator(private val context: Context) {
})
}

/** Returns the orientation-corrected pixel size of an encoded image. */
private fun orientedBounds(data: ByteArray): Pair<Int, Int> {
val opts = BitmapFactory.Options().apply { inJustDecodeBounds = true }
BitmapFactory.decodeByteArray(data, 0, data.size, opts)
val w = if (opts.outWidth > 0) opts.outWidth else 2
val h = if (opts.outHeight > 0) opts.outHeight else 2
return when (readExifOrientation(data)) {
ExifInterface.ORIENTATION_ROTATE_90,
ExifInterface.ORIENTATION_ROTATE_270,
ExifInterface.ORIENTATION_TRANSPOSE,
ExifInterface.ORIENTATION_TRANSVERSE -> Pair(h, w)
else -> Pair(w, h)
}
}

/**
* Decodes an encoded image downscaled to roughly [reqW]×[reqH] and applies
* its EXIF orientation, so portrait photos are not rendered sideways.
*/
private fun decodeOriented(data: ByteArray, reqW: Int, reqH: Int): Bitmap? {
val bounds = BitmapFactory.Options().apply { inJustDecodeBounds = true }
BitmapFactory.decodeByteArray(data, 0, data.size, bounds)
if (bounds.outWidth <= 0 || bounds.outHeight <= 0) return null

val opts = BitmapFactory.Options().apply {
inSampleSize = calcInSampleSize(bounds.outWidth, bounds.outHeight, reqW, reqH)
}
val raw = BitmapFactory.decodeByteArray(data, 0, data.size, opts) ?: return null
return applyOrientation(raw, readExifOrientation(data))
}

private fun readExifOrientation(data: ByteArray): Int {
return try {
ExifInterface(ByteArrayInputStream(data))
.getAttributeInt(
ExifInterface.TAG_ORIENTATION, ExifInterface.ORIENTATION_NORMAL
)
} catch (e: Exception) {
ExifInterface.ORIENTATION_NORMAL
}
}

private fun applyOrientation(bitmap: Bitmap, orientation: Int): Bitmap {
val matrix = Matrix()
when (orientation) {
ExifInterface.ORIENTATION_ROTATE_90 -> matrix.postRotate(90f)
ExifInterface.ORIENTATION_ROTATE_180 -> matrix.postRotate(180f)
ExifInterface.ORIENTATION_ROTATE_270 -> matrix.postRotate(270f)
ExifInterface.ORIENTATION_FLIP_HORIZONTAL -> matrix.postScale(-1f, 1f)
ExifInterface.ORIENTATION_FLIP_VERTICAL -> matrix.postScale(1f, -1f)
ExifInterface.ORIENTATION_TRANSPOSE -> {
matrix.postRotate(90f)
matrix.postScale(-1f, 1f)
}
ExifInterface.ORIENTATION_TRANSVERSE -> {
matrix.postRotate(270f)
matrix.postScale(-1f, 1f)
}
else -> return bitmap
}
val rotated = Bitmap.createBitmap(
bitmap, 0, 0, bitmap.width, bitmap.height, matrix, true
)
if (rotated != bitmap) bitmap.recycle()
return rotated
}

/** Largest power-of-two sample size that keeps the image ≥ the target size. */
private fun calcInSampleSize(srcW: Int, srcH: Int, reqW: Int, reqH: Int): Int {
if (reqW <= 0 || reqH <= 0) return 1
var sample = 1
var halfW = srcW / 2
var halfH = srcH / 2
while (halfW >= reqW && halfH >= reqH) {
sample *= 2
halfW /= 2
halfH /= 2
}
return sample
}

/** Rounds a dimension down to the nearest even value, minimum 2. */
private fun evenize(value: Int): Int {
val v = if (value <= 0) 2 else value
Expand Down
Loading
Loading