Skip to content

Commit 6d772fb

Browse files
committed
comskip: fix SIGSEGV when lowres=1 on x86_64 MPEG-2 content
codecCtx->lowres was being set after avcodec_open2, but ff_idctdsp_init runs during avcodec_open2 and — seeing lowres=0 — installs the full 8x8 AVX IDCT (ff_simple_idct8_put_avx). With lowres=1 set post-open, frame buffers are half-sized while the IDCT still writes 8 rows, causing an out-of-bounds write and intermittent SIGSEGV. Fix: set codecCtx->lowres before avcodec_open2 so ff_idctdsp_init correctly selects the 4x4 IDCT (ff_jref_idct4_put) for lowres=1.
1 parent 0374038 commit 6d772fb

1 file changed

Lines changed: 7 additions & 1 deletion

File tree

mpeg2dec.c

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1837,6 +1837,12 @@ int stream_component_open(VideoState *is, int stream_index)
18371837

18381838
if (!hardware_decode) av_dict_set_int(&myoptions, "gray", 1, 0);
18391839

1840+
// lowres must be set before avcodec_open2 so ff_idctdsp_init selects the
1841+
// correct IDCT (e.g. 4x4 for lowres=1). Setting it after open causes the
1842+
// full 8x8 AVX IDCT to be used while frame buffers are half-sized, leading
1843+
// to an out-of-bounds write and SIGSEGV.
1844+
if (codec && codecCtx->codec_type == AVMEDIA_TYPE_VIDEO)
1845+
codecCtx->lowres = min(codec->max_lowres, lowres);
18401846

18411847
// av_dict_set_int(&myoptions, "fastint", 1, 0);
18421848
// av_dict_set_int(&myoptions, "skip_alpha", 1, 0);
@@ -1887,7 +1893,7 @@ int stream_component_open(VideoState *is, int stream_index)
18871893
is->pFrame = av_frame_alloc();
18881894
if (!hardware_decode) codecCtx->flags |= AV_CODEC_FLAG_GRAY;
18891895
// codecCtx->thread_type = 1; // Frame based threading
1890-
codecCtx->lowres = min(codecCtx->codec->max_lowres, lowres);
1896+
// codecCtx->lowres = min(codecCtx->codec->max_lowres, lowres); // moved before avcodec_open2
18911897
if (codecCtx->codec_id == AV_CODEC_ID_H264)
18921898
{
18931899
is_h264 = 1;

0 commit comments

Comments
 (0)