feat: vlm bench warehouse tasks - #668

Open

MagdalenaKotynia wants to merge 18 commits into

mainfrom

mk/feat/vlm-bench-warehouse-tasks

Commits on Aug 18, 2025

feat: interfaces for quantity and multiple choice tasks
MagdalenaKotynia
committed
e3fdd47
feat: created new tasks with warehouse simulation images
MagdalenaKotynia
committed
143e7c3
feat: added vlm bench config to test many models
MagdalenaKotynia
committed
e5652c0
chore: moved old image files from vlm benchmark to lfs
MagdalenaKotynia
committed
2b2e360
refactor: extracted common logic for vlm tasks inputs and answers to base ImageReasoningTaskInput and ImageReasoningAnswer classes
MagdalenaKotynia
committed
d566994
docs: added info about increasing event size for Langfuse tracing of vlm benchmark
MagdalenaKotynia
committed
02b459b
feat: added merging results summaries across all repeats and all models
MagdalenaKotynia
committed
466ec59
fix: fixed typing after refactor extracting common logic for vlm tasks inputs and answers
MagdalenaKotynia
committed
1adc3d4
refactor: reorganized tasks order by images
MagdalenaKotynia
committed
f791e85
chore: removed unused comment
MagdalenaKotynia
committed
fe1b5a8
refactor: created separate structure for storing summary for the model across all tasks and repeats
MagdalenaKotynia
committed
90d163b
fix: fixed validation of parsed llm output
MagdalenaKotynia
committed
01179b6
refactor: added task id, refactor of storing task_input variables
MagdalenaKotynia
committed
30c6b6a
fix: aggregate tasks repeats results by the task id
MagdalenaKotynia
committed
084b484
refactor: renamed csv storing model summary
MagdalenaKotynia
committed
8e3c01c
feat: added std success rate and std time for model across all repeats
MagdalenaKotynia
committed
5417762

Commits on Aug 27, 2025

Merge branch 'main' into mk/feat/vlm-bench-warehouse-tasks
maciejmajek
authored
cab9ed6
fix: image paths
maciejmajek
committed
17ba499

Navigation Menu

Uh oh!

feat: vlm bench warehouse tasks - #668

feat: vlm bench warehouse tasks#668
MagdalenaKotynia wants to merge 18 commits into
mainfrom
mk/feat/vlm-bench-warehouse-tasks

Commits on Aug 18, 2025

feat: interfaces for quantity and multiple choice tasks

feat: created new tasks with warehouse simulation images

feat: added vlm bench config to test many models

chore: moved old image files from vlm benchmark to lfs

refactor: extracted common logic for vlm tasks inputs and answers to base ImageReasoningTaskInput and ImageReasoningAnswer classes

docs: added info about increasing event size for Langfuse tracing of vlm benchmark

feat: added merging results summaries across all repeats and all models

fix: fixed typing after refactor extracting common logic for vlm tasks inputs and answers

refactor: reorganized tasks order by images

chore: removed unused comment

refactor: created separate structure for storing summary for the model across all tasks and repeats

fix: fixed validation of parsed llm output

refactor: added task id, refactor of storing task_input variables

fix: aggregate tasks repeats results by the task id

refactor: renamed csv storing model summary

feat: added std success rate and std time for model across all repeats

Commits on Aug 27, 2025

Merge branch 'main' into mk/feat/vlm-bench-warehouse-tasks

fix: image paths