Configs are organized as configs/<model>/<benchmark>/<prototype>.yaml:
configs/
4omini/ # gpt-4o-mini
qwen2.5-7b/ # Qwen2.5-7B-Instruct
qwen27b/ # Qwen3.5-27B-AWQ-4bit
alfworld/ sas.yaml sas_contextflow.yaml mas.yaml sas_plan.yaml sas_plan_fewshot.yaml mas_relay_insufficient.yaml env_config.yaml env_config_hetero.yaml
webshop/ sas.yaml sas_contextflow.yaml mas.yaml sas_plan.yaml mas_relay_insufficient.yaml
workbench/ sas.yaml sas_contextflow.yaml mas.yaml
widesearch/ sas.yaml sas_contextflow.yaml mas.yaml
travelplanner/ sas.yaml sas_contextflow.yaml mas.yaml sas_plan.yaml
sas / sas_contextflow / mas / sas_plan correspond to the four prototypes in the paper (Section 5.1);
mas_relay_insufficient and sas_plan_fewshot are the two ablation studies (Section 6).
Each config needs:
- API key — set
llm.api_key, or exportOPENAI_API_KEY/ANTHROPIC_API_KEY(picked up automatically ifapi_keyis left unset). - Data paths — configs reference data under
${MAS_SAS_DATA_DIR}; export that env var to point at your local copy of the ALFWorld/WebShop/WorkBench/TravelPlanner data (see repo root README for dataset setup). - (Optional)
--max-tasks,--experiment-id,--output-dircan be overridden on the command line without editing the YAML.
Point llm.base_url at your running vLLM server (must end with /v1):
llm:
model: Qwen/Qwen2.5-7B-Instruct # or cyankiwi/Qwen3.5-27B-AWQ-4bit
api_key: dummy
base_url: http://HOST:PORT/v1
default_temperature: 0.0
max_retries: 1
extra_body:
chat_template_kwargs:
enable_thinking: falsepython main.py run --config configs/qwen2.5-7b/alfworld/sas.yaml --experiment-id my_alfworld_run