Skip to content

technology_selection_en.md

Latest commit

 

History

History
393 lines (246 loc) · 11.5 KB

technology_selection_en.md

File metadata and controls

393 lines (246 loc) · 11.5 KB
Raw

MARGPA Runtime LLM Technology Selection

document_type: public_technology_selection
document_state: current
language: en
created_at: 2026-07-31
updated_at: 2026-07-31
public_author: Nazuna Research
project: MARGPA Runtime LLM

1. Overview

The current configuration is a lightweight LLM Runtime centered on Python, running Local LLMs in GGUF format with llama.cpp, providing CLI／Web UI with FastAPI＋Vanilla JavaScript, and supporting macOS Metal, Linux CPU, and CUDA. Some parts are planned to be reconfigured as development progresses.

This Document describes the technologies currently used in MARGPA Runtime LLM and the general technologies currently planned for introduction.

Internal design, the structure of original functions, and undecided detailed design are outside the scope of this Document.

2. Development Languages

2.1 Python

Python is used as the primary language for the Application, LLM Runtime, CLI, Web API, configuration loading, RAG, and Tests.

The supported range is Python 3.12 or later and earlier than 3.14.

Environment Version Selection Policy
macOS Python 3.13.14 Adopted in consideration of the balance between the support period of the Python 3.13 series and compatibility with the Libraries in use
Lightning Linux Python 3.12.11 Maintained as the Version on which the Lightning environment was constructed and operation was verified

Because the verified Version differs by environment, a single Python Version is not fixed at the Repository Root and is instead managed through the construction procedures and settings of each execution environment.

2.2 JavaScript, HTML, and CSS

HTML, CSS, and Vanilla JavaScript are used for the Web UI.

Frontend Frameworks such as React, Vue, and Svelte are not currently used.

2.3 Bash

Bash Scripts are used for environment construction, startup, shutdown, Health Checks, and other operational assistance in macOS and Linux environments.

3. Package and Build Management

3.1 uv

uv is used for Python environment, Dependency, and Build management.

The main management Files are as follows.

  • pyproject.toml
  • uv.lock

As a general rule, Dependency Versions are fixed to ensure reproducibility.

3.2 uv_build

uv_build is used as the Build Backend for the Python Package.

4. LLM Runtime

4.1 llama.cpp

llama.cpp is used as the inference Backend for Local LLMs.

llama-cpp-python is used to access it from Python.

Its main uses are as follows.

  • Model loading
  • Chat Completion
  • Streaming generation
  • Retrieval of Token usage
  • Generation stop
  • Processing allocation to CPU and GPU

4.2 GGUF

GGUF is used as the Artifact format for Local Models.

GGUF is used for Local inference with llama.cpp and the operation of quantized Models.

4.3 Model Configuration

The following are the Models currently adopted or planned for introduction and may be changed depending on verification results, development status, and execution environments.

Role Model Status
Main Qwen3-4B-Q4_K_M.gguf Currently in use
Guardrail Model Qwen.Qwen3Guard-Gen-0.6B.Q8_0.gguf Planned for introduction
LLM-as-a-Judge Selene-1-Mini-Llama-3.1-8B-Q5_K_M.gguf Planned for introduction

Model Artifacts are not duplicated within the Repository. They are centrally managed outside the Repository and referenced by each Project.

5. CLI and Web

5.1 CLI

The CLI is configured using argparse from the Python Standard Library.

The configuration allows a Local LLM to be executed from the CLI by specifying the Model, execution Profile, and generation settings.

5.2 FastAPI

FastAPI is used for the Web API.

Its main uses are as follows.

  • Acceptance of Chat Requests
  • Provision of Runtime information
  • Streaming Responses
  • Generation stop
  • Health Check

5.3 Uvicorn

Uvicorn is used as the ASGI Server for the FastAPI Application.

5.4 Server-Sent Events

Server-Sent Events are used for LLM Streaming Responses.

Generated content is sent sequentially to the Browser and displayed as a Stream in the Web UI.

5.5 Vanilla JavaScript

The current Web UI is built with Vanilla JavaScript.

Streamlit has not been adopted because substantial changes and expansion of the UI are planned in the future, and flexibility in the UI structure must be maintained.

Migration to or addition of Frontend technologies such as React will be considered as development progresses.

6. Configuration and Data Validation

6.1 TOML

TOML is used for Application settings, Model settings, RAG settings, and Profiles for each execution environment.

The main managed items are as follows.

  • Model
  • Context Size
  • Generation settings
  • Runtime settings
  • CPU and GPU settings
  • Backend settings
  • RAG settings
  • Execution environment Profile

6.2 tomllib

tomllib from the Python Standard Library is used to read TOML.

6.3 Pydantic

Pydantic is used for settings, API Contracts, and Data Validation.

Its main uses are as follows.

  • Type validation
  • Schema Validation
  • Definition of API Requests and Responses
  • Validation of setting values

7. Hardware and Execution Environments

7.1 macOS Metal

In macOS environments equipped with Apple Silicon, Metal is used to execute LLM inference.

7.2 Linux CPU

Linux environments support CPU execution without using a GPU.

7.3 NVIDIA CUDA

In Linux environments where an NVIDIA GPU is available, LLM inference using CUDA is supported.

7.4 Lightning AI

Lightning AI is used for the Public Demo and as an external Linux execution environment.

CPU and CUDA configurations are assumed for the Lightning AI environment.

8. RAG

8.1 Scope

A RAG function targeting Local Documents is implemented and used for Document search and reference to related information.

The initial configuration targets the macOS side and searches Documentation in Markdown format.

8.2 Basic Configuration

The initial Documentation RAG adopts Lexical Retrieval centered on the Python Standard Library.

Category Adopted Technology
Language Python 3.12／3.13
Document Markdown／UTF-8
Normalization unicodedata／re
File and Integrity pathlib／hashlib／SHA-512
Chunking Heading-aware deterministic Parser
Retrieval Japanese character n-grams, Latin／Identifier Tokens
Ranking BM25-style Ranking
Index In-Memory Snapshot
Config TOML／tomllib／Pydantic
Web FastAPI／Server-Sent Events／Vanilla JavaScript
Test pytest／mypy／Ruff
New Runtime Dependency None
New Model None

8.3 Document Processing

The initial Reader targets Markdown UTF-8.

The following structures are mainly handled as Text Data.

  • ATX Heading
  • Paragraph
  • Blank Line
  • Fenced Code Block
  • List

HTML, JavaScript, or Links within Documents are not executed.

8.4 Normalization and Search Terms

Unicode NFKC is used for normalization before searching.

Character 2-grams／3-grams are used for Japanese searches, and Latin／digit／underscore／dot／slash／hyphen Tokens are also used for alphanumeric characters, File names, Paths, and Identifiers.

A morphological analyzer is not an initial required Dependency.

8.5 Ranking

A BM25-style method is used for Ranking search results.

The initial Parameters are based on the following.

k1 = 1.5
b = 0.75
top_k = 4
max_chunks_per_document = 2

Weights for Headings, body text, Paths, exact matches, and similar items can be adjusted through Config.

8.6 Index

The initial configuration does not use a Persistent Index and instead uses an Index that can be rebuilt In-Memory.

This avoids increasing the management of additional Services, Migrations, and generated Index Files at the initial stage.

SHA-512 is used to verify the identity of Documents and the Corpus.

8.7 Technologies Currently on Hold

The following technologies are not permanently rejected.

Their introduction into the initial Documentation RAG is on hold because the additional load and operational scope are large under the current Hardware, Memory, execution environments, and implementation stage.

  • LangChain
  • LangGraph
  • LlamaIndex
  • Sentence Transformers
  • Transformers
  • PyTorch
  • MLX／MLX-LM
  • Embedding Model
  • llama.cpp Embedding
  • SQLite FTS5
  • FAISS
  • Chroma
  • Qdrant
  • Other Vector Stores
  • rank-bm25
  • scikit-learn

Dense Retrieval, Vector Stores, and Hybrid Retrieval will be re-evaluated if synonyms, paraphrases, or conceptual search become primary requirements, if the Corpus grows substantially, if the target Document formats are expanded, or if computing resources capable of continuously running additional Models are secured.

9. Security-Related Technologies

General Security measures in the Public Demo and Preview environments use the following technologies.

  • HTTP Basic authentication
  • Credential management through Environment Variables
  • Security Headers
  • Content Security Policy
  • Safe Markdown Rendering
  • Bind control for external publication

The Guardrail Model and additional Security functions will be introduced sequentially based on the Roadmap.

10. Tests and Quality Management

10.1 pytest

pytest is used as the Python Test Framework.

pytest-asyncio and pytest-cov are used for Async processing and Coverage checks.

10.2 httpx

httpx is used for FastAPI Endpoint and Web Integration Tests.

10.3 mypy

mypy is used for Python Static Type Checking.

10.4 Ruff

Ruff is used for Python Linting and Import organization.

10.5 Node.js Test Runner

The Node.js Standard Test Runner is used for Tests of some Frontend processing implemented with Vanilla JavaScript.

11. Notebook Environment

JupyterLab, Jupyter Notebook, and ipykernel may be used for verification of external environments and the Runtime.

Notebooks are not a required execution environment for the Application and are used mainly for development and verification.

12. Future Candidates and Technologies on Hold

The following technologies are retained as candidates according to future functions, execution environments, or operational requirements.

At present, they are kept on hold rather than fixed in advance because computing resources, implementation progress, and necessity have not been determined.

Category Candidate
Frontend React, Next.js
Runtime for Apple Silicon MLX, mlx-lm
Model Runtime Transformers, PyTorch
Cloud GPU Inference vLLM
Hybrid Configuration Remote Inference API
Local Storage／Search SQLite
Server Storage PostgreSQL
Cloud Storage Object Storage
Container Docker
Cloud AWS, Azure

Retaining candidates and future connectivity and fixing them as currently adopted technologies are treated separately.

Specific technologies for Agent, Tool, Evaluation, persistent Conversation, and other unimplemented functions will be selected once detailed design and necessary conditions have been determined.

13. Future Changes

This Technology Selection is based on the current implementation, Accepted Decisions, and planned introductions.

The adopted technologies, Versions, and configuration may change due to the following factors.

  • Development progress
  • Test results
  • Library support status
  • Hardware and execution environments
  • Security requirements
  • Performance
  • Maintainability
  • Expansion of the UI and functions
  • Computing resources
  • Corpus Size and search requirements

Specific technologies for unimplemented functions will be added after detailed design and implementation details have been determined.