-Norma is a cloud-agnostic, LLM-agnostic platform that uses [LiteLLM](https://docs.litellm.ai/) to abstract away model providers. Any LLM backend — Vertex AI, OpenAI, Azure, or others — can be swapped without code changes.
+
+ AI compliance, simplified
+
+
+
+
+
+
+
+
+
+
+
+---
+
+
+
+## What is Norma?
+
+Norma helps teams achieve **EU AI Act** compliance by combining structured project management with an AI-powered assistant that understands your project context, uploaded documents, codebase, and compliance obligations to guide you through risk classification, evidence gathering, and reporting.
+
+Norma is **cloud-agnostic** and **LLM-agnostic**. It uses [LiteLLM](https://docs.litellm.ai/) to abstract model providers, so any backend (Vertex AI, OpenAI, Azure, or others) can be swapped without code changes.
+
+## Features
+
+**Risk Classification.** Interactive questionnaire following the EU AI Act decision tree to determine your AI system's risk level.
+
+**Compliance Reporting.** Structured checklists with evidence tracking and validation to build your EU AI Act compliance case.
+
+**Document Management.** Upload compliance documents as PDFs. Norma extracts and summarises the content so the AI assistant can reference it in context.
+
+**Norma AI Assistant.** A context-aware chat assistant that understands your project details, uploaded documents, codebase, and reporting progress to provide tailored compliance guidance.
+
+**Codebase Analysis.** Connect a GitHub repository to generate architecture diagrams and technical summaries that feed into compliance assessments.
+
+**Multi-Language Support.** Full interface localisation in English, Spanish, French, and German, with AI responses adapted to the user's preferred language.
+
+**LLM-Agnostic.** Swap between any LLM provider (Vertex AI, OpenAI, Azure, local models) by changing a single environment variable.
## Architecture
@@ -22,83 +61,56 @@ graph TD
end
```
-| Service | Stack | Port |
-| ------------ | ------------------------------ | ---- |
-| **Frontend** | React + Vite + Tailwind/shadcn | 3000 |
-| **Backend** | FastAPI + ADK + LiteLLM | 8000 |
-| **Pipelines**| FastAPI + ADK + LiteLLM | 8001 |
-| **Database** | PostgreSQL 16 + pgvector | 5432 |
+| Service | Stack | Port |
+| --- | --- | --- |
+| **Frontend** | React 19, Vite, Tailwind CSS, shadcn/ui | 3000 |
+| **Backend** | FastAPI, Google ADK, LiteLLM, SQLAlchemy | 8000 |
+| **Pipelines** | FastAPI, Google ADK, LiteLLM, PyMuPDF | 8001 |
+| **Database** | PostgreSQL 16 + pgvector | 5432 |
## Quick Start
```bash
-# 1. Clone the repo
git clone https://github.com/airbus/norma.git
cd norma
-
-# 2. Copy environment config
cp .env.example .env
-
-# 3. Start the full stack
+# Edit .env to configure your LLM provider (see .env.example for options)
docker compose up
```
-The frontend is available at `http://localhost:3000`, the backend API at `http://localhost:8000`, and the pipelines service at `http://localhost:8001`.
+The frontend is available at [localhost:3000](http://localhost:3000), the backend API at [localhost:8000](http://localhost:8000), and the pipelines service at [localhost:8001](http://localhost:8001).
## Development
-See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed development setup instructions.
-
-### Frontend
+See [CONTRIBUTING.md](CONTRIBUTING.md) for the full development setup, code style, and PR process.
```bash
-cd frontend
-npm install
-npm run dev
-```
+# Frontend
+cd frontend && npm install && npm run dev
-### Backend / Pipelines
+# Backend or Pipelines
+cd backend && uv sync && uv run python main.py
-```bash
-cd backend # or cd pipelines
-uv sync
-uv run python main.py
+# Database migrations
+cd backend && uv run alembic upgrade head
```
-### Database Migrations
+## Documentation
-```bash
-cd backend
-uv run alembic upgrade head # apply migrations
-uv run alembic revision --autogenerate -m "description" # create migration
-```
+| Document | Description |
+| --- | --- |
+| [Getting Started](docs/getting-started.md) | Step-by-step guide to setting up Norma and creating your first project |
+| [Architecture](docs/architecture.md) | System design, data models, and integration patterns |
+| [API Reference](docs/api-reference.md) | REST endpoint documentation |
+| [Chat Agent](docs/chat-agent.md) | Norma AI assistant architecture and prompt assembly |
+| [Document Processing](docs/document-processing.md) | Document and codebase processing pipelines |
+| [Risk Evaluation](docs/risk-evaluation.md) | EU AI Act risk classification decision tree |
+| [Deployment](docs/deployment.md) | Docker Compose configuration and production setup |
-## Project Structure
+## Contributing
-```
-norma/
-├── frontend/ # React + Vite + Tailwind CSS
-│ └── src/
-├── backend/ # FastAPI + ADK + LiteLLM
-│ ├── app/
-│ │ ├── api/ # API routes
-│ │ ├── agents/ # ADK agents
-│ │ ├── core/ # Config, database
-│ │ ├── models/ # SQLAlchemy models
-│ │ ├── schemas/ # Pydantic schemas
-│ │ └── services/ # Business logic
-│ └── alembic/ # Database migrations
-├── pipelines/ # Data processing service
-│ ├── app/
-│ │ ├── api/ # API routes
-│ │ ├── core/ # Config, database
-│ │ ├── models/ # SQLAlchemy models
-│ │ ├── services/ # Business logic
-│ │ └── tasks/ # Pipeline tasks
-│ └── ...
-└── docker-compose.yml
-```
+Contributions are welcome. Please read the [contributing guidelines](CONTRIBUTING.md) and [code of conduct](CODE_OF_CONDUCT.md) before opening a pull request.
-## License
+## Licence
-See [LICENSE](LICENSE) for details.
+This project is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details.
diff --git a/docker-compose.yml b/docker-compose.yml
index b0830ed..e4b6fe1 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -26,7 +26,7 @@ services:
- GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/application_default_credentials.json
volumes:
- norma-data:/data
- - ${GOOGLE_APPLICATION_CREDENTIALS:-.empty}:/root/.config/gcloud/application_default_credentials.json:ro
+ - ${GOOGLE_APPLICATION_CREDENTIALS:-/dev/null}:/root/.config/gcloud/application_default_credentials.json:ro
depends_on:
db:
condition: service_healthy
@@ -48,7 +48,7 @@ services:
- GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/application_default_credentials.json
volumes:
- norma-data:/data
- - ${GOOGLE_APPLICATION_CREDENTIALS:-.empty}:/root/.config/gcloud/application_default_credentials.json:ro
+ - ${GOOGLE_APPLICATION_CREDENTIALS:-/dev/null}:/root/.config/gcloud/application_default_credentials.json:ro
depends_on:
db:
condition: service_healthy
@@ -71,7 +71,7 @@ services:
secrets:
ca_cert:
- file: ${CA_CERT_PATH:-.empty}
+ file: ${CA_CERT_PATH:-/dev/null}
volumes:
pgdata:
diff --git a/docs/api-reference.md b/docs/api-reference.md
index 504ce47..3350720 100644
--- a/docs/api-reference.md
+++ b/docs/api-reference.md
@@ -130,6 +130,10 @@ Upload a file for a framework document. Accepts `multipart/form-data` with a `fi
Supported file types: PDF, Markdown, TXT, CSV, JSON, XML, HTML.
+### `DELETE /api/projects/{project_id}/documents/{document_id}/upload`
+
+Remove an uploaded file from a framework document. Clears the file path, file name, summary, and upload timestamp. The document placeholder remains.
+
### Custom Documents
Free-form document uploads not tied to any framework definition. Custom document summaries are included in the chat agent and reporting suggestion context.
@@ -249,7 +253,7 @@ Get a single framework by ID.
### `POST /api/chat/sessions`
-Create a new chat session for a project. The system prompt is assembled at creation time from the project's current state (frameworks, questionnaire answers, uploaded document summaries, reporting evidence).
+Create a new chat session for a project. The system prompt is assembled at creation time from the project's current state (frameworks, questionnaire answers, uploaded document summaries, reporting evidence, codebase analysis).
**Body:**
```json
@@ -289,6 +293,117 @@ data: of the response.
data: [DONE]
```
+### `GET /api/chat/context`
+
+Fetch the assembled context for a project, optionally filtered by section. Used by the debug context viewer.
+
+**Query parameters:**
+- `project_id` (required) — project UUID
+- `section` (optional, default `full`) — one of `full`, `overview`, `frameworks`, `documents`, `reporting`, `github`
+
+**Response:**
+```json
+{
+ "context": "## Current Project Context\n..."
+}
+```
+
+Returns `{"context": null}` if the requested section has no content.
+
+---
+
+## Integrations
+
+### `GET /api/projects/{project_id}/integrations`
+
+Get the integration configuration for a project. Returns 404 if no integration is configured.
+
+**Response:**
+```json
+{
+ "id": "uuid",
+ "project_id": "uuid",
+ "provider": "github",
+ "repo_owner": "org",
+ "repo_name": "repo",
+ "github_project_number": 1,
+ "summary": "LLM-generated summary or null",
+ "architecture_mermaid": "graph TD\n...",
+ "sync_status": "idle",
+ "last_synced_at": "2026-05-27T10:00:00"
+}
+```
+
+### `POST /api/projects/{project_id}/integrations`
+
+Create a new integration for a project.
+
+**Body:**
+```json
+{
+ "provider": "github",
+ "github_pat": "ghp_...",
+ "repo_owner": "org",
+ "repo_name": "repo",
+ "github_project_number": 1
+}
+```
+
+### `PATCH /api/projects/{project_id}/integrations`
+
+Update integration configuration. All fields are optional.
+
+### `DELETE /api/projects/{project_id}/integrations`
+
+Remove the integration and all associated data (tasks, repo files). Returns 204.
+
+### `POST /api/projects/{project_id}/integrations/sync`
+
+Trigger a repository sync. Fetches the repo tree, file contents, issues, and project items from GitHub, then generates an LLM summary and architecture diagram. The sync runs asynchronously; poll the sync status endpoint to check progress.
+
+**Response:**
+```json
+{
+ "status": "syncing"
+}
+```
+
+### `GET /api/projects/{project_id}/integrations/sync-status`
+
+Check the current sync status.
+
+**Response:**
+```json
+{
+ "sync_status": "idle"
+}
+```
+
+Possible values: `idle`, `syncing`, `error`.
+
+### `GET /api/projects/{project_id}/integrations/tasks`
+
+List GitHub tasks (issues and project items) for the integration. Supports optional query filters: `status`, `assignee`, `label`, `milestone`.
+
+### `GET /api/projects/{project_id}/integrations/tasks/{task_id}`
+
+Get a single GitHub task by ID.
+
+### `GET /api/projects/{project_id}/integrations/context-file`
+
+Get the integration's context file content (user-editable supplementary context).
+
+### `PUT /api/projects/{project_id}/integrations/context-file`
+
+Update the integration's context file content.
+
+**Body:**
+```json
+{
+ "content": "Additional context about the codebase..."
+}
+```
+
---
## Admin
diff --git a/docs/architecture.md b/docs/architecture.md
index dac1943..6866877 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -39,7 +39,7 @@ graph TB
### Frontend
-**Stack:** React 18, Vite, Tailwind CSS, shadcn/ui, Lucide icons, React Router
+**Stack:** React 19, Vite, TypeScript, Tailwind CSS, shadcn/ui, Lucide icons, React Router, i18next
The single-page application provides the full user workflow:
@@ -48,8 +48,10 @@ The single-page application provides the full user workflow:
- **Risk banner** — real-time display of the LLM-evaluated risk classification with re-evaluate controls
- **Document checklist** — per-framework required documents with file upload and LLM-generated summaries, plus free-form custom PDF uploads
- **Reporting checklist** — compliance evidence tracking with free-text comments per item and AI-powered suggestion generation
+- **Codebase integration** — GitHub repository connection with auto-generated architecture diagrams and technical summaries
- **Chat interface** — streaming conversation with the Norma AI assistant, with full project context
-- **Settings** — user and admin management (invite links, user listing)
+- **Internationalisation** — full UI localisation in English, Spanish, French, and German (i18next)
+- **Settings** — user preferences (language), admin management (invite links, user listing)
In production, the frontend is served via Nginx which proxies `/api` requests to the backend service.
@@ -64,24 +66,34 @@ The central API server handling all business logic:
| **Authentication** (`app/api/routes/auth.py`) | JWT-based auth with bcrypt password hashing. Users register via single-use invite tokens created by admins. Tokens expire after 7 days. |
| **Projects** (`app/api/routes/projects.py`) | Full CRUD for compliance projects. PATCH requests that modify risk-relevant fields (description, intended purpose, intended users, deployment context, questionnaire answers) automatically trigger an LLM-based risk evaluation. |
| **Risk Evaluation** (`app/services/risk_evaluation.py`) | One-shot LLM call using `litellm.acompletion()` that classifies a project as `unacceptable`, `high`, `limited`, or `minimal` based on the EU AI Act self-assessment decision tree. |
-| **Chat / Norma Agent** (`app/agents/norma.py`, `app/api/routes/chat.py`) | Multi-turn AI assistant built on Google ADK. System prompt assembled from framework knowledge, project context, uploaded document summaries, and reporting evidence. Supports both synchronous and SSE streaming responses. |
+| **Chat / Norma Agent** (`app/agents/norma.py`, `app/api/routes/chat.py`) | Multi-turn AI assistant built on Google ADK. System prompt assembled from framework knowledge, project context, uploaded document summaries, reporting evidence, and codebase analysis. Supports both synchronous and SSE streaming responses. Includes a debug context endpoint for inspecting assembled context per section. |
| **Documents** (`app/api/routes/documents.py`) | Manages per-project document instances derived from framework-defined templates, plus free-form custom PDF uploads. Handles file uploads and delegates processing to the Pipelines service. |
| **Reporting** (`app/api/routes/reporting.py`) | Bulk upsert of compliance checklist evidence entries keyed by item identifier. Each entry stores the user's comment alongside an LLM-generated validation result (`covered` flag and `feedback` text). Includes a suggestion endpoint that generates context-aware comments and a validation endpoint that evaluates whether an answer satisfies the compliance question. |
+| **Integrations** (`app/api/routes/integrations.py`) | GitHub repository integration. Stores PAT credentials, triggers repository sync via the Pipelines service, and serves the generated summary, architecture diagram, and task list. |
| **Frameworks** (`app/api/routes/frameworks.py`) | Read-only endpoints for regulatory frameworks and their metadata. |
-| **Seeding** (`app/services/seed.py`) | On startup, seeds the EU AI Act framework with its required document definitions and loads knowledge base content from markdown files. Also creates a sample project for newly registered users. |
+| **Seeding** (`app/services/seed.py`) | On startup, seeds three compliance frameworks (EU AI Act, UNDP Human Rights Assessment, Environmental Impact) with their required document definitions and loads knowledge base content from markdown files. Creates a sample project for newly registered users. |
| **Migrations** (`alembic/`) | Alembic manages schema migrations, run automatically on backend startup. |
### Pipelines
**Stack:** FastAPI, LiteLLM, PyMuPDF
-A lightweight, independently scalable service dedicated to heavy processing tasks. Currently handles document processing:
+A lightweight, independently scalable service dedicated to heavy processing tasks:
+**Document processing:**
1. **Text extraction** — reads uploaded files (PDF via PyMuPDF, or plain text for Markdown/TXT/CSV/JSON/XML/HTML)
2. **LLM summarisation** — sends extracted text to `litellm.acompletion()` with a structured prompt requesting comprehensive markdown summaries
-3. **Database update** — writes the generated summary back to the `documents.summary` or `custom_documents.summary` column (determined by the `table_name` parameter)
+3. **Database update** — writes the generated summary back to the `documents.summary` or `custom_documents.summary` column
-The Pipelines service shares the `norma-data` volume with the Backend for file access and connects directly to PostgreSQL for writing summaries.
+**GitHub processing:**
+1. **Repository sync** — fetches the repository file tree and file contents via the GitHub API
+2. **Summary generation** — produces a structured technical summary (overview, tech stack, tasks, key observations) via LLM
+3. **Architecture diagram** — generates a Mermaid diagram of the system architecture from the codebase structure
+4. **Task sync** — fetches open issues and project items, storing them as `github_tasks`
+
+All LLM calls in the Pipelines service respect the user's language preference, generating summaries and descriptions in the configured language.
+
+The Pipelines service shares the `norma-data` volume with the Backend for file access and connects directly to PostgreSQL for writing results.
### Database
@@ -95,6 +107,9 @@ erDiagram
projects ||--o{ custom_documents : has
projects ||--o{ reporting_evidence : has
projects ||--o{ chat_sessions : has
+ projects ||--o| integrations : has
+ integrations ||--o{ github_tasks : has
+ integrations ||--o{ github_repo_files : has
frameworks ||--o{ document_definitions : defines
document_definitions ||--o{ documents : instantiates
chat_sessions ||--o{ chat_messages : contains
@@ -105,6 +120,7 @@ erDiagram
string name
string hashed_password
string role
+ string language_preference
boolean is_active
datetime created_at
}
@@ -187,6 +203,43 @@ erDiagram
text content
datetime created_at
}
+
+ integrations {
+ uuid id PK
+ uuid project_id FK
+ string provider
+ string github_pat
+ string repo_owner
+ string repo_name
+ integer github_project_number
+ text summary
+ text architecture_mermaid
+ string sync_status
+ datetime last_synced_at
+ datetime created_at
+ }
+
+ github_tasks {
+ uuid id PK
+ uuid integration_id FK
+ integer github_id
+ string title
+ text body
+ string status
+ jsonb assignees
+ jsonb labels
+ string milestone
+ string github_url
+ datetime created_at
+ }
+
+ github_repo_files {
+ uuid id PK
+ uuid integration_id FK
+ string file_path
+ text content
+ datetime created_at
+ }
```
Key design decisions:
@@ -209,6 +262,7 @@ graph LR
subgraph Pipelines
DP[Document Processor litellm.acompletion]
+ GP[GitHub Processor litellm.acompletion]
end
LLM_ADAPTER[LiteLLM Model Router]
@@ -216,6 +270,7 @@ graph LR
RE -->|one-shot| LLM_ADAPTER
NA -->|multi-turn via ADK LiteLlm adapter| LLM_ADAPTER
DP -->|one-shot| LLM_ADAPTER
+ GP -->|one-shot| LLM_ADAPTER
LLM_ADAPTER -->|vertex_ai/| VERTEX[Vertex AI]
LLM_ADAPTER -->|openai/| OPENAI[OpenAI]
@@ -227,7 +282,7 @@ Two distinct LLM usage patterns:
| Pattern | Used By | How | Purpose |
|---------|---------|-----|---------|
-| **One-shot classification** | Risk evaluation, Document processing | `litellm.acompletion()` directly | Single-turn, structured output. Risk eval returns exactly one word. Document processing returns a markdown summary. |
+| **One-shot classification** | Risk evaluation, Document processing, GitHub processing | `litellm.acompletion()` directly | Single-turn, structured output. Risk eval returns exactly one word. Document and GitHub processing return markdown summaries and Mermaid diagrams. |
| **Multi-turn agent** | Norma chat | Google ADK `Agent` + `Runner` + `InMemorySessionService`, with `LiteLlm` model adapter | Conversational AI assistant with full project context in the system prompt. Supports streaming via SSE. |
## Data Flows
@@ -297,7 +352,7 @@ sequenceDiagram
User->>FE: Creates chat session
FE->>BE: POST /api/chat/sessions
- BE->>BE: Assemble system prompt from: - Framework knowledge base - Project fields + questionnaire - Uploaded document summaries - Reporting evidence
+ BE->>BE: Assemble system prompt from: - Framework knowledge base - Project fields + questionnaire - Uploaded document summaries - Reporting evidence - Codebase analysis (if connected)
BE->>BE: Store frozen system prompt
BE-->>FE: Session created
@@ -316,6 +371,39 @@ sequenceDiagram
BE-->>FE: SSE data: [DONE]
```
+### Codebase Sync
+
+```mermaid
+sequenceDiagram
+ actor User
+ participant FE as Frontend
+ participant BE as Backend
+ participant PL as Pipelines
+ participant GH as GitHub API
+ participant LLM as LLM Provider
+ participant DB as PostgreSQL
+
+ User->>FE: Clicks Sync
+ FE->>BE: POST /api/projects/{id}/integrations/sync
+ BE->>BE: Set sync_status = "syncing"
+ BE->>PL: POST /api/github/process {integration_id, language}
+
+ PL->>GH: Fetch repo tree + file contents
+ PL->>DB: Store files in github_repo_files
+ PL->>GH: Fetch issues / project items
+ PL->>DB: Store tasks in github_tasks
+
+ PL->>LLM: acompletion() with repo context
+ LLM-->>PL: Technical summary
+ PL->>LLM: acompletion() with file tree
+ LLM-->>PL: Mermaid architecture diagram
+ PL->>DB: UPDATE integrations SET summary, architecture_mermaid
+
+ PL->>DB: Set sync_status = "idle"
+ BE-->>FE: Sync initiated
+ FE->>FE: Poll sync_status until idle
+```
+
## Knowledge Base
Regulatory knowledge is stored as markdown files organised by framework and loaded into the database at startup:
@@ -323,12 +411,24 @@ Regulatory knowledge is stored as markdown files organised by framework and load
```
backend/app/data/knowledge/
eu_ai_act/
- 00_getting_started.md -- Introduction and overview (sorted first)
- eu_ai_act_summary.md -- Full regulation summary
- high_risk_guidelines.md -- High-risk system compliance guide
- sandbox_guidelines.md -- AI regulatory sandbox guide
+ 00_getting_started.md -- Introduction and overview
+ eu_ai_act_summary.md -- Full regulation summary
+ high_risk_guidelines.md -- High-risk system compliance guide
+ sandbox_guidelines.md -- AI regulatory sandbox guide
+ undp_human_rights_assessment/
+ 01_summary.md -- UNDP toolkit summary
+ environmental_impact_framework/
+ (knowledge files added as available)
```
The seed service scans each framework's subdirectory by name slug (e.g., `eu_ai_act` for "EU AI Act"), concatenates all `.md` files in sorted order, and stores the result in `frameworks.content`. On subsequent startups, if file content has changed, the database is updated automatically.
This content is included in the Norma chat agent's system prompt, giving it deep regulatory knowledge to draw from when answering user questions.
+
+## Internationalisation
+
+The frontend supports English, Spanish, French, and German via i18next. Translation files are stored in `frontend/src/locales/{en,es,fr,de}/`. Each user has a `language_preference` field that controls:
+
+- **UI language** — all interface text, labels, and messages
+- **AI responses** — the Norma chat agent adapts its output language to match the user's preference
+- **Pipeline outputs** — document summaries and codebase analyses are generated in the user's language
diff --git a/docs/assets/norma-icon.svg b/docs/assets/norma-icon.svg
new file mode 100644
index 0000000..1c94a29
--- /dev/null
+++ b/docs/assets/norma-icon.svg
@@ -0,0 +1,24 @@
+
diff --git a/docs/assets/norma-logo.png b/docs/assets/norma-logo.png
new file mode 100644
index 0000000..205a893
Binary files /dev/null and b/docs/assets/norma-logo.png differ
diff --git a/docs/assets/screenshot.png b/docs/assets/screenshot.png
new file mode 100644
index 0000000..e8762b4
Binary files /dev/null and b/docs/assets/screenshot.png differ
diff --git a/docs/chat-agent.md b/docs/chat-agent.md
index c3cc330..8ba16f0 100644
--- a/docs/chat-agent.md
+++ b/docs/chat-agent.md
@@ -24,6 +24,7 @@ graph TB
DOCS[(documents.summary)]
CDOCS[(custom_documents.summary)]
RPT[(reporting_evidence)]
+ INT[(integrations)]
HIST[(chat_messages)]
end
@@ -36,6 +37,7 @@ graph TB
BUILDER --> DOCS
BUILDER --> CDOCS
BUILDER --> RPT
+ BUILDER --> INT
BUILDER -->|frozen system prompt| AGENT
API --> RUNNER
RUNNER --> SESSION
@@ -73,6 +75,8 @@ flowchart LR
DOC_SECTION[Document Summaries LLM-generated summaries of framework and custom documents]
DOC_SECTION --> RPT_SECTION
RPT_SECTION[Reporting Evidence Comments from compliance reporting checklist]
+ RPT_SECTION --> GH_SECTION
+ GH_SECTION[Codebase Context GitHub summary and architecture diagram]
end
```
@@ -80,11 +84,12 @@ flowchart LR
| Section | Source | Gives the agent... |
|---------|--------|--------------------|
-| **Base instruction** | Hardcoded in `norma.py` | Its identity, role boundaries, and response format |
-| **Framework knowledge** | `frameworks.content` column | Deep regulatory knowledge (EU AI Act articles, risk classification logic, compliance procedures) |
+| **Base instruction** | Hardcoded in `norma.py` | Its identity, role boundaries, and response format. Includes a language directive based on the user's preference. |
+| **Framework knowledge** | `frameworks.content` column | Deep regulatory knowledge (EU AI Act articles, UNDP Human Rights guidelines, Environmental Impact criteria) |
| **Project context** | `projects` table | Awareness of the specific AI system being assessed |
| **Document summaries** | `documents.summary` and `custom_documents.summary` columns | Understanding of compliance documents the user has already prepared (both framework-required and custom uploads) |
| **Reporting evidence** | `reporting_evidence` table | Knowledge of which checklist items the user has addressed |
+| **Codebase context** | `integrations.summary` and `integrations.architecture_mermaid` | Technical understanding of the AI system's implementation, architecture, and open tasks |
## Knowledge Base
@@ -93,14 +98,26 @@ Framework knowledge is stored in the `frameworks.content` database column, loade
```
backend/app/data/knowledge/
eu_ai_act/
- 00_getting_started.md # Introduction and overview (sorted first)
- eu_ai_act_summary.md # Full regulation summary
- high_risk_guidelines.md # High-risk system compliance guide
- sandbox_guidelines.md # AI regulatory sandbox guide
+ 00_getting_started.md # Introduction and overview
+ eu_ai_act_summary.md # Full regulation summary
+ high_risk_guidelines.md # High-risk system compliance guide
+ sandbox_guidelines.md # AI regulatory sandbox guide
+ undp_human_rights_assessment/
+ 01_summary.md # UNDP toolkit summary
+ environmental_impact_framework/
+ (knowledge files added as available)
```
The seed service (`backend/app/services/seed.py`) reads all `.md` files from each framework's subdirectory (sorted alphabetically), concatenates them with `---` separators, and stores the result in `frameworks.content`. On subsequent startups, if file content has changed, the database is updated automatically.
+## Language Support
+
+The system prompt includes a language directive based on the user's `language_preference` setting. Norma adapts its responses to match: English, Spanish, French, or German. The base instruction and response format remain the same regardless of language; only the output language changes.
+
+## Debug Context Viewer
+
+The `GET /api/chat/context` endpoint allows inspecting the assembled context by section (`overview`, `frameworks`, `documents`, `reporting`, `github`, or `full`). The frontend exposes this as a debug overlay on each page (enabled via `NORMA_DEBUG=true`), showing only the context section relevant to that page.
+
## Chat Session Lifecycle
```mermaid
@@ -148,6 +165,7 @@ Sends a user message and returns the response as Server-Sent Events (SSE). Each
| File | Purpose |
|------|---------|
| `backend/app/agents/norma.py` | Agent definition, system prompt builder |
-| `backend/app/api/routes/chat.py` | Chat API endpoints (create session, send message, stream) |
+| `backend/app/api/routes/chat.py` | Chat API endpoints (create session, send message, stream, debug context) |
+| `backend/app/data/evidence_questions.py` | Structured compliance questions for all frameworks |
| `backend/app/services/seed.py` | Knowledge base loading and framework seeding |
| `backend/app/data/knowledge/` | Markdown knowledge base files |
diff --git a/docs/deployment.md b/docs/deployment.md
index 7ea0711..1cfad21 100644
--- a/docs/deployment.md
+++ b/docs/deployment.md
@@ -139,7 +139,7 @@ sequenceDiagram
|---------|------|-------|-------|
| `frontend` | 3000 | Custom (React + Nginx) | Proxies `/api` to backend |
| `backend` | 8000 | Custom (FastAPI + uv) | Runs Alembic migrations and seeds DB on startup |
-| `pipelines` | 8001 | Custom (FastAPI + uv) | Document processing worker |
+| `pipelines` | 8001 | Custom (FastAPI + uv) | Document and codebase processing worker |
| `db` | 5432 | `pgvector/pgvector:pg16` | PostgreSQL with pgvector extension |
### Volumes
@@ -175,7 +175,7 @@ uv run alembic revision --autogenerate -m "description"
### Seeding
-On startup, the backend seeds regulatory frameworks and their required documents from `backend/app/services/seed.py`. Knowledge base content is loaded from markdown files in `backend/app/data/knowledge/`. If framework content has changed since the last startup, the database is updated automatically.
+On startup, the backend seeds three compliance frameworks (EU AI Act, UNDP Human Rights Assessment, Environmental Impact) and their required documents from `backend/app/services/seed.py`. Knowledge base content is loaded from markdown files in `backend/app/data/knowledge/`. If framework content has changed since the last startup, the database is updated automatically.
## Local Development (Without Docker)
diff --git a/docs/document-processing.md b/docs/document-processing.md
index 5822722..79bde9d 100644
--- a/docs/document-processing.md
+++ b/docs/document-processing.md
@@ -1,4 +1,8 @@
-# Document Processing
+# Document and Codebase Processing
+
+The Pipelines service handles two types of processing: document uploads (text extraction and summarisation) and GitHub repository analysis (codebase summary and architecture diagram generation).
+
+## Document Processing
The document processing pipeline handles file uploads, text extraction, and LLM-powered summarisation for compliance documents.
@@ -124,13 +128,69 @@ The summary is stored in the `documents.summary` or `custom_documents.summary` c
Both the Backend and Pipelines services mount the `norma-data` volume at `/data`. The Backend writes files during upload; the Pipelines service reads them during processing.
+## GitHub Processing
+
+When a user triggers a repository sync, the Backend calls the Pipelines service to process the connected GitHub repository.
+
+### Sync Flow
+
+```mermaid
+sequenceDiagram
+ participant BE as Backend
+ participant PL as Pipelines
+ participant GH as GitHub API
+ participant LLM as LLM Provider
+ participant DB as PostgreSQL
+
+ BE->>PL: POST /api/github/process {integration_id, language}
+
+ PL->>GH: GET repo tree (filtered)
+ PL->>GH: GET file contents (relevant files)
+ PL->>DB: Store in github_repo_files
+
+ PL->>GH: GET issues + project items
+ PL->>DB: Store in github_tasks
+
+ PL->>LLM: acompletion() with tasks + files
+ LLM-->>PL: Structured summary
+ PL->>DB: UPDATE integrations SET summary
+
+ PL->>LLM: acompletion() with file tree + contents
+ LLM-->>PL: Mermaid architecture diagram
+ PL->>DB: UPDATE integrations SET architecture_mermaid
+
+ PL->>DB: Set sync_status = "idle"
+```
+
+### Summary Generation
+
+The GitHub processor (`pipelines/app/tasks/github_processor.py`) generates two outputs:
+
+**Technical summary** with four sections:
+1. Overview of the repository
+2. Tech stack and dependencies
+3. Current tasks and priorities
+4. Key observations
+
+**Architecture diagram** as a Mermaid graph showing the system's main components and their relationships, accompanied by a bullet-point description.
+
+Both outputs respect the user's language preference. Prompts are truncated at 200,000 characters to stay within LLM context limits.
+
+### File Filtering
+
+The processor fetches the repository tree from GitHub and filters out files that are unlikely to be relevant to compliance analysis (e.g., lock files, build artefacts, test fixtures). Only files matching common source code and configuration patterns are fetched and stored.
+
## Files
| File | Purpose |
|------|---------|
| `backend/app/api/routes/documents.py` | Upload endpoint, document listing, lazy document creation, custom document CRUD |
+| `backend/app/api/routes/integrations.py` | Integration CRUD, sync trigger, task listing |
| `backend/app/models/document.py` | `Document` and `DocumentDefinition` SQLAlchemy models |
| `backend/app/models/custom_document.py` | `CustomDocument` SQLAlchemy model for free-form uploads |
+| `backend/app/models/integration.py` | `Integration`, `GitHubTask`, `GitHubRepoFile` models |
| `backend/app/services/seed.py` | Seeds `document_definitions` from framework config |
| `pipelines/app/tasks/document_processor.py` | Text extraction and LLM summary generation |
-| `pipelines/app/api/routes/documents.py` | Processing endpoint called by the backend (supports `table_name` parameter) |
+| `pipelines/app/tasks/github_processor.py` | GitHub repository analysis and diagram generation |
+| `pipelines/app/api/routes/documents.py` | Document processing endpoint called by the backend |
+| `pipelines/app/api/routes/github.py` | GitHub processing endpoint called by the backend |
diff --git a/docs/getting-started.md b/docs/getting-started.md
new file mode 100644
index 0000000..d17f3f8
--- /dev/null
+++ b/docs/getting-started.md
@@ -0,0 +1,131 @@
+# Getting Started
+
+This guide walks you through setting up Norma and creating your first AI compliance project.
+
+## Prerequisites
+
+- [Docker](https://docs.docker.com/get-docker/) and Docker Compose
+- An LLM provider account (Vertex AI, OpenAI, Anthropic, Azure, or any [LiteLLM-compatible provider](https://docs.litellm.ai/docs/providers))
+
+## 1. Clone and configure
+
+```bash
+git clone https://github.com/airbus/norma.git
+cd norma
+cp .env.example .env
+```
+
+Edit `.env` and set your LLM provider. For example, with Vertex AI:
+
+```bash
+NORMA_LITELLM_MODEL=vertex_ai/gemini-2.5-flash
+VERTEXAI_PROJECT=my-gcp-project
+VERTEXAI_LOCATION=europe-west1
+GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
+```
+
+Or with OpenAI:
+
+```bash
+NORMA_LITELLM_MODEL=openai/gpt-4o
+OPENAI_API_KEY=sk-...
+```
+
+See the [LiteLLM docs](https://docs.litellm.ai/docs/providers) for all supported providers.
+
+## 2. Start the stack
+
+```bash
+docker compose up --build
+```
+
+This starts four services:
+
+| Service | URL | Purpose |
+| --- | --- | --- |
+| Frontend | [localhost:3000](http://localhost:3000) | Web interface |
+| Backend | [localhost:8000](http://localhost:8000) | API server |
+| Pipelines | [localhost:8001](http://localhost:8001) | Document and codebase processing |
+| Database | localhost:5432 | PostgreSQL 16 + pgvector |
+
+On first startup, the backend automatically runs database migrations and seeds the compliance frameworks (EU AI Act, UNDP Human Rights Assessment, Environmental Impact).
+
+## 3. Create an account
+
+1. Open [localhost:3000](http://localhost:3000) in your browser.
+2. The first user is created via an admin invite. Check the backend logs for the initial invite token, or create one via the API:
+
+```bash
+curl -X POST http://localhost:8000/api/admin/invites \
+ -H "Content-Type: application/json" \
+ -d '{"email": "you@example.com", "role": "admin"}'
+```
+
+3. Register with the invite token at the login page.
+
+## 4. Create a project
+
+Once logged in, create a new project describing your AI system:
+
+- **Name** and **description** of the AI system
+- **Intended purpose**, **intended users**, and **deployment context**
+
+These details feed into the risk classification and give the Norma AI assistant context about your system.
+
+## 5. Classify risk
+
+Navigate to the **Risk Classification** page and complete the self-assessment questionnaire. Norma evaluates your answers against the EU AI Act decision tree and assigns a risk level:
+
+| Level | Meaning |
+| --- | --- |
+| Unacceptable | Prohibited AI practice (Article 5) |
+| High | Requires conformity assessment and full documentation (Articles 6-7) |
+| Limited | Transparency obligations apply (Article 50) |
+| Minimal | No specific regulatory requirements |
+
+The risk classification updates automatically as you modify project details or questionnaire answers.
+
+## 6. Upload documents
+
+On the **Documents** page, upload compliance documents for each framework. Supported format: PDF.
+
+Norma extracts the text and generates an AI summary for each document. These summaries become part of the context available to the Norma assistant and the reporting suggestion engine.
+
+You can also upload additional documents not tied to any specific framework requirement.
+
+## 7. Complete reporting checklists
+
+The **Reporting** page presents compliance checklists for each framework. For each item:
+
+- Write your evidence or comment
+- Use **Ask Norma** to get AI-generated suggestions based on your project context and uploaded documents
+- Validate your answers against auditor standards
+
+## 8. Connect a GitHub repository (optional)
+
+On the **Codebase** page, connect a GitHub repository to give Norma visibility into your codebase:
+
+1. Add a GitHub Personal Access Token (PAT) with repo read access
+2. Configure the repository owner and name
+3. Sync to generate an architecture diagram and technical summary
+
+The codebase analysis feeds into the Norma assistant's context, allowing it to reference your actual implementation when answering compliance questions.
+
+## 9. Chat with Norma
+
+The **Chat** page provides a context-aware AI assistant. Norma has access to:
+
+- Your project details and risk classification
+- Framework knowledge (EU AI Act articles, guidelines, compliance procedures)
+- Uploaded document summaries
+- Reporting evidence you have provided
+- Codebase analysis (if a GitHub repo is connected)
+
+Each chat session captures a snapshot of your project state, so the assistant's context remains consistent throughout the conversation.
+
+## Next steps
+
+- [Architecture](architecture.md) for a deep dive into the system design
+- [API Reference](api-reference.md) for endpoint documentation
+- [Deployment](deployment.md) for production configuration
+- [Contributing](../CONTRIBUTING.md) to get involved
diff --git a/frontend/public/favicon.svg b/frontend/public/favicon.svg
index 6893eb1..1c94a29 100644
--- a/frontend/public/favicon.svg
+++ b/frontend/public/favicon.svg
@@ -1 +1,24 @@
-
\ No newline at end of file
+
+
+ 
