CATROID-1664, CATROID-1666 First AI tutor integration steps#5218
Open
spipau wants to merge 41 commits into
Open
CATROID-1664, CATROID-1666 First AI tutor integration steps#5218spipau wants to merge 41 commits into
spipau wants to merge 41 commits into
Conversation
* Implement `getFormulaString` in `Formula` to generate a visual tree structure of the formula * Add `getFlattenedAllListsString` to retrieve a trimmed extern formula string with flattened user lists * Add `getLeftChild` and `getRightChild` getters to `FormulaElement`
* Add `getAllList` to generate a tree-like string representation of the project structure (Scenes, Sprites, Scripts, Bricks, and Formulas) * Add `getAllListAsJson` and `getAllListAsJsonString` to provide a structured JSON representation of the project * Include debug logging for the generated project export strings
… as strings and JSON * Add `getAllListForSprite` and `getAllListForScene` to generate tree-like text representations of scripts and bricks * Add `getAllListAsJsonStringForSprite` and `getAllListAsJsonStringForScene` for JSON-formatted component hierarchies * Add `getProjectSummary` to provide a high-level overview of project statistics, including scene counts, sprite counts, and global variables/lists
* Introduce `getXmlAsStringFromSprite(Sprite)` to allow serializing individual Sprite objects to XML strings. * Ensure thread safety by using the existing `loadSaveLock`.
* Add strings for AI Assist format options including Tree, JSON, XML, and Summary * Implement `handleAiAssistButton` in `SpriteActivity` and `ProjectActivity` to display a format selection dialog * Serialize project, scene, or sprite data based on user selection using `ProjectManager` and `XstreamSerializer` * Navigate to `AiAssistFragment` with the generated structure string * Hide options menu items in `SpriteActivity` when `AiAssistFragment` is active
* Add Kotzilla Maven repository for Koin-Embedded * Apply Kotlin Compose compiler plugin * Enable Compose build feature in the catroid module * Add Catrobat AI Tutor dependency * Add Compose UI and Runtime dependencies (v1.7.5)
- Update `fragment_ai_assist.xml` to include a `ComposeView` and wrap existing content in a `FrameLayout` - Initialize `AiTutorView` within `AiAssistFragment` using Jetpack Compose - Pass the project structure context from fragment arguments to the AI Tutor component - Configure `ViewCompositionStrategy` to manage the ComposeView lifecycle correctly
…unctions (required for AI Tutor)
- Implement `Formula.getFormulaInlineString()` to generate human-readable mathematical expressions from formula trees - Enhance project summaries to include script triggers (e.g., broadcast messages, conditions, start events) - Include variable names and broadcast messages in brick descriptions within tree and JSON outputs - Handle duplicate script and brick types in JSON serialization by appending index suffixes (e.g., `Formula_2`) - Update `GsonBuilder` to disable HTML escaping for better readability of project exports - Add helper methods in `ProjectManager` to extract metadata from scripts and bricks for debugging purposes
…ation and material3 dependencies
… fragments if AI Assist feature is enabled
…L to AiAssistFragment
- Remove `AiAssistFragment` and its associated layout `fragment_ai_assist.xml` - Delete `handleAiAssistButton` and project serialization logic from `ProjectActivity` - Replace the local `compose_ai_tutor_preview` in `fragment_script.xml` with a global `compose_ai_overlay` in `activity_sprite.xml` - Update `activity_sprite.xml` to use a `FrameLayout` root to host the AI Tutor flow as a full-screen overlay - Simplify `fragment_script.xml` by removing redundant nested layouts and the deleted Compose view components
- Replace basic script-level diffing with a granular brick-level comparison using a Longest Common Subsequence (LCS) algorithm - Implement `buildDiffRows` to align original and AI-generated bricks, supporting ADDED, REMOVED, MODIFIED, and UNCHANGED statuses - Flatten scripts into ordered lists to include script headers, nested bricks, and markers in the comparison view - Reconcile adjacent remove/add operations of the same brick type into a single "MODIFIED" state for better readability - Update `AiTutorDiffScreen` UI with a summary header, color-coded legend, and humanized brick names and field labels - Improve visual styling using project-specific theme colors, rounded corners, and formatted subtitles for formula changes - Add defensive error handling and logging for script flattening and formula string extraction - Remove the now-obsolete `ScriptDiff` data class and `AiTutorPreviewHelper` object
- Replace `AiAssistFragment` with a Compose-based overlay managed by `AiAssistOverlayHelper` in `SpriteActivity` - Update `SpriteActivity` to handle overlay visibility and hardware back button behavior - Remove deprecated AI Tutor preview logic and fragment result listeners from `ScriptFragment` - Integrate `AiTutorSpriteValidator` in `ScriptFragment` to ensure AI-generated sprites are renderable before application - Clean up options menu handling logic in `SpriteActivity` previously required for the AI Assist fragment - Dispose of Compose compositions when hiding the AI overlay to manage resources correctly
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
detekt found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
There was a problem hiding this comment.
Checkstyle found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
…ing overlay imports
- Organize AI Assist classes into specialized sub-packages: `overlay`, `validation`, `error`, and `diff` - Update `AiAssistOverlay` to use `getSpriteFromXmlString` for clipboard processing and update related imports - Enhance `AiTutorErrorDialog` by applying a `RoundedCornerShape` to the `AlertDialog` - Relocate `AiTutorSpriteValidator`, `AiAssistOverlay`, `AiAssistOverlayCallbacks`, and `AiTutorErrorDialog` to their respective new packages
- Move `AiTutorDiffScreen` to a dedicated `.diff` subpackage and decompose logic into specialized helper files - Implement a GitHub-style diff layout using status stripes, leading icons, and row tinting for better change visualization - Add `BrickDiffDialog` to allow detailed inspection of "Before" and "After" states for individual bricks - Introduce `DiffToken` logic to highlight specific changed values within brick subtitles using `AnnotatedString` and bold styling - Improve script header rendering by resolving real string resources via `BrickBaseType` instead of class name humanization - Update `brickSignature` to include variable and list selections, ensuring data-only changes are correctly identified as MODIFIED - Extract diffing algorithms to `SpriteDiffer.kt` and styling constants to `DiffStatusStyle.kt` for better maintainability - Replace the dual-column comparison with a more natural single-column flow for MODIFIED rows, showing a clear "Old → New" value transition
…nges - Implement a responsive `ModifiedValues` composable that switches from a horizontal to a vertical stacked layout for long text changes - Use `TextMeasurer` and `onSizeChanged` to dynamically calculate text width and determine the optimal layout for modified brick values - Replace the static legend `Row` with a `FlowRow` to better handle screen width constraints - Update `LegendChip` to include status-specific icons and improved styling for the unchanged state - Add visual borders and refine padding for unchanged brick rows to improve visual distinction - Add vertical spacing and a `Spacer` before script headers for better grouping in the diff list - Introduce Compose previews for "Changes" and "No changes" states using sample sprite data
… parameter naming - Add `onClick` support to `ScriptHeaderRow` to allow selecting script headers in the diff view - Refactor `AiTutorDiffScreen` and its sub-composables to use named arguments for improved readability - Update `ScriptHeaderRow` to include a `clickable` modifier and standardized styling - Remove unused `context` parameter from the `BrickDiffDialog` call - Standardize parameter usage across `StatusContainer`, `BrickContent`, and `SummaryAndLegend` components
- Replace manual brick field rendering in `BrickDiffDialog` with an `AndroidView` that leverages existing `getPrototypeView` logic for a more accurate representation - Simplify `DialogSection` by removing redundant parameters such as `oldForDiff`, `accent`, and `context` - Add error handling and logging for prototype view generation to provide a fallback `TextView` if the view cannot be created - Refine UI layout and spacing by adjusting modifiers, padding, and alignment within the dialog components - Remove the dependency on `brickValueTokens` within the dialog, favoring the native brick view instead
spipau
changed the title
- Remove logic in `onResume` that checks for the existence of the undo XML file to update the undo menu item visibility in `SpriteActivity`
- Wrap the technical reason display logic in `AiTutorErrorDialog` with a `BuildConfig.DEBUG` check to prevent showing internal error details in production environments.
- Improve readability in `SpriteDiffer` by renaming LCS variables and adding comments explaining the alignment algorithm - Remove redundant error logging and defensive `try-catch` block from `SpriteDiffer.flatten` - Reorganize `BrickTextFormatter` into logical sections (Titles, Value Text, Helpers) for better maintainability - Add KDoc documentation to internal utility functions to clarify their purpose within the diffing process
- Replace manual subtitle generation with a view-traversal approach to reconstruct the full brick editor phrase in its natural UI order - Update `DiffToken` to distinguish between static labels and dynamic values (formulas and spinner selections) - Implement specialized styling in the diff screen to highlight modified dynamic values with bold and underline attributes - Enhance data extraction using reflection to automatically include names from any `Nameable` fields, such as looks, sounds, or scenes - Update `BrickDiffDialog` to display the full reconstructed editor phrase as the dialog title - Add a fallback layout inflation mechanism to `BrickTextFormatter` to ensure phrases can be generated even if full view population fails
|
This was referenced Jun 29, 2026
harissabil
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First integration of the Catrobat AI Tutor into Catroid. This PR enables Jetpack Compose, wires in the AI Tutor SDK, and adds the AI Assist overlay in
SpriteActivitytogether with the supporting serialization, validation, error handling, and abrick-level diff screen for reviewing AI-suggested sprite changes. It also carries the OkHttp/Okio and Lifecycle API updates that the SDK requires.
Base branch:
dev-ai-tutorJira: CATROID-XXXX
Build and dependencies
FEATURE_AI_ASSIST_ENABLEDfor theallanddebugbuild variants.API updates required by the AI Tutor SDK
Response.body()toResponse.body).CameraManagertoLifecycleRegistry(renamelifecycletolifecycleRegistry), update the LiveData extension, and update all references.Serialization and validation
Spriteto/from XML string methods inXstreamSerializer(getXmlAsStringFromSprite,getSpriteFromXmlString), using the existingloadSaveLockfor thread safety and handling parse errors.AiTutorSpriteValidatorto validate AI-generated sprite XML before it is applied.AI Assist overlay
activity_sprite.xml(FrameLayoutroot,compose_ai_overlay), managed byAiAssistOverlayHelperwithAiAssistOverlayCallbacks.SpriteActivity.AiTutorSpriteValidatorinScriptFragmentso AI-generated sprites are validated as renderable before being applied.AI Tutor diff screen
AiTutorDiffScreenwith brick-level diffing via an LCS algorithm (ADDED, REMOVED, MODIFIED, UNCHANGED), implemented inSpriteDifferwith styling inDiffStatusStyle.BrickDiffDialogshowing native brick prototype views (viagetPrototypeViewin anAndroidView) for before/after inspection.ic_minusandic_edit_pencildrawables.Error handling and structure
AiTutorErrorDialogfor surfacing AI response errors.overlay,validation,error, anddiffsubpackages.Note for reviewers (building locally)
This branch depends on the AI Tutor library (
org.catrobat:aitutor), which is resolved from your Maven Local repository. To build, you first need to publish it locally:integration/pocket-codebranch: https://github.com/Catrobat/catrobat-ai-tutor/tree/integration/pocket-code./gradlew publishToMavenLocal@harissabil - ping me on Slack if the local install gives you any trouble and I'll walk you through it.
Your checklist for this pull request
Please review the contributing guidelines and wiki pages of this repository.