A-Team JARVIS AI

Local-first JARVIS desktop agent using Ollama/Qwen, optional local voice, optional AI visuals, Barehands hand tracking, persistent memory, and optional web research.

This README documents the current integrated v2.2 package, including the GTK3 desktop shell, embedded visualizer/Barehands panels, local voice pipeline, diagnostic logging, desktop launcher, service recovery, and the current Barehands tracking integration.

1. What This Package Is

A-Team JARVIS AI is a local desktop agent built around a Python runtime and local Ollama model service.

The package combines:

JARVIS agent runtime — Python + Ollama/Qwen

GTK3 desktop GUI — PyGObject

Local voice — Whisper/faster-whisper input + Kokoro output

AI Visualizer — configurable animated JARVIS faces

Barehands — webcam-based hand-tracked interface

Persistent Memory Vault — local Markdown-based memory

Research tools — local file/archive tools plus optional web research

Durable diagnostics — rotating file logs and crash logging

Desktop launcher creation — .desktop files for the Linux desktop and application menu

Release/update checking — Updates/version.json on the configured Releases branch

The design is local-first. Ollama, voice processing, visualizer state, Barehands state, and the main agent runtime operate on the local machine. Internet access is optional and is controlled separately for research and for the external assets used by Barehands.

2. Current v2.2 Desktop GUI

The GTK3 shell is implemented in:

scripts/jarvis_gui.py

Launch it with:

./start_gui.sh

or:

./venv/bin/python scripts/jarvis_gui.py

Main GUI layout

Left side: Agent Options and JARVIS chat

Agent Options:

Voice

Visuals

Barehands

Internet

Diagnostics

Outline/theme selection

Desktop launcher creation

Setup

Start/Stop JARVIS

Right side: optional Face and Barehands panes

Face selector: visible when Visuals is selected

Voice selector: visible when Voice is selected

Face pane: embedded with WebKitGTK when available

Barehands pane: embedded with WebKitGTK when available

Browser fallback: used when an embedded WebKit view is unavailable

Status indicator: shows Stopped, Starting, Running, Working, or Stopping states

Session startup behavior

The GUI intentionally starts each new GUI session with the optional runtime features turned off:

Voice

Visuals

Barehands

Internet

Diagnostics

This prevents stale feature selections from unexpectedly launching optional services when the GUI is opened.

Persistent settings such as the selected face, voice, and GUI theme are still retained.

When the user presses Start JARVIS, the currently selected feature toggles are written to config/jarvis.json and used for that runtime session.

3. JARVIS Runtime

The main agent is:

scripts/jarvis.py

The normal launcher is:

./scripts/start_jarvis.sh

The launcher:

Validates the project virtual environment.

Activates the virtual environment.

Installs the selected core/optional requirements.

Reads runtime feature selections from config/jarvis.json.

Initializes the visualizer and Barehands state files.

Checks the local Ollama service.

Replaces a stale user-owned Ollama process when its working directory has become invalid.

Starts Ollama locally when necessary.

Starts the AI Visualizer when enabled.

Starts Barehands when enabled.

Starts the JARVIS Python agent.

Service recovery

The launcher contains checks for stale services from previous extracted/copied installations.

For the AI Visualizer, it does not blindly trust a successful /state response. It also verifies the configured face URL and the visualizer health response before accepting an existing server.

This prevents a stale visualizer process from another copy of the project from serving the wrong document root and producing a literal 404/not found face page.

Barehands similarly validates the stage endpoint and replaces stale Barehands server instances when necessary.

4. GUI Status States

The GUI reports the runtime state at the top of the Agent Options area.

Typical states are:

JARVIS Stopped JARVIS Starting.. JARVIS Running JARVIS Working… JARVIS Stopping…

The GUI changes to Working while JARVIS is listening/processing and returns to Running after a completed JARVIS: response.

Stopping JARVIS removes the live Face/Barehands panes while retaining the user's checkbox selections for the next run.

5. Voice

Voice is optional and is controlled by:

"voice": { "enabled": true }

The voice pipeline uses local components:

Whisper/faster-whisper for speech recognition

Kokoro for local text-to-speech

sounddevice/soundfile and related local audio packages

The current voice requirements deliberately pin the versions associated with the working Whisper/AV integration:

faster-whisper==1.1.1 av==12.3.0

This avoids the metadata_errors incompatibility that occurred with the affected newer AV/faster-whisper combination.

Thinking sound

The thinking sound is now coordinated with the JARVIS backend.

JARVIS writes:

ai-visualizer/.voice_loading_pid

while the backend thinking sound is active.

The visualizer respects that marker so the browser-side thinking sound does not play simultaneously with the backend sound.

The thinking sound asset is:

ai-visualizer/assets/thinking.wav

6. AI Visualizer

The visualizer lives under:

ai-visualizer/

and runs on:

http://127.0.0.1:8790/

The visualizer is an embedded upstream component by Jared Rhodenizer (@jaredrhod).

Upstream project:

https://github.com/jaredrhod/ai-visualizer

The package retains the upstream copyright/license notices in the visualizer source.

Included faces

The current package contains:

rain a-team a-team-android a-team-android-1 a-team-command a-team-moto-pad a-team-ops board neural radial

The configured face is selected through:

"visuals": { "face": "board" }

Visualizer signal bus

The face reads the local state bus:

ai-visualizer/.voice_state ai-visualizer/.voice_waveform ai-visualizer/.voice_loading_pid

States include:

idle listening thinking speaking

Current visualizer logging

Routine /state polling is intentionally not written as individual log lines.

The visualizer logs useful events instead:

startup

configured face

WebView/face boot

configuration loading

state transitions

duration of the previous state

periodic heartbeat

JavaScript errors

unhandled promise rejections

state-poll failures

audio/microphone problems

thinking-sound initialization/playback problems

HTTP failures

The visualizer log is:

logs/faces.log

This keeps the log useful instead of filling it with hundreds of normal polling requests.

7. Barehands

Barehands lives under:

barehands/

and runs on:

http://127.0.0.1:8794/stage.html

Barehands is an upstream component by Jared Rhodenizer (@jaredrhod).

Upstream project:

https://github.com/jaredrhod/barehands

The package retains the upstream copyright and AGPL license notices in the Barehands source.

Barehands uses the webcam to track hand position and provides the interactive board/ring interface.

Embedded camera support

When WebKitGTK is available, the GTK GUI embeds Barehands directly.

The GUI explicitly handles WebKitGTK's user-media permission request and allows the Barehands video request for the dedicated Barehands WebView.

This is required because WebKitGTK otherwise denies getUserMedia() requests by default.

The embedded settings enable the WebGL/media functionality required by the stage.

If WebKitGTK is unavailable, the GUI provides the browser fallback.

Current hand-tracking implementation

The current integrated package uses the legacy browser MediaPipe Hands runtime rather than the Tasks Vision HandLandmarker path that was previously tested.

The package currently loads:

@mediapipe/hands@0.4.1675469240

and configures tracking for up to two hands.

The tracker is asynchronous and throttled so it does not attempt to process every camera frame at full camera rate.

The current implementation records detailed tracker information specifically so two-hand tracking can be diagnosed without filling the log with routine HTTP traffic.

Barehands logging

The Barehands log is:

logs/barehands.log

Routine /state and /orb polling is intentionally quiet.

Useful tracker information includes:

camera initialization

capture resolution

tracker initialization

tracker errors

0 → 1 → 2 hand transitions

left/right hand identification

confidence values

bounding boxes

landmark positions

wrist/index positions

pinch gap

tracker latency

tracker FPS

periodic heartbeat

JavaScript errors

HTTP failures

A two-hand test should produce compact records similar to:

TRACK hands=2 Left:0.91,Right:0.87 TRACK hands=2 Left:0.90,Right:0.84 TRACK hands=1 Left:0.92 TRACK hands=0

This makes it possible to determine whether a second hand is being missed because of detection confidence, tracking loss, hand size/distance, camera conditions, or another runtime problem.

8. Barehands State Bus

Barehands uses:

barehands/state/state barehands/state/mood.json barehands/state/wave.json

The basic assistant states are:

idle listening thinking speaking

The state timeout prevents a dead session from leaving the assistant ring permanently stuck in a non-idle state.

The Barehands board also supports the bundled command helpers:

barehands/bin/board.sh barehands/bin/board-state.sh

The bundled upstream documentation explains the board protocol, gestures, media airlock, and assistant integration.

9. Persistent Memory

Persistent memory lives under:

Memory_Vault/

The project uses local Markdown files rather than a hosted memory service.

Relevant project files include:

Memory_Vault/VAULT-INDEX.md Memory_Vault/Active Priorities.md Memory_Vault/01 - Daily Notes/ Memory_Vault/02 - Profile/ Memory_Vault/04 - Knowledge/

The memory implementation is:

scripts/memory.py

The project documentation notes that this architecture is inspired by Jared Rhodenizer's AI Memory Vault.

10. Research and Tools

The project includes:

scripts/agent_tools.py scripts/research_tools.py

The runtime can work with:

local files

project files

archives

local research sources

optional web research

Internet research is controlled separately through:

"research": { "internet": true }

Turning Internet off does not disable durable local logging or the local agent.

11. Diagnostics and Logging

JARVIS has two different concepts:

Console diagnostics

Controlled by:

"diagnostics": { "enabled": true }

When enabled, diagnostic records are also visible in the terminal/GUI output.

Durable file logging

Durable logging remains active regardless of the Diagnostics toggle.

The centralized logger is:

scripts/logging_config.py

Primary logs include:

logs/jarvis.log logs/crash.log logs/faces.log logs/barehands.log logs/ollama.log

The purpose of the Diagnostics toggle is therefore visibility, not disabling background logging.

Crash handling also captures uncaught main/thread exceptions and enables Python fault diagnostics.

12. GUI Themes

The GTK GUI uses:

assets/ateam_header.png

and the supplied banner artwork for the selectable outline theme.

Available themes:

Blue

Red

Green

Purple

Orange

Cyan

Pink

The selected theme is saved under:

"gui": { "outline_color": "Cyan" }

The outline is applied to the Agent Options, Agent Chat, and active right-side panels.

13. Face and Voice Selectors

Face selection is stored under:

"visuals": { "face": "board" }

Voice selection is stored in the GUI's persistent configuration.

The selector controls are conditional:

Face selector appears when Visuals is enabled.

Voice selector appears when Voice is enabled.

The actual Face and Barehands panes do not start merely because their checkboxes are selected. They become live after Start JARVIS launches the backend.

14. Face/Barehands Pane Controls

Both right-side panels have small maximize controls in their upper-right corner.

Maximizing one optional panel gives it the available right-side workspace while hiding the other optional panel.

Restoring the normal view returns the layout to approximately the original right-pane allocation.

The controls are intentionally compact so they do not cover the visualizer/Barehands content.

15. Desktop Launcher

The GUI provides:

Create Desktop Launcher

It creates:

~/Desktop/A-Team-JARVIS-AI.desktop ~/.local/share/applications/A-Team-JARVIS-AI.desktop

The launcher:

uses assets/desktop_icon_jarvis.png when available

falls back to assets/desktop_icon.png

starts the root start_gui.sh

is marked executable

The generated launcher does not replace the running installation.

16. Configuration

The main runtime configuration is:

config/jarvis.json

Important settings include:

voice.enabled visuals.enabled visuals.barehands_enabled visuals.face research.internet diagnostics.enabled gui.outline_color

The GUI is the normal interface for changing these values.

Feature toggles are treated as session controls: a new GUI launch starts with optional features off, and the current selections are committed when JARVIS is started.

17. Setup

Run:

./scripts/setup_jarvis.sh

The setup process creates or reuses:

venv/

and installs dependencies based on the selected features.

Core requirements:

requirements/requirements-core.txt

Voice requirements:

requirements/requirements-voice.txt

Web/research requirements:

requirements/requirements-web.txt

The setup process can also check Ollama and optionally pull the configured model.

18. Project Layout

A-Team-JARVIS_AI/ ├── start_gui.sh ├── scripts/ │ ├── jarvis.py │ ├── jarvis_gui.py │ ├── agent_tools.py │ ├── research_tools.py │ ├── memory.py │ ├── local_voice.py │ ├── logging_config.py │ ├── start_gui.sh │ ├── start_jarvis.sh │ ├── setup_jarvis.sh │ ├── select_jarvis_face.sh │ └── select_jarvis_voice.sh ├── config/ │ ├── jarvis.json │ ├── system_prompt.md │ └── .aider.conf.yml ├── ai-visualizer/ ├── barehands/ ├── Memory_Vault/ ├── Project_Folder/ ├── requirements/ ├── logs/ ├── Updates/ ├── assets/ ├── docs/ └── VERSION

The root keeps the simple GUI launcher. Project Python modules are under scripts/, user-facing Bash utilities/selectors are under scripts/, and component servers remain in their component directories.

19. Updates

The GUI checks the dedicated Releases branch:

https://ateam.gotadell.com/git/PizzaG/A-Team-Jarvis_AI/raw/branch/Releases/Updates/version.json

Release archives are expected on that same branch.

The GUI checks the archive at the branch root first and then under Updates/.

Current metadata format:

{ "version": "1", "display": "2.2", "file": "A-Team-Jarvis_AI-v2.2.7z", "changelog": "Initial GTK3 GUI Code Refactor, Public Release v2.2" }

Downloaded release archives are stored under:

Updates/

The GUI does not automatically overwrite the running installation.

20. Current Release

Version: 2.2 Primary runtime: Python + Ollama GUI: GTK3 / PyGObject Embedded browser: WebKitGTK 4.x AI Visualizer: 127.0.0.1:8790 Barehands: 127.0.0.1:8794

Version source:

VERSION

Release metadata:

Updates/version.json

Upstream Attribution

AI Visualizer

The AI Visualizer integrated into this package originates from:

Jared Rhodenizer (@jaredrhod)

Upstream repository:

https://github.com/jaredrhod/ai-visualizer

The upstream visualizer source files retain their copyright and AGPL-3.0-or-later notices.

Barehands

The Barehands integrated into this package originates from:

Jared Rhodenizer (@jaredrhod)

Upstream repository:

https://github.com/jaredrhod/barehands

The upstream Barehands source files retain their copyright and AGPL-3.0-or-later notices.

Upstream Author

Jared Rhodenizer's GitHub:

https://github.com/jaredrhod

The upstream projects and their source files retain their applicable copyright and license notices.

Other Upstream Components

Barehands uses Google MediaPipe for hand tracking and three.js for browser-side 3D rendering.

The AI Visualizer includes the VT323 typeface. Its applicable font license is preserved at:

ai-visualizer/assets/VT323-OFL.txt

The upstream component licenses and notices remain applicable to their respective source files. This project-level README does not relicense those upstream components.

License

GNU General Public License v3.0 (GPL-3.0).

2019-Present — A-Team Digital Solutions