SLEAP App

Pose estimation and tracking app for SLEAP.

A modern rewrite of SLEAP's Qt/Python desktop labeling interface as a web app, with an optional Tauri v2 desktop shell for native file access. Runs entirely in the browser -- no server or Python required.

Install the desktop app

Until the first release with attached builds exists, use the /dev/ URLs below (they go live on the next merge to main). After that, drop the /dev/.

macOS / Linux:

curl -fsSL https://app.sleap.ai/dev/install.sh | sh

Windows (PowerShell):

irm https://app.sleap.ai/dev/install.ps1 | iex

Or use the app in any browser at app.sleap.ai -- no install needed.

macOS builds are universal, so one .dmg covers both Apple Silicon and Intel. Linux gets a .deb, an .AppImage and an .rpm; Windows gets an NSIS installer and an .msi.

Installing a specific version, a pre-release, or a build you already downloaded

# A specific release tag (pre-releases included when named explicitly)
curl -fsSL https://app.sleap.ai/dev/install.sh | sh -s -- --tag v0.1.2

# The newest build even if it is a pre-release
curl -fsSL https://app.sleap.ai/dev/install.sh | sh -s -- --pre

# Read it before you run it
curl -fsSL https://app.sleap.ai/dev/install.sh | less

To install a file you already have -- a .dmg, .deb, .AppImage, .rpm, or the .zip straight off a GitHub Actions artifact page -- download the script first, then pass it the file. This path also strips the quarantine flag:

curl -fsSL https://app.sleap.ai/dev/install.sh -o install.sh
sh install.sh ~/Downloads/SLEAP_0.1.2_universal.dmg
sh install.sh ~/Downloads/sleap-app-macos-universal.zip

irm https://app.sleap.ai/dev/install.ps1 -OutFile install.ps1

# Windows clients default to an ExecutionPolicy of Restricted, which refuses to
# run ANY .ps1 -- so invoke it explicitly rather than as `.\install.ps1`. This
# bypasses the policy for one process only; it changes nothing machine-wide.
powershell -ExecutionPolicy Bypass -File .\install.ps1 -Path $HOME\Downloads\sleap-app-windows.zip

# `| iex` cannot forward parameters, so build a script block for -Tag / -Pre.
# (This route is unaffected by ExecutionPolicy -- nothing is ever written to disk.)
& ([scriptblock]::Create((irm https://app.sleap.ai/dev/install.ps1))) -Tag v0.1.2

install.sh --help and Get-Help .\install.ps1 list the rest (--prefix, --force, -Interactive).

Why use the installer?

macOS builds are signed with a Developer ID and notarized by Apple, so the .dmg from the Releases page works on its own -- double-click it, drag SLEAP.app to Applications, done. The first launch shows the standard "downloaded from the Internet, are you sure?" confirmation with an Open button. That appears once, for any downloaded app, and never again.

The installer is a convenience on top of that, not a workaround. It:

• skips even that one-time prompt, because curl never sets com.apple.quarantine
• replaces the app atomically (stages alongside, then renames)
• refuses to overwrite a running copy, so you cannot lose unsaved labels
• picks the right artifact for your platform and architecture automatically

On Windows, SmartScreen may still warn, because the installer is not signed with an EV certificate; that warning has a More info > Run anyway.

On Linux, nothing gates the install. The script prefers the .AppImage, because that is the only Linux payload the in-app updater can replace without root; set SLEAP_PREFER_DEB=1 if you would rather have the .deb in your package manager.

If macOS refuses to open the app

You should not hit this on a release build. If you do -- most likely a build from a fork or a PR, which get no signing secrets and fall back to ad-hoc signing -- clear the quarantine tag on the .dmg, before opening it, which stops the tag propagating to the app in the first place:

xattr -dr com.apple.quarantine ~/Downloads/SLEAP_*.dmg

If you already tried and got blocked, clear it on the installed app instead:

xattr -dr com.apple.quarantine /Applications/SLEAP.app

The GUI route is System Settings > Privacy & Security > Security > Open Anyway, which needs your login password and only offers itself for about an hour after a blocked launch. Control-click > Open no longer works -- Apple removed that bypass in macOS 15.

Two dialogs are worth telling apart. "Apple could not verify..." means valid signature, not notarized. "SLEAP is damaged and can't be opened" means an invalid signature, and has no override at all -- if you ever see that on a release build, the signing step regressed; see docs/macos-code-signing.md.

Tech Stack

Layer	Technology
UI	React 19, TypeScript 5.7, Vite 6, Tailwind CSS v4
Components	shadcn/ui (Radix primitives), black/orange theme
State	Zustand + Immer
Rendering	Canvas 2D API (two-layer: video frame + skeleton overlay)
Data model	@talmolab/sleap-io.js -- SLP/HDF5 via h5wasm
Video	WebCodecs + mp4box.js
Desktop	Tauri v2 (~5 MB vs ~244 MB Electron)
Shortcuts	tinykeys (~400 B)
Testing	bun test (200+ unit tests), Playwright (E2E)

Features

File I/O

• Open .slp files via file picker or drag-and-drop (including .pkg.slp with embedded videos)
• Save / Save As in native SLP format (browser h5wasm writer)
• Export labels as JSON

Video & Navigation

• MP4 playback via WebCodecs with frame-accurate seeking
• Seekbar with labeled frame marks, track occupancy bars, and snap-to-labeled-frame
• Playback speed control (0.25x -- 8x)
• Go to Frame dialog, next/prev labeled frame, next/prev suggestion

Labeling & Editing

• Skeleton overlay with nodes (circles), edges (lines / wedges), and labels
• Click to select instances, drag nodes to reposition
• Add / delete instances and nodes
• Copy / paste instances and tracks
• Right-click context menu for instance and node actions
• Undo / redo with frame-level snapshots via the command pattern

View Controls

• Zoom, pan, and fit-to-instances
• Show / hide: instances, node labels, edges, non-visible nodes
• Color-by mode: Track, Instance, Node, or Edge (View > Apply Distinct Colors To)
• Three color palettes: standard, five+, alphabet
• Edge style: Line or Wedge
• Configurable node marker size

Panels

• Videos -- list and switch between project videos
• Skeleton -- view and edit skeleton nodes and edges
• Instances -- current frame's instances with track, type, score
• Suggestions -- suggested frames for labeling

Keyboard Shortcuts

40+ shortcuts matching SLEAP's defaults:

Action	Shortcut
Open / Save / Save As	Ctrl+O / Ctrl+S / Ctrl+Shift+S
Undo / Redo	Ctrl+Z / Ctrl+Shift+Z
Next / prev frame	Right / Left
Next / prev labeled frame	Alt+Right / Alt+Left
Next / prev suggestion	Space / Shift+Space
Go to frame	Ctrl+J
Add / delete instance	Ctrl+I / Ctrl+Backspace
Select next instance	`
Fit view	Ctrl+=
Transpose tracks	Ctrl+T
New track	Ctrl+0

Development

# Install dependencies
bun install

# Start dev server (browser)
bun run dev          # http://localhost:5173

# Start Tauri dev mode (desktop, requires system deps)
bun run tauri:dev

# Run tests
bun run test         # unit tests (bun, --isolate)
bun run test:e2e     # Playwright E2E tests

# Production builds
bun run build        # Browser (dist/)
bun run tauri:build  # Desktop installer (.msi / .dmg / .deb)

System Dependencies (Linux, for Tauri)

sudo apt-get install libwebkit2gtk-4.1-dev libgtk-3-dev \
  libjavascriptcoregtk-4.1-dev librsvg2-dev patchelf \
  libglib2.0-dev libayatana-appindicator3-dev libdbus-1-dev

sleap-io.js Dependency

The data model and SLP file handling come from @talmolab/sleap-io.js. It can be used from a local checkout or from the npm registry.

Local development (default) -- links to a sibling checkout for developing against unpublished changes:

# Expects ../sleap-io.js to exist (git clone it alongside this repo)
bun add file:../sleap-io.js

bun add updates package.json + bun.lock and installs in one step (there is no bun pkg set equivalent). Alternatively, hand-edit the dependencies."@talmolab/sleap-io.js" field in package.json to "file:../sleap-io.js", then run bun install.

Published package (CI / standalone) -- uses the package from the npm registry:

bun add @talmolab/sleap-io.js@<version>

Or hand-edit the dependencies."@talmolab/sleap-io.js" version in package.json, then run bun install.

The Vite config auto-detects which mode is active by checking whether @talmolab/sleap-io.js in node_modules is a local link (symlink to an out-of-tree checkout) rather than a normal install.

Architecture

src/
├── main.tsx                     # Entry point, exposes window.sleap debug API
├── App.tsx                      # Root component
├── stores/appStore.ts           # Zustand store (selection, view, project state)
├── commands/                    # SLEAP-style command pattern with undo/redo
│   ├── CommandContext.ts        #   Executor with frame-level snapshots
│   ├── fileCommands.ts          #   New, Open, Save, SaveAs, ExportJson
│   ├── navCommands.ts           #   Frame/suggestion/video navigation
│   ├── editCommands.ts          #   Instance/node editing, copy/paste
│   └── trackCommands.ts         #   Track assignment, transpose, copy/paste
├── canvas/SkeletonRenderer.ts   # Canvas 2D overlay renderer + hit testing
├── components/
│   ├── layout/                  #   AppShell, MenuBar, StatusBar, WelcomeScreen
│   ├── video/                   #   VideoPlayer (two-canvas), Seekbar, ContextMenu
│   ├── panels/                  #   Videos, Skeleton, Instances, Suggestions
│   ├── dialogs/                 #   GoToFrame, Training, Inference
│   └── ui/                      #   shadcn/ui component library
├── hooks/                       #   useKeyboardShortcuts, useFileIO
├── lib/
│   ├── colorPalettes.ts         #   Palette definitions + color-by-mode logic
│   ├── loadProject.ts           #   Consolidated SLP loading pipeline
│   ├── saveProject.ts           #   Save SLP via upstream saveSlpToBytes
│   ├── resolveVideos.ts         #   Video backend resolution for .pkg.slp
│   └── shortcuts.ts             #   40+ keyboard shortcut definitions
├── platform/                    #   Tauri vs browser file I/O abstraction
└── types/                       #   TypeScript type definitions

src-tauri/                       # Tauri v2 desktop shell (Rust)
tests/                           # bun unit tests + Playwright E2E

Key Patterns

• Two-canvas rendering -- video frame canvas + skeleton overlay canvas, independently updated for performance
• Command pattern -- every edit goes through CommandContext for undo/redo with frame-level snapshots
• overlayVersion counter -- bumped to force overlay re-renders when mutable data changes without React state changes
• Reference equality -- all labeledFrame lookups use === on video objects (avoids a basename-matching bug in sleap-io.js Labels.find())
• Platform abstraction -- src/platform/ abstracts file I/O so the same codebase runs in Tauri and the browser

Deployment

Deployment is automated via GitHub Actions:

• On merge to main -- the browser app is built and deployed to the dev site at https://app.sleap.ai/dev/ (.github/workflows/deploy.yml, published to the gh-pages branch).

• On GitHub Release (published) -- the desktop installers are built for all three platforms and attached to the release (.github/workflows/build.yml): Linux .deb / .AppImage / .rpm, a universal macOS .dmg, and Windows .msi / -setup.exe, along with a latest.json auto-update manifest. A non-pre-release additionally deploys the browser app to production at https://app.sleap.ai; a pre-release deploys to /dev/ instead, so tester builds never replace the production site.

  Note that latest.json is served from releases/latest/download/, which skips pre-releases -- so the in-app updater only ever sees full releases.

Both targets can also be run manually from the Actions tab (deploy.yml / build.yml workflow_dispatch).

License

BSD-3-Clause. See LICENSE.