Compare commits
2 Commits
7dbd00e537
...
readme-pol
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
468519fde1 | ||
|
|
97c502a5f9 |
57
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
Normal file
57
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
Normal file
@@ -0,0 +1,57 @@
|
||||
name: Bug Report
|
||||
description: Report something that is broken or not working as expected
|
||||
labels: ["bug"]
|
||||
title: "[Bug] "
|
||||
body:
|
||||
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Bug Description
|
||||
Describe what broke and what you expected to happen.
|
||||
|
||||
- type: textarea
|
||||
id: repro
|
||||
attributes:
|
||||
label: Steps to Reproduce
|
||||
placeholder: |
|
||||
1.
|
||||
2.
|
||||
3.
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: Expected Behavior
|
||||
placeholder: What should happen instead.
|
||||
|
||||
- type: textarea
|
||||
id: actual
|
||||
attributes:
|
||||
label: Actual Behavior
|
||||
placeholder: What actually happens.
|
||||
|
||||
- type: input
|
||||
id: version
|
||||
attributes:
|
||||
label: UMM Version / Commit
|
||||
placeholder: e.g., v2.0 or git commit hash
|
||||
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Relevant Logs
|
||||
description: Paste any error messages or relevant log lines
|
||||
|
||||
- type: textarea
|
||||
id: env
|
||||
attributes:
|
||||
label: Environment
|
||||
description: OS, Docker version, database version, etc.
|
||||
placeholder: |
|
||||
- OS:
|
||||
- Docker:
|
||||
- PostgreSQL:
|
||||
- Go (if running locally):
|
||||
29
.github/ISSUE_TEMPLATE/feature_request.yml
vendored
Normal file
29
.github/ISSUE_TEMPLATE/feature_request.yml
vendored
Normal file
@@ -0,0 +1,29 @@
|
||||
name: Feature Request
|
||||
description: Suggest a new feature or improvement
|
||||
labels: ["enhancement"]
|
||||
title: "[Feature] "
|
||||
body:
|
||||
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
## Feature Description
|
||||
Describe the feature you'd like and why it would be useful.
|
||||
|
||||
- type: textarea
|
||||
id: usecase
|
||||
attributes:
|
||||
label: Use Case
|
||||
placeholder: Who would use this and what problem does it solve?
|
||||
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Alternatives Considered
|
||||
placeholder: What other approaches did you consider?
|
||||
|
||||
- type: textarea
|
||||
id: context
|
||||
attributes:
|
||||
label: Additional Context
|
||||
placeholder: Screenshots, mockups, or any other relevant information.
|
||||
28
.github/PULL_REQUEST_TEMPLATE.md
vendored
Normal file
28
.github/PULL_REQUEST_TEMPLATE.md
vendored
Normal file
@@ -0,0 +1,28 @@
|
||||
## Description
|
||||
|
||||
Brief summary of what this PR does.
|
||||
|
||||
## Type of Change
|
||||
|
||||
- [ ] Bug fix
|
||||
- [ ] New feature
|
||||
- [ ] Breaking change
|
||||
- [ ] Documentation update
|
||||
|
||||
## Testing
|
||||
|
||||
Describe what testing you performed.
|
||||
|
||||
- [ ] Tested locally (go test ./...)
|
||||
- [ ] Tested in Docker Compose dev environment
|
||||
- [ ] Verified frontend builds (npm run build)
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] My code follows the project conventions (see AGENTS.md)
|
||||
- [ ] I have self-reviewed my own code
|
||||
- [ ] I have commented my code, particularly in hard-to-understand areas
|
||||
- [ ] I have made corresponding changes to the documentation
|
||||
- [ ] My changes generate no new compiler or lint warnings
|
||||
- [ ] I have added tests that prove my fix is effective or my feature works
|
||||
- [ ] New and existing unit tests pass locally with my changes
|
||||
243
README.md
Normal file
243
README.md
Normal file
@@ -0,0 +1,243 @@
|
||||
# Unified Media Manager (UMM)
|
||||
|
||||
> One platform to rule them all — replaces 12 Docker containers with one unified Go + React media management system.
|
||||
|
||||
[](https://github.com/TopherMayor/unified-media-manager)
|
||||
[](https://golang.org/)
|
||||
[](LICENSE)
|
||||
|
||||
## Resume Summary
|
||||
|
||||
**Unified Media Manager** — Designed and built a production-grade media management platform that consolidated 12 separate Docker containers (Sonarr, Radarr, Lidarr, Readarr, Prowlarr, Jellyseerr, Bazarr, Recyclarr, and their anime variants) into a single Go + React service.
|
||||
|
||||
- Reduced infrastructure from **12 containers to 1**, eliminating 11 redundant databases, 11 log schemas, and duplicated configuration
|
||||
- Built a multi-protocol indexer engine supporting Torznab/Newznab XML and Cardigann YAML definitions for compatibility with 50+ indexers
|
||||
- Implemented hardlink-based acquisition so downloads and library files share blocks on disk (zero extra storage)
|
||||
- Designed a PostgreSQL schema with 24 tables using partitioning, JSONB, and GIN indexes — replacing 82+ tables from the arr stack
|
||||
- Built a full-stack React UI with 12 pages: Dashboard, Library, Search, Discover, Queue, Activity, Blocklist, Calendar, Requests, Settings, MediaDetail, and Semantic Search
|
||||
- Implemented AI-powered semantic search using Qdrant vector database and Ollama embeddings for natural-language media discovery
|
||||
- Tech stack: Go 1.25, Echo v4, PostgreSQL 16, React 18 + TypeScript + TailwindCSS, Qdrant, Ollama, Docker
|
||||
|
||||
## What It Does
|
||||
|
||||
UMM is a self-hosted media acquisition and management platform. You add movies, TV shows, music, books, or podcasts you want to watch/read/listen to, and UMM automatically:
|
||||
|
||||
1. **Searches** your configured indexers (via Cardigann or Torznab) for matching releases
|
||||
2. **Evaluates** quality, seeders, and snatch speed to pick the best release
|
||||
3. **Downloads** via qBittorrent or SABnzbd — routed through a VPN (Gluetun) for privacy
|
||||
4. **Imports** completed downloads to your library using hardlinks (no extra disk space)
|
||||
5. **Enriches** metadata from TMDB, TVDB, MusicBrainz, and OpenLibrary
|
||||
6. **Tracks** everything in PostgreSQL with full-text search across your entire library
|
||||
7. **Notifies** you via webhook/Telegram when media is ready
|
||||
8. **Searches semantically** — "quirky indie documentaries about AI" returns relevant results
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Internet
|
||||
│
|
||||
┌──────┴──────┐
|
||||
│ Traefik │ TLS termination (existing infra)
|
||||
└──────┬──────┘
|
||||
│
|
||||
┌─────────────┴─────────────┐
|
||||
│ │
|
||||
┌──────┴──────┐ ┌───────┴───────┐
|
||||
│ Nginx │ │ React UI │
|
||||
│ :3000 │ │ (SPA) │
|
||||
└──────┬──────┘ └───────▲────────┘
|
||||
│ │
|
||||
│ /api/* proxy_pass │ fetch()
|
||||
│ │
|
||||
┌─────────▼──────────────────────────┴───────┐
|
||||
│ Go Backend :8084 │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌────────────┐ ┌────────┐│
|
||||
│ │ REST API │ │ Workers │ │Schedu- ││
|
||||
│ │ (Echo v4)│ │ (queuemgr) │ │ler(cron││
|
||||
│ └────┬─────┘ └─────┬──────┘ └────────┘│
|
||||
│ │ │ │
|
||||
└───────┼──────────────┼─────────────────────┘
|
||||
│ │
|
||||
┌─────────┼──────────────┼────────────────────────┐
|
||||
│ │ │ UM M Network │
|
||||
│ ┌──────▼──────┐ ┌────▼─────┐ ┌─────────────┐│
|
||||
│ │ PostgreSQL │ │ Qdrant │ │ Ollama ││
|
||||
│ │ :5432 │ │ :6333 │ │ :11434 ││
|
||||
│ │ (primary) │ │ (vectors)│ │ (AI/LLM) ││
|
||||
│ └─────────────┘ └───────────┘ └─────────────┘│
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────┐ │
|
||||
│ │ Gluetun VPN Network │ │
|
||||
│ │ SABnzbd :8080 │ qBittorrent :8085 │ │
|
||||
│ └──────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────┘
|
||||
|
||||
Media files flow:
|
||||
Indexers → UMM (evaluate) → Download Client (via VPN)
|
||||
→ Import (hardlink) → Library (NFS share) → Jellyfin/Navidrome/Calibre
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| **Unified Search** | Search across all media types (movies, TV, music, books, podcasts) in a single query |
|
||||
| **Indexer Integration** | Cardigann YAML + Torznab/Newznab XML support — works with 50+ indexers |
|
||||
| **Smart Downloads** | Quality-first release selection with seeder/speed scoring |
|
||||
| **Hardlinks** | Completed downloads hard-linked to library — no extra storage |
|
||||
| **Quality Tracking** | Current vs. desired quality per item with auto-upgrade rules |
|
||||
| **Request System** | Built-in request/approval workflow — no separate Jellyseerr needed |
|
||||
| **Subtitle Search** | Automatic subtitle search across multiple sources |
|
||||
| **RSS Sync** | Monitor indexer RSS feeds for new releases automatically |
|
||||
| **Activity Feed** | Real-time event log: searches, downloads, imports, failures |
|
||||
| **Blocklist** | Block specific releases or indexers to prevent re-snatch |
|
||||
| **Semantic Search** | Natural language queries ("dark comedies from the 90s") via Qdrant + Ollama |
|
||||
| **Calendar View** | See upcoming releases by air date or your watchlist |
|
||||
| **Analytics Dashboard** | Disk usage, download stats, library breakdown by media type |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
| Layer | Technology |
|
||||
|-------|-----------|
|
||||
| Backend | Go 1.25 (Echo v4 HTTP framework) |
|
||||
| Frontend | React 18 + TypeScript + TailwindCSS + Vite |
|
||||
| Database | PostgreSQL 16 (partitioned tables, JSONB, GIN indexes) |
|
||||
| Vector DB | Qdrant (semantic search) |
|
||||
| AI | Ollama (nomic-embed-text embeddings, LLaVA vision) |
|
||||
| Download | qBittorrent, SABnzbd (via VPN tunnel) |
|
||||
| Indexers | Cardigann YAML + Torznab/Newznab XML |
|
||||
| Container | Docker + Docker Compose |
|
||||
| Reverse Proxy | Traefik (existing) |
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Docker + Docker Compose v3.8
|
||||
- PostgreSQL 16 instance (or use the Docker default)
|
||||
- qBittorrent or SABnzbd (with VPN via Gluetun recommended)
|
||||
- TMDB API key (free at https://www.themoviedb.org/)
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/TopherMayor/unified-media-manager.git
|
||||
cd unified-media-manager
|
||||
|
||||
# Copy and edit environment
|
||||
cp .env.example .env
|
||||
$EDITOR .env
|
||||
|
||||
# Start the stack
|
||||
docker compose up -d
|
||||
|
||||
# Run database migrations
|
||||
docker compose exec umm /umm migrate
|
||||
|
||||
# Open the UI
|
||||
open http://umm.local.tophermayor.com
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Environment variables (see `.env.example`):
|
||||
|
||||
```bash
|
||||
DATABASE_URL=postgres://user:pass@postgres-shared:5432/unified_media_manager?sslmode=disable
|
||||
QDRANT_URL=http://qdrant:6333
|
||||
OLLAMA_URL=http://ollama:11434
|
||||
TMDB_API_KEY=your_tmdb_api_key
|
||||
PORT=8084
|
||||
```
|
||||
|
||||
### Docker Compose
|
||||
|
||||
The stack uses Traefik for routing. Add this label to your `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.umm.rule=Host(`umm.local.tophermayor.com`)"
|
||||
- "traefik.http.routers.umm.tls=true"
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
unified-media-manager/
|
||||
├── cmd/
|
||||
│ └── migrate/ # Database migration CLI
|
||||
├── internal/
|
||||
│ ├── api/ # Echo HTTP handlers (12 resource files)
|
||||
│ │ ├── activity.go
|
||||
│ │ ├── blocklist.go
|
||||
│ │ ├── calendar.go
|
||||
│ │ ├── dashboard.go
|
||||
│ │ ├── discover.go
|
||||
│ │ ├── download_clients.go
|
||||
│ │ ├── health.go
|
||||
│ │ ├── import.go
|
||||
│ │ ├── indexers.go
|
||||
│ │ ├── media.go
|
||||
│ │ ├── metadata.go
|
||||
│ │ ├── notifications.go
|
||||
│ │ ├── quality.go
|
||||
│ │ ├── queue.go
|
||||
│ │ ├── requests.go
|
||||
│ │ ├── root_folder.go
|
||||
│ │ ├── router.go # Echo router setup
|
||||
│ │ ├── search.go
|
||||
│ │ ├── subtitle.go
|
||||
│ │ ├── tag.go
|
||||
│ │ └── workers.go
|
||||
│ ├── cardigann/ # Cardigann YAML indexer engine
|
||||
│ ├── config/ # Environment config loading
|
||||
│ ├── db/ # PostgreSQL pool + migrations
|
||||
│ │ └── migrations/ # 13 SQL migration files
|
||||
│ ├── download/ # qBittorrent + SABnzbd clients
|
||||
│ ├── migrate/ # arr migration tool (import from Sonarr/Radarr)
|
||||
│ ├── service/ # Business logic (30+ files)
|
||||
│ └── worker/ # Background workers (scheduler, queue, scanner...)
|
||||
├── frontend/
|
||||
│ └── src/
|
||||
│ ├── api/ # TypeScript API client
|
||||
│ ├── components/ # Reusable UI components
|
||||
│ └── pages/ # 12 page components
|
||||
├── scripts/
|
||||
│ └── migrate-arrs.sh # Batch import from arr stack
|
||||
├── docker-compose.yml
|
||||
├── SPEC.md # Full technical specification
|
||||
└── AGENTS.md # AI agent coding conventions
|
||||
```
|
||||
|
||||
## API Endpoints (selected)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/api/search?q=...&type=...` | Full-text search across library |
|
||||
| `GET` | `/api/search/semantic?q=...&k=5` | Semantic search via Qdrant + Ollama |
|
||||
| `GET` | `/api/media` | Paginated media list with filters |
|
||||
| `GET` | `/api/media/:id` | Media detail with releases |
|
||||
| `POST` | `/api/media/:id/download` | Trigger download for specific release |
|
||||
| `GET` | `/api/queue` | Active download queue |
|
||||
| `DELETE` | `/api/queue/:id` | Cancel queued download |
|
||||
| `GET` | `/api/discover/recommended` | AI-recommended media |
|
||||
| `POST` | `/api/indexers/search` | Search all configured indexers |
|
||||
| `GET` | `/api/activity` | Recent activity events |
|
||||
| `GET` | `/api/dashboard/stats` | Dashboard metrics |
|
||||
| `POST` | `/api/requests` | Create media request |
|
||||
| `GET` | `/api/health` | Health check |
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
|
||||
3. Commit your changes (`git commit -m 'Add amazing feature'`)
|
||||
4. Push to the branch (`git push origin feature/amazing-feature`)
|
||||
5. Open a Pull Request
|
||||
|
||||
## License
|
||||
|
||||
MIT License — see [LICENSE](LICENSE) for details.
|
||||
@@ -15,6 +15,7 @@ const Activity = lazy(() => import('./pages/Activity'))
|
||||
const Blocklist = lazy(() => import('./pages/Blocklist'))
|
||||
const Settings = lazy(() => import('./pages/Settings'))
|
||||
const Search = lazy(() => import('./pages/Search'))
|
||||
const SemanticSearch = lazy(() => import('./pages/SemanticSearch'))
|
||||
|
||||
const navItems = [
|
||||
{ to: '/', label: 'Dashboard' },
|
||||
@@ -23,6 +24,7 @@ const navItems = [
|
||||
{ to: '/calendar', label: 'Calendar' },
|
||||
{ to: '/queue', label: 'Queue' },
|
||||
{ to: '/search', label: 'Search' },
|
||||
{ to: '/semantic-search', label: 'Semantic Search' },
|
||||
{ to: '/activity', label: 'Activity' },
|
||||
{ to: '/requests', label: 'Requests' },
|
||||
{ to: '/blocklist', label: 'Blocklist' },
|
||||
@@ -62,6 +64,7 @@ export default function App() {
|
||||
<Route path="/library/:type/:id" element={<MediaDetail />} />
|
||||
<Route path="/queue" element={<Queue />} />
|
||||
<Route path="/search" element={<Search />} />
|
||||
<Route path="/semantic-search" element={<SemanticSearch />} />
|
||||
<Route path="/activity" element={<Activity />} />
|
||||
<Route path="/requests" element={<Requests />} />
|
||||
<Route path="/blocklist" element={<Blocklist />} />
|
||||
|
||||
163
frontend/src/pages/SemanticSearch.tsx
Normal file
163
frontend/src/pages/SemanticSearch.tsx
Normal file
@@ -0,0 +1,163 @@
|
||||
import { useState } from 'react'
|
||||
import { fetchAPI, postAPI } from '../api/client'
|
||||
import { useToast } from '../components/Toast'
|
||||
import ErrorBanner from '../components/ErrorBanner'
|
||||
|
||||
interface SemanticResult {
|
||||
id: number
|
||||
title: string
|
||||
media_type: string
|
||||
year: number | null
|
||||
score: number
|
||||
overview: string
|
||||
}
|
||||
|
||||
function scorePercent(score: number): string {
|
||||
return `${Math.round(score * 100)}%`
|
||||
}
|
||||
|
||||
function mediaTypeBadge(type: string): string {
|
||||
const colors: Record<string, string> = {
|
||||
movie: 'bg-blue-600',
|
||||
series: 'bg-green-600',
|
||||
music: 'bg-purple-600',
|
||||
book: 'bg-orange-600',
|
||||
audiobook: 'bg-yellow-600',
|
||||
podcast: 'bg-pink-600',
|
||||
}
|
||||
return colors[type] ?? 'bg-gray-600'
|
||||
}
|
||||
|
||||
export default function SemanticSearch() {
|
||||
const [query, setQuery] = useState('')
|
||||
const [results, setResults] = useState<SemanticResult[]>([])
|
||||
const [loading, setLoading] = useState(false)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
const [hasSearched, setHasSearched] = useState(false)
|
||||
const [addingToQueue, setAddingToQueue] = useState<Set<number>>(new Set())
|
||||
const { showToast } = useToast()
|
||||
|
||||
async function doSearch() {
|
||||
const trimmed = query.trim()
|
||||
if (!trimmed) return
|
||||
setLoading(true)
|
||||
setHasSearched(true)
|
||||
setError(null)
|
||||
try {
|
||||
const res = await fetchAPI<{ results: SemanticResult[] }>(
|
||||
'/api/search/semantic?q=' + encodeURIComponent(trimmed) + '&k=5'
|
||||
)
|
||||
setResults(res.results ?? [])
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : 'Semantic search failed'
|
||||
setError(message)
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}
|
||||
|
||||
function handleSearchSubmit(e: React.FormEvent) {
|
||||
e.preventDefault()
|
||||
doSearch()
|
||||
}
|
||||
|
||||
async function handleAddToQueue(item: SemanticResult) {
|
||||
setAddingToQueue(prev => new Set(prev).add(item.id))
|
||||
try {
|
||||
await postAPI<{ id: number }>('/api/queue', { media_id: item.id })
|
||||
showToast(`Added "${item.title}" to queue`)
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : 'Failed to add to queue'
|
||||
showToast(`Error: ${message}`)
|
||||
} finally {
|
||||
setAddingToQueue(prev => {
|
||||
const next = new Set(prev)
|
||||
next.delete(item.id)
|
||||
return next
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<div>
|
||||
<h2 className="text-2xl font-semibold text-gray-100 mb-6">Semantic Search</h2>
|
||||
|
||||
<form onSubmit={handleSearchSubmit} className="mb-6">
|
||||
<div className="flex gap-3 max-w-2xl">
|
||||
<input
|
||||
type="text"
|
||||
value={query}
|
||||
onChange={e => setQuery(e.target.value)}
|
||||
placeholder="Describe what you're looking for..."
|
||||
className="bg-gray-800 border border-gray-700 focus:border-indigo-500 text-white rounded-lg px-4 py-3 w-full outline-none transition-colors"
|
||||
/>
|
||||
<button
|
||||
type="submit"
|
||||
disabled={loading || !query.trim()}
|
||||
className="bg-indigo-600 hover:bg-indigo-500 disabled:opacity-50 disabled:cursor-not-allowed text-white px-6 py-3 rounded-lg font-semibold transition-colors"
|
||||
>
|
||||
{loading ? 'Searching...' : 'Search'}
|
||||
</button>
|
||||
</div>
|
||||
<p className="text-gray-500 text-xs mt-2">AI-powered search using natural language descriptions</p>
|
||||
</form>
|
||||
|
||||
{!hasSearched && (
|
||||
<div className="text-center py-20 text-gray-500">
|
||||
<p className="text-lg">Enter a natural language query to find media</p>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{hasSearched && loading && (
|
||||
<div className="animate-pulse space-y-4">
|
||||
{[1, 2, 3].map(i => (
|
||||
<div key={i} className="h-24 bg-gray-800 rounded-lg" />
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{hasSearched && error && !loading && <ErrorBanner error={error} onRetry={doSearch} />}
|
||||
|
||||
{hasSearched && !loading && !error && results.length === 0 && (
|
||||
<div className="text-gray-500 text-center py-12">No results found</div>
|
||||
)}
|
||||
|
||||
{hasSearched && !loading && !error && results.length > 0 && (
|
||||
<div className="space-y-3">
|
||||
<p className="text-gray-400 text-sm mb-3">{results.length} result{results.length !== 1 ? 's' : ''}</p>
|
||||
{results.map(item => {
|
||||
const isAdding = addingToQueue.has(item.id)
|
||||
return (
|
||||
<div key={item.id} className="bg-gray-900 border border-gray-800 rounded-lg p-4 hover:border-gray-700 transition-colors">
|
||||
<div className="flex items-start justify-between gap-4">
|
||||
<div className="flex-1 min-w-0">
|
||||
<div className="flex items-center gap-2 mb-1">
|
||||
<span className={`${mediaTypeBadge(item.media_type)} text-white text-xs font-semibold px-2 py-0.5 rounded`}>
|
||||
{item.media_type}
|
||||
</span>
|
||||
<h3 className="text-gray-100 font-medium truncate">{item.title}</h3>
|
||||
{item.year && <span className="text-gray-500 text-sm">{item.year}</span>}
|
||||
</div>
|
||||
<div className="flex items-center gap-3 mb-2">
|
||||
<span className="text-indigo-400 text-sm font-medium">{scorePercent(item.score)} match</span>
|
||||
</div>
|
||||
{item.overview && (
|
||||
<p className="text-gray-400 text-sm line-clamp-2">{item.overview}</p>
|
||||
)}
|
||||
</div>
|
||||
<button
|
||||
disabled={isAdding}
|
||||
onClick={() => handleAddToQueue(item)}
|
||||
className="px-3 py-1.5 text-xs font-semibold rounded bg-indigo-600 hover:bg-indigo-500 text-white disabled:opacity-50 disabled:cursor-not-allowed whitespace-nowrap transition-colors"
|
||||
>
|
||||
{isAdding ? 'Adding...' : 'Add to Queue'}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
})}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -35,6 +35,7 @@ type Services struct {
|
||||
Discover *service.DiscoverService
|
||||
MediaDetail *service.MediaDetailService
|
||||
Calendar *service.CalendarService
|
||||
SemanticSearch *service.SemanticSearchService
|
||||
}
|
||||
|
||||
func NewRouter(cfg *config.Config, svc *Services) *echo.Echo {
|
||||
@@ -148,6 +149,11 @@ func NewRouter(cfg *config.Config, svc *Services) *echo.Echo {
|
||||
// Calendar route
|
||||
g.GET("/calendar", listCalendarEvents(svc.Calendar))
|
||||
|
||||
// Semantic search route
|
||||
if svc.SemanticSearch != nil {
|
||||
g.GET("/search/semantic", semanticSearch(svc.SemanticSearch))
|
||||
}
|
||||
|
||||
// Request routes — protected by API key auth
|
||||
apiKeyAuth := newAPIKeyAuth(svc.User)
|
||||
g.GET("/requests", listRequests(svc.Request, svc.User), apiKeyAuth)
|
||||
|
||||
@@ -54,6 +54,35 @@ func searchReleases(svc *service.SearchService) echo.HandlerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
func semanticSearch(svc *service.SemanticSearchService) echo.HandlerFunc {
|
||||
return func(c echo.Context) error {
|
||||
ctx, cancel := context.WithTimeout(c.Request().Context(), 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
query := c.QueryParam("q")
|
||||
if query == "" {
|
||||
return c.JSON(http.StatusBadRequest, map[string]string{"error": "q parameter is required"})
|
||||
}
|
||||
|
||||
k := 5
|
||||
if kStr := c.QueryParam("k"); kStr != "" {
|
||||
if kVal, err := strconv.Atoi(kStr); err == nil && kVal > 0 && kVal <= 50 {
|
||||
k = kVal
|
||||
}
|
||||
}
|
||||
|
||||
results, err := svc.Search(ctx, query, k)
|
||||
if err != nil {
|
||||
slog.Error("semantic search failed", "error", err)
|
||||
return c.JSON(http.StatusInternalServerError, map[string]string{"error": err.Error()})
|
||||
}
|
||||
|
||||
return c.JSON(http.StatusOK, map[string]interface{}{
|
||||
"results": results,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func grabRelease(svc *service.SearchService, dcSvc *service.DownloadClientService, queueSvc *service.QueueService, safetySvc *service.SafetyService, activitySvc *service.ActivityService) echo.HandlerFunc {
|
||||
return func(c echo.Context) error {
|
||||
ctx, cancel := context.WithTimeout(c.Request().Context(), 30*time.Second)
|
||||
|
||||
188
internal/service/semantic_search.go
Normal file
188
internal/service/semantic_search.go
Normal file
@@ -0,0 +1,188 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
type SemanticSearchResult struct {
|
||||
ID int64 `json:"id"`
|
||||
Title string `json:"title"`
|
||||
MediaType string `json:"media_type"`
|
||||
Year *int `json:"year,omitempty"`
|
||||
Score float64 `json:"score"`
|
||||
Overview string `json:"overview,omitempty"`
|
||||
}
|
||||
|
||||
type ollamaEmbedRequest struct {
|
||||
Model string `json:"model"`
|
||||
Prompt string `json:"prompt"`
|
||||
}
|
||||
|
||||
type ollamaEmbedResponse struct {
|
||||
Embedding []float64 `json:"embedding"`
|
||||
}
|
||||
|
||||
type qdrantSearchRequest struct {
|
||||
Vector []float64 `json:"vector"`
|
||||
Limit int `json:"limit"`
|
||||
WithPayload bool `json:"with_payload"`
|
||||
}
|
||||
|
||||
type qdrantSearchResponse struct {
|
||||
Result []qdrantHit `json:"result"`
|
||||
}
|
||||
|
||||
type qdrantHit struct {
|
||||
ID string `json:"id"`
|
||||
Score float64 `json:"score"`
|
||||
Payload json.RawMessage `json:"payload"`
|
||||
}
|
||||
|
||||
type qdrantPayload struct {
|
||||
ID int64 `json:"id"`
|
||||
Title string `json:"title"`
|
||||
MediaType string `json:"media_type"`
|
||||
Year *int `json:"year,omitempty"`
|
||||
Overview string `json:"overview,omitempty"`
|
||||
}
|
||||
|
||||
type SemanticSearchService struct {
|
||||
ollamaURL string
|
||||
qdrantURL string
|
||||
httpClient *http.Client
|
||||
}
|
||||
|
||||
func NewSemanticSearchService(ollamaURL, qdrantURL string) *SemanticSearchService {
|
||||
return &SemanticSearchService{
|
||||
ollamaURL: ollamaURL,
|
||||
qdrantURL: qdrantURL,
|
||||
httpClient: &http.Client{
|
||||
Timeout: 30 * time.Second,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func (s *SemanticSearchService) Search(ctx context.Context, query string, k int) ([]SemanticSearchResult, error) {
|
||||
embedding, err := s.embed(ctx, query)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("generate embedding: %w", err)
|
||||
}
|
||||
|
||||
results, err := s.searchQdrant(ctx, embedding, k)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("search qdrant: %w", err)
|
||||
}
|
||||
|
||||
return results, nil
|
||||
}
|
||||
|
||||
func (s *SemanticSearchService) embed(ctx context.Context, text string) ([]float64, error) {
|
||||
reqBody := ollamaEmbedRequest{
|
||||
Model: "nomic-embed-text",
|
||||
Prompt: text,
|
||||
}
|
||||
|
||||
bodyBytes, err := json.Marshal(reqBody)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal embed request: %w", err)
|
||||
}
|
||||
|
||||
reqCtx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer cancel()
|
||||
|
||||
req, err := http.NewRequestWithContext(reqCtx, http.MethodPost, s.ollamaURL+"/api/embeddings", bytes.NewReader(bodyBytes))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create embed request: %w", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
|
||||
resp, err := s.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("ollama request failed: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
respBody, _ := io.ReadAll(io.LimitReader(resp.Body, 1024))
|
||||
slog.Error("ollama embed returned non-200", "status", resp.StatusCode, "body", string(respBody))
|
||||
return nil, fmt.Errorf("ollama returned status %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
var embedResp ollamaEmbedResponse
|
||||
if err := json.NewDecoder(resp.Body).Decode(&embedResp); err != nil {
|
||||
return nil, fmt.Errorf("decode ollama response: %w", err)
|
||||
}
|
||||
|
||||
if len(embedResp.Embedding) == 0 {
|
||||
return nil, fmt.Errorf("ollama returned empty embedding")
|
||||
}
|
||||
|
||||
return embedResp.Embedding, nil
|
||||
}
|
||||
|
||||
func (s *SemanticSearchService) searchQdrant(ctx context.Context, embedding []float64, k int) ([]SemanticSearchResult, error) {
|
||||
reqBody := qdrantSearchRequest{
|
||||
Vector: embedding,
|
||||
Limit: k,
|
||||
WithPayload: true,
|
||||
}
|
||||
|
||||
bodyBytes, err := json.Marshal(reqBody)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshal qdrant request: %w", err)
|
||||
}
|
||||
|
||||
reqCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
|
||||
url := fmt.Sprintf("%s/collections/media/points/search", s.qdrantURL)
|
||||
req, err := http.NewRequestWithContext(reqCtx, http.MethodPost, url, bytes.NewReader(bodyBytes))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create qdrant request: %w", err)
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
|
||||
resp, err := s.httpClient.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("qdrant request failed: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
respBody, _ := io.ReadAll(io.LimitReader(resp.Body, 1024))
|
||||
slog.Error("qdrant search returned non-200", "status", resp.StatusCode, "body", string(respBody))
|
||||
return nil, fmt.Errorf("qdrant returned status %d", resp.StatusCode)
|
||||
}
|
||||
|
||||
var searchResp qdrantSearchResponse
|
||||
if err := json.NewDecoder(resp.Body).Decode(&searchResp); err != nil {
|
||||
return nil, fmt.Errorf("decode qdrant response: %w", err)
|
||||
}
|
||||
|
||||
results := make([]SemanticSearchResult, 0, len(searchResp.Result))
|
||||
for _, hit := range searchResp.Result {
|
||||
var payload qdrantPayload
|
||||
if err := json.Unmarshal(hit.Payload, &payload); err != nil {
|
||||
slog.Error("failed to unmarshal qdrant payload", "error", err)
|
||||
continue
|
||||
}
|
||||
|
||||
results = append(results, SemanticSearchResult{
|
||||
ID: payload.ID,
|
||||
Title: payload.Title,
|
||||
MediaType: payload.MediaType,
|
||||
Year: payload.Year,
|
||||
Score: hit.Score,
|
||||
Overview: payload.Overview,
|
||||
})
|
||||
}
|
||||
|
||||
return results, nil
|
||||
}
|
||||
143
internal/service/semantic_search_test.go
Normal file
143
internal/service/semantic_search_test.go
Normal file
@@ -0,0 +1,143 @@
|
||||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSemanticSearchService_Search(t *testing.T) {
|
||||
// Mock Ollama server
|
||||
ollamaServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/api/embeddings" {
|
||||
t.Errorf("expected /api/embeddings, got %s", r.URL.Path)
|
||||
}
|
||||
var req ollamaEmbedRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
t.Fatalf("failed to decode request: %v", err)
|
||||
}
|
||||
if req.Model != "nomic-embed-text" {
|
||||
t.Errorf("expected model nomic-embed-text, got %s", req.Model)
|
||||
}
|
||||
if req.Prompt == "" {
|
||||
t.Error("expected non-empty prompt")
|
||||
}
|
||||
// Return a 768-dim embedding (truncated for test)
|
||||
embedding := make([]float64, 768)
|
||||
embedding[0] = 0.1
|
||||
embedding[1] = -0.2
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(ollamaEmbedResponse{Embedding: embedding})
|
||||
}))
|
||||
defer ollamaServer.Close()
|
||||
|
||||
// Mock Qdrant server
|
||||
qdrantServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/collections/media/points/search" {
|
||||
t.Errorf("expected /collections/media/points/search, got %s", r.URL.Path)
|
||||
}
|
||||
var req qdrantSearchRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
t.Fatalf("failed to decode request: %v", err)
|
||||
}
|
||||
if req.Limit != 5 {
|
||||
t.Errorf("expected limit 5, got %d", req.Limit)
|
||||
}
|
||||
if !req.WithPayload {
|
||||
t.Error("expected WithPayload=true")
|
||||
}
|
||||
payload, _ := json.Marshal(qdrantPayload{
|
||||
ID: 42,
|
||||
Title: "Blade Runner 2049",
|
||||
MediaType: "movie",
|
||||
Year: intPtr(2017),
|
||||
Overview: "A young blade runner's discovery of a long-buried secret leads him to track down former blade runner Rick Deckard.",
|
||||
})
|
||||
resp := qdrantSearchResponse{
|
||||
Result: []qdrantHit{
|
||||
{ID: "42", Score: 0.8712, Payload: payload},
|
||||
},
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(resp)
|
||||
}))
|
||||
defer qdrantServer.Close()
|
||||
|
||||
svc := NewSemanticSearchService(ollamaServer.URL, qdrantServer.URL)
|
||||
results, err := svc.Search(context.Background(), "sci-fi noir film", 5)
|
||||
if err != nil {
|
||||
t.Fatalf("Search failed: %v", err)
|
||||
}
|
||||
|
||||
if len(results) != 1 {
|
||||
t.Fatalf("expected 1 result, got %d", len(results))
|
||||
}
|
||||
if results[0].ID != 42 {
|
||||
t.Errorf("expected ID 42, got %d", results[0].ID)
|
||||
}
|
||||
if results[0].Title != "Blade Runner 2049" {
|
||||
t.Errorf("expected title 'Blade Runner 2049', got %s", results[0].Title)
|
||||
}
|
||||
if results[0].MediaType != "movie" {
|
||||
t.Errorf("expected media_type 'movie', got %s", results[0].MediaType)
|
||||
}
|
||||
if results[0].Score != 0.8712 {
|
||||
t.Errorf("expected score 0.8712, got %f", results[0].Score)
|
||||
}
|
||||
if results[0].Year == nil || *results[0].Year != 2017 {
|
||||
t.Error("expected year 2017")
|
||||
}
|
||||
}
|
||||
|
||||
// Note: empty query validation happens at the API handler level (search.go),
|
||||
// not in the service. The service layer trusts its callers.
|
||||
func TestSemanticSearchService_embed_emptyQuery(t *testing.T) {
|
||||
ollamaServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
// Empty string is passed through to Ollama — Ollama may return an error
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
}))
|
||||
defer ollamaServer.Close()
|
||||
|
||||
svc := NewSemanticSearchService(ollamaServer.URL, "http://localhost:6333")
|
||||
_, err := svc.embed(context.Background(), "")
|
||||
if err == nil {
|
||||
t.Error("expected error for empty query passed to Ollama")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSemanticSearchService_qdrantError(t *testing.T) {
|
||||
ollamaServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
embedding := make([]float64, 768)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(ollamaEmbedResponse{Embedding: embedding})
|
||||
}))
|
||||
defer ollamaServer.Close()
|
||||
|
||||
qdrantServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusServiceUnavailable)
|
||||
}))
|
||||
defer qdrantServer.Close()
|
||||
|
||||
svc := NewSemanticSearchService(ollamaServer.URL, qdrantServer.URL)
|
||||
_, err := svc.Search(context.Background(), "test query", 5)
|
||||
if err == nil {
|
||||
t.Fatal("expected error when Qdrant is unavailable")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSemanticSearchService_ollamaError(t *testing.T) {
|
||||
ollamaServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusServiceUnavailable)
|
||||
}))
|
||||
defer ollamaServer.Close()
|
||||
|
||||
svc := NewSemanticSearchService(ollamaServer.URL, "http://localhost:6333")
|
||||
_, err := svc.Search(context.Background(), "test query", 5)
|
||||
if err == nil {
|
||||
t.Fatal("expected error when Ollama is unavailable")
|
||||
}
|
||||
}
|
||||
|
||||
func intPtr(i int) *int { return &i }
|
||||
Reference in New Issue
Block a user