Web Interface Implementation for Scraper Bots Management
Web interface transforms CLI tool or API into product usable by non-technical employees. Parser management interface includes task list, configuration, run/stop, view results, and error monitoring.
Key Screens
Dashboard — summary statistics: number of active parsers, records collected per day, success rate percentage, recent errors.
Parser List — table with columns: name, source, status (active/paused/error), last run, next scheduled run.
Parser Card — configuration, run history with logs, view latest collected data.
Data — results table with filtering, search, CSV export.
Stack
- React + TypeScript with Vite
- TanStack Query — data fetching and status auto-update
- TanStack Table — tables with sorting and filtering
- Recharts — statistics graphs
- WebSocket / SSE — real-time progress updates
Real-Time Progress Component
import { useEffect, useState } from 'react';
interface RunProgress {
status: 'pending' | 'running' | 'completed' | 'failed';
processed: number;
total: number;
errors: number;
}
function ScraperRunProgress({ runId }: { runId: number }) {
const [progress, setProgress] = useState<RunProgress | null>(null);
useEffect(() => {
const es = new EventSource(`/api/v1/runs/${runId}/progress`);
es.onmessage = (e) => setProgress(JSON.parse(e.data));
es.onerror = () => es.close();
return () => es.close();
}, [runId]);
if (!progress) return <Spinner />;
const pct = progress.total > 0
? Math.round(progress.processed / progress.total * 100)
: 0;
return (
<div className="space-y-2">
<div className="flex justify-between text-sm">
<span>{progress.processed} / {progress.total}</span>
<span className="text-red-500">{progress.errors} errors</span>
</div>
<Progress value={pct} />
</div>
);
}
Parser Configurator
Form with fields: source URL, schedule (cron-picker), headers, proxy settings, field mapping (drag-and-drop from found fields to schema fields).
Timeframe
Web interface with dashboard, results table, and monitoring: 6–10 working days.