# Generation cancellation (client stop / disconnect)

## Purpose

When a user **stops** an in-flight answer (browser cancel, closes the tab, or `AbortController` on `fetch`), the backend should **stop doing work for that request** as soon as practical: no unnecessary LLM calls, SQL generation steps, or database work after the client has gone.

This document describes what was implemented, **why** it works the way it does, and what **clients and operators** need to know.

---

## Why this was needed

1. **Long-running handlers**  
   Ask flows run heavy work: talkback orchestration, training-driven `multi_agent_pipeline`, node-based SQL generation (`PlannerNode` / `GeneratorNode`), API calls, and SQL execution. Without explicit cancellation, the server could keep running **after** the user had already abandoned the request.

2. **Thread pool execution**  
   Much work runs in `run_in_threadpool` (sync code). The async disconnect signal from Starlette does **not** automatically cancel a thread; we must **cooperate** by checking a shared flag between stages.

3. **SQL pipeline was opaque to cancellation**  
   Initially, `cancel_event` did not reach `SQLGenerationPipeline`. Generation could run to completion inside `generate_sql_cached` even when the client had disconnected.

---

## What was implemented (summary)

| Area | Behavior |
|------|----------|
| **HTTP disconnect detection** | A background asyncio task polls `Request.is_disconnected()` (~200ms) and sets a `threading.Event` when the client disconnects. |
| **Before threadpool work** | `_threadpool_if_not_cancelled` checks the event and disconnect state **before** scheduling sync work in the default pool (ask routes). |
| **Training / multi-agent pipeline** | `cancel_event` is passed into `multi_agent_pipeline`, `build_execution_plan`, `execute_plan`, and sub-tasks; `check_cancel` runs between planner steps and at sub-task entry. |
| **Node SQL pipeline** | `cancel_event` is passed via `generate_sql_cached` → `run_sql_generation_pipeline` → `SQLGenerationPipeline.run` and stored on `SQLGenerationState`. `_check_pipeline_cancel` runs between context, planner, generator, and before/around LLM and test-SQL steps. |
| **Mixed multi-agent** | `execute_mixed_multi_agent_sync` checks between FILE iterations and before DB/API parallel work. |
| **Talkback** | The raw ASGI `Request` is passed through `TalkbackState` into `multi_agent_ask_question` as `(body, http_request, current_user)` so the same cancellation path applies. **`PersistTurnNode` skips** `append_turn_to_talkback_session` when the handler result has `cancelled: true`, so “Generation stopped” is not stored as a normal assistant turn. |
| **Stopped response** | `GenerationCancelledError` is mapped to **HTTP 499** with JSON: `success: false`, `cancelled: true`, `message: "Generation stopped"` (where the handler catches it). |

Helpers live in **`app/services/askme_service.py`** (`GenerationCancelledError`, `check_cancel`). The disconnect watcher is defined next to **`_threadpool_if_not_cancelled`** in **`app/routes/askme.py`**.

---

## What the client must do

1. **Actually cancel the HTTP request**  
   Stopping generation in the UI must **abort** the ongoing request (e.g. `AbortController.abort()` with `fetch`, or axios `CancelToken` / `signal`). Simply hiding the spinner without closing the connection will **not** trigger `is_disconnected()` reliably.

2. **Same origin / proxies**  
   Some proxies buffer responses; disconnect detection depends on the ASGI server seeing the connection close. For local `uvicorn` this is usually immediate.

---

## Limitations (important)

1. **Blocking LLM HTTP calls**  
   While a **single** synchronous `submit_prompt` / OpenAI HTTP request is in flight, Python cannot inject cancellation **inside** that call. Stopping takes effect **after** that request returns, at the **next** `check_cancel` / `_check_pipeline_cancel`.

2. **Blocking database calls**  
   Similarly, a long-running `run_sql` or driver call is not interrupted mid-query unless the driver supports cancellation (not implemented here).

3. **499 status**  
   HTTP **499** is a common convention for “client closed request”; not all clients treat it uniformly. The JSON body still carries `cancelled: true` for programmatic handling.

---

## Key files

| File | Role |
|------|------|
| `app/routes/askme.py` | `start_disconnect_watcher`, `_threadpool_if_not_cancelled`, `cancel_event` + `finally` cleanup on `/single-agent-ask-question` and `/multi-agent-ask-question` |
| `app/services/askme_service.py` | `GenerationCancelledError`, `check_cancel`; `generate_sql_cached` passes `cancel_event` into `run_sql_generation_pipeline` (multi-agent loop) |
| `app/services/multi_agent_pipeline.py` | `cancel_event` through plan + execute; `execute_db_agent` passes `cancel_event` into `generate_sql_cached` and checks before `run_sql_cached` |
| `askme_open_agent/askme_base/sql_nodes.py` | `SQLGenerationState.cancel_event`, `_check_pipeline_cancel`, checks in `SQLGenerationPipeline.run`, `PlannerNode`, `GeneratorNode` |
| `app/services/talkback_orchestrator.py` | `http_request` on `TalkbackState`; `ExecuteTalkback` calls `multi_agent_ask_question(body, http_request, current_user)` |

---

## Operational notes

- **Reload** the server after code changes (`uvicorn --reload` picks up edits).
- **Logging**: Pipeline logs (`[multi_agent_pipeline]`, `[Pipeline]`, `[NodePipeline]`) may still print lines until the next cancellation boundary after a stop; that is expected.
- **Tests** that call `multi_agent_ask_question` programmatically should pass a real Starlette `Request` (e.g. from `TestClient`) if cancellation is under test; otherwise `http_request` / `cancel_event` behavior may be skipped where optional.

---

## Related concepts

- **Cooperative cancellation**: Workers must **poll** a flag; there is no universal “kill thread safely” for arbitrary Python code.
- **async vs sync**: Disconnect is observed in **async** code; sync pipelines **must** call `check_cancel` at safe points.
