Commit Graph

6 Commits

Author SHA1 Message Date
ylytdeng
b93e85a1a0 feat(monitor_web): 导出筛选模态框 — 不再一点就跑全量 (closes #112)
## 痛点

之前 Web UI 工具箱里 "③ 导出聊天" / "⑦ 企业微信导出" 一点就跑全量。
用户 3142 个个人微信会话 + 14 个企微会话, 全量导出几个 GB JSON 几小时,
还没法只导某几个群。

## 设计

点击导出按钮 → 弹模态框选会话 + 格式 → 确认后才跑。

```
┌────────────────────────────────────────┐
│  导出个人微信聊天             [×]      │
├────────────────────────────────────────┤
│  🔍 按名字 / wxid 搜索...               │
│  ┌──────────────────────────────────┐ │
│  │ ☐ [群]   交易所消息    2026-05-14│ │
│  │ ☐ [单聊] 张三          2026-05-13│ │
│  │ ☐ [公众号] xxx日报     2026-05-10│ │
│  │ ...                              │ │
│  └──────────────────────────────────┘ │
│  [全选] [清空] [选最近30天活跃]  已选 N│
│                                        │
│  格式 (仅企微) ☑CSV ☐HTML ☐JSON         │
│                                        │
│         [取消]  [确认导出 →]            │
└────────────────────────────────────────┘
```

## 实现拆解

### Backend
1. **GET /api/sessions?source=wechat|wxwork** — 列会话
   - wechat: 从 decrypted/session/session.db SessionTable 读, 拼合
     contact.db 的 nick_name/remark, type 分群/单聊/公众号
   - wxwork: 从 wxwork_decrypted/session.db conversation_table 读,
     id 前缀分 R/S/E/Y → 群/单聊/外部/其他
   - 按 last_ts 降序

2. **TOOL_TASKS schema 加 build_steps + needs_modal**
   - 旧固定任务: `steps: [cmd, ...]`
   - 新动态任务: `build_steps: fn(users, formats) → [cmd, ...]`
   - export_all / wxwork_export 用 build_steps + needs_modal 标记

3. **_run_tool_task 接收 args**, 优先调 build_steps 生成 cmd

4. **POST /api/tool body 接收 args**: `{task, args: {users:[...], formats:[...]}}`

### CLI 脚本
- `export_all_chats.py` 加 `--users wxid1,wxid2` (alt: env WECHAT_EXPORT_USERS),
  在加载完 sessions 后做白名单过滤, 空集报错退出
- `export_wxwork_messages.py` 原本就支持 `--conversation` (multi-arg) +
  `--formats csv,html,json`, 不动它, monitor_web 拼 argv 即可

### Frontend
- HTML 加 `#exportModal` 模态框骨架 (overlay + dialog + search + list +
  format checkboxes + footer buttons)
- CSS .modal-* 一套 (用 design tokens, 跟整体暗色风格统一)
- JS:
  - NEEDS_MODAL = {export_all: 'export_wechat', wxwork_export: 'export_wxwork'}
  - runTool 拦截这两个 task → openExportModal
  - openExportModal fetch /api/sessions → renderSessions (复选框列表)
  - filterSessions 实时搜索过滤
  - selectAllSessions / selectRecentSessions(30) 批量选择
  - confirmExport 收集 selected usernames + formats → runToolWithArgs
- runTool 拆成 runTool (entry, 拦截/取消) + runToolWithArgs (实际跑)

## 实测验证

```
$ curl -s "http://localhost:5678/api/sessions?source=wxwork"
[{"username": "R:358645240322", "name": "交易所消息", "type": "群", "last_ts": 1778658597, ...},
 {"username": "E:10223", "name": "E:10223", "type": "外部", ...}, ...]

$ curl -s "http://localhost:5678/api/sessions?source=wechat"
[{"username": "46222992238@chatroom", "name": "月下健人", "type": "群", "last_ts": 1779014016, ...}, ...]
```

API 返回正确, 个人微信 3142 个 / 企微 14 个会话, 按 last_ts 降序。
测试 185/185 通过。

## 副带改进

- runToolWithArgs 拆出来后, fix 之前一个小 bug: error 路径里的
  `b.textContent = b.dataset.origText` 改成 innerHTML 路径 (一致性)

## 仍未跟进 (follow-up)

- export_all_chats 加 `--format csv,html` 多格式输出 (个人微信脚本
  目前只支持 JSON, 模态框格式选项对它隐藏了, 看 #109 推进)
- 个人微信导出 sessions 列表性能: 3142 个一次 fetch ~5MB JSON,
  搜索 + render 在低端机可能卡, 可加分页 / 虚拟列表
2026-05-17 18:35:44 +08:00
Davy
de4cb092d9 feat(export): add incremental mode, date range filter, and dry-run
Three new flags for export_all_chats.py:

- -i / --incremental: reads existing JSON, appends only new messages
  (deduplicates by local_id, preserves transcription field on merge)
- --start / --end: filter messages by date range (YYYY-MM-DD or timestamp)
  passes start_ts/end_ts directly to mcp_server._query_messages
- --dry-run: preview mode (shows counts without writing files)

Voice transcription in incremental mode only processes newly appended
voice messages — existing transcribed entries are untouched.
2026-05-14 15:48:38 +08:00
Davy
0ff354138d feat: tqdm progress bar + setup.py wizard 2026-05-14 15:45:00 +08:00
Davy
8645fe4210 feat(export_all): add --with-transcriptions flag for voice transcription during export (#89) 2026-05-13 13:33:36 +08:00
Belugary
f03df51561 feat: parse WeChat transfer messages (appmsg type=2000) (#85)
Add structured parsing for transfer messages so they no longer fall
through to the generic `[链接/文件]` fallback in chat history exports.
Mirrors the dispatch + helper pattern PR #65 (merged-forward type=19)
established for `base_type=49` appmsg sub-types.

## What is added

**Helpers (mcp_server.py):**
- `_TRANSFER_PAYSUBTYPE_LABEL` — maps the 6 community-consensus paysubtypes
  (1 发起 / 3 已收款 / 4 已退还 / 5 过期已退还 / 7 待领取 / 8 已领取);
  unknown values degrade to `未知(paysubtype=N)` so a new variant in a
  future WeChat build is visible rather than silently dropped.
- `_extract_transfer_info(appmsg)` — pulls fields out of `<wcpayinfo>`,
  with snake/camelCase fallback (`feedesc`/`feeDesc`, `pay_memo`/`paymemo`)
  observed across WeChat versions.
- `_format_transfer_message_text(appmsg, title)` — one-line render
  for chat history: `[转账·已收款] ¥100.00 备注: lunch`.

**Dispatch (mcp_server.py):**
- `_format_app_message_text` gains an `app_type == 2000` branch that
  routes to `_format_transfer_message_text`. `get_chat_history`,
  `export_chat`, `export_all_chats` and `monitor_web` all inherit
  automatically.

**New MCP tool (mcp_server.py):**
- `decode_transfer(chat_name, local_id, create_time=0)` — full
  structured view: direction, amount, memo, payer/receiver wxid,
  transfer id, transcation id, begin/invalid timestamps. Uses the
  same multi-shard scan + ambiguity-by-create_time pattern as
  `decode_file_message` / `decode_record_item`.

**CLI wrapper:**
- `decode_transfer.py` at the repo root — argparse wrapper that prints
  the same text as the MCP tool, returning non-zero exit when the
  message can't be decoded (script-friendly).

**JSON export (chat_export_helpers.py + export_chat.py + export_all_chats.py):**
- `_extract_content` now returns `(rendered, extras)`. `extras` carries
  structured fields when a message type has more signal than the
  human-readable string (currently: transfers → `type:"transfer" +
  transfer:{direction, fee_desc, pay_memo, ...}`). The channel is
  forward-compatible — future additions (video号 metadata, expanded
  merged-forward, etc.) flow through the same shape without changing
  the caller signature. JSON consumers that only read `content` are
  unaffected; the change is additive.

**monitor_web (monitor_web.py):**
- Backend dispatch branch + orange-yellow `.msg-transfer` card CSS +
  `renderRich` JS handler.

## Tests

12 new cases in `tests/test_record_decoders.py`:

- `TransferPaysubTypeLabelTests` — locks the 6-value label table.
- `ExtractTransferInfoTests` (6 cases) — full field round-trip, missing
  `<wcpayinfo>` fallback, snake/camelCase variants, unknown paysubtype
  degradation, empty paysubtype handling.
- `FormatTransferMessageTextTests` (4 cases) — initiate / received-with-memo /
  missing-wcpayinfo / missing-fee-desc.
- `AppMessageDispatchTransferTests` — `_format_app_message_text` routes
  type=2000 correctly so `get_chat_history` / `export_chat` both pick
  it up.

All fixtures use synthetic placeholder values (`wxid_payer_synth`,
`¥100.00`, `1` + 27×`0`); no real PII or transaction IDs.

## Scope

7 files, +546 / -15 (additions only — no behavior change for existing
message types). All 180 tests pass locally (168 baseline + 12 new).
2026-05-12 21:03:08 +08:00
Davy
67de4a1d0c feat: 批量导出所有聊天为 JSON + 提取共享 helper 模块 (#77)
* Add export_all_chats.py: batch export all WeChat chats to JSON

Mirrors export_chat.py functionality to export every chat in the
decrypted WeChat database at once. Output format is byte-for-byte
identical to export_chat.py.

Usage: python3 export_all_chats.py [output_dir]

- Reads all sessions from decrypted/session/session.db
- Uses the same content extraction pipeline as export_chat.py:
  _resolve_sender, _extract_content, _msg_type_str, sticker/video/system
- Outputs group_<display>.json or single_<display>.json with same schema
- Progress reporting every 100 exports with summary at end

* refactor: 将 7 个重复 helper 函数提取到 chat_export_helpers.py

根据 PR #77 review 反馈,将 export_chat.py 和 export_all_chats.py 中
逐字复制的消息格式化函数提取到共享模块 chat_export_helpers.py。

提取的函数:
  MSG_TYPE_MAP, _msg_type_str, _resolve_sender,
  _decode_sticker_desc, _format_sticker_message,
  _format_system_message, _format_video_message, _extract_content

两个导出脚本现在从 chat_export_helpers import 所需函数,
消除代码漂移风险。
2026-05-12 11:42:48 +08:00