e26a83afd9fbf5647e14d132b3f1c8fc0f47d99a
20 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Belugary
|
403f014ac0 |
feat: 新增 decode-images 子命令(批量解密 .dat 图片到明文图片树)
## 问题 \`decode_image.py\` 目前只有 \`decrypt_dat_file()\` 单文件 API,以及 \`monitor_web\` 在收到新消息时\"按需解一张\"的路径。**没有\"一次性扫 attach 目录、产出明文图片树到固定路径\"的批量入口**。结果是任何想把微信图片做下游消费(数据分析、搜索索引、归档、第三方 viewer)的用户都得各自写一遍 walk + decrypt 的 wrapper,且各自约定输出布局,生态不收敛。 ## 修复 - \`decode_image.py\` 新增 \`decode_all_dats(attach_dir, out_dir, aes_key, xor_key, force, on_file)\` 函数,扫描 \`<attach_dir>/<chat_hash>/<YYYY-MM>/Img/*.dat\` 并镜像产出 \`<out_dir>/<chat_hash>/<YYYY-MM>/<file_md5>.<ext>\`。 - \`main.py\` 新增 \`decode-images\` 子命令(早路由,跳过 \`check_wechat_running\` 和 \`ensure_keys\` —— 这条路径只读 \`.dat\` 文件,既不需要微信进程也不需要 DB 密钥)。 设计选择: - **输出布局 1:1 镜像 attach**,只做最小 path massage(去 \`Img/\`、去 \`_t/_h\` 缩略图后缀、换扩展名),不发明新结构。下游能用 \`md5(username)\` 反推路径,无需读 mapping 文件。 - **幂等性 = 按 basename 存在性 skip**,不做 mtime 比较 —— \`.dat\` 是 content-hash 命名(\`file_md5 = 文件内容 md5\`),实际上 write-once。\`--force\` 强制重解。 - **原子写**:解密先写 \`<basename>.<ext>.tmp\`(同目录),\`os.replace\` 到正式路径。中断不留半个 jpg。残留 \`.tmp\` 不会被 skip 误判(glob 显式排除)。 - **错误隔离**:单文件失败计入 \`failed\` 继续下一个,stderr 打 \`[WARN]\` 指出相对路径。退出码 2 表示\"部分失败,产物部分可用\"。 - **V2 无 key**:计入 \`skipped_no_key\` 而非 \`failed\` —— 这是可恢复状态(跑 \`find_image_key_macos.py\` / \`find_image_key.py\` 后重跑即可),跟\"真失败\"区分对待。V1 / 老 XOR 不依赖 \`image_aes_key\`。 - **wxgf 容器**只产 \`.hevc\` 裸流,**不**做 mp4 转换:上游不引入 ffmpeg subprocess 依赖,转换是消费层职责。 - **CLI override**:\`--attach-dir\` / \`--decoded-dir\` / \`--aes-key\` / \`--xor-key\` / \`--force\` 都可覆盖 \`config.json\`,适合 CI / 多账号 / 容器化场景。 ## 测试 新文件 \`tests/test_decode_images_batch.py\`,13 个新测试: - \`PathParsingTests\` (4):glob 命中 / \`_t\` 后缀剥离 / \`_h\` 后缀剥离 / chat_hash + YYYY-MM 镜像 - \`IdempotentTests\` (3):已存在跳过 / \`--force\` 覆写 / 残留 \`.tmp\` 不误判 - \`AtomicWriteTests\` (3):成功路径无 \`.tmp\` / decrypt 返回 None 无 \`.tmp\` / decrypt 抛异常无 \`.tmp\` - \`V2NoKeyTests\` (2):V2 + 无 key → skipped_no_key / V1 + 无 key 仍解码 - \`CallbackTests\` (1):\`on_file\` 回调每文件触发 基线 183 → 196 通过(+13 新增),0 回归。\`decrypt_dat_file\` 用 mock 隔离(避免依赖真实加密图片);\`is_v2_format\` 走真实 magic 检测路径。 ## 范围 - \`decode_image.py\`:新增 \`decode_all_dats\` 函数,134 行,纯加,不改任何现有 API。 - \`main.py\`:新增 \`_run_decode_images\` helper + 早路由 + 用法 hint,104 行加 2 行删。无 backward-compat 影响。 - \`tests/test_decode_images_batch.py\`:新增,295 行。合成 fixture(假 V1/V2 magic + mock decrypt_dat_file),不依赖真实加密素材。 |
||
|
Belugary
|
a6cb3d0497 |
feat: 解析微信引用回复消息 (appmsg type=57) + 新增 decode_refer MCP 工具
> 高价值改动 rationale (override 路径) > > 引用回复 (appmsg type=57) 是聊天里第 3 高频的消息类型 (仅次于纯文本和 > 图片)。当前 _format_app_message_text 的 type=57 分支直接把 refermsg/ > content 按 [:160] 截断当摘要,对内层 type=3 (图片) / 34 (语音) / > 43 (视频) / 47 (动画表情) / 49 (嵌套卡片) 这些"二进制"被引用消息, > 会把 cdnurl / aeskey / md5 / cdnthumb / voiceurl / externurl 一坨乱码 > 渲染到 LLM 可见的 chat history,严重污染上下文。issue #44 #45 重复反馈 > 一个月无人接 —— 这是个明确的用户痛点,fork 实测覆盖 5 种内层 type 的真 > 实数据,渲染长度从原本几千字降到 21-58 字。改动较大但 review 风险低: > 替换的就是 19 行 inline 截断逻辑,新加的 helpers / decode_refer 都是 > 纯加,不动现有 API。 \`_format_app_message_text\` 当前 type=57 分支用 19 行 inline 逻辑直接 \`refer.findtext('content')[:160]\` 当摘要。这对 type=1 (文本) 工作正常, 但对其他内层 type 是个隐藏的 bug: - type=3 图片: 渲染 \`<msg><img cdnthumburl="…" aeskey="…" md5="…" cdnurl="…" />\` 截断 - type=34 语音: 渲染 \`<voicemsg voicelength="…" voiceurl="…" />\` 截断 - type=43 视频: 渲染 \`<videomsg cdnvideourl="…" cdnthumburl="…" />\` 截断 - type=47 动画表情: 渲染 \`<emoji md5="…" externurl="…" />\` 截断 - type=49 嵌套卡片: 渲染外层 escape 后的 XML 字符串截断 后果: cdnurl / aeskey / md5 / voiceurl / externurl 等二进制元数据泄漏到 LLM 可见的聊天历史,污染上下文且无信息量。引用回复是 type=57 是高频消息,影响面大。 按 refer_type 分发 schema-aware 摘要: 1. **新增三组 helpers (mcp_server.py +135 行,纯加)**: - \`_REFER_INNER_TYPE_LABEL\`: 内层 type → 中文标签 (1 文本 / 3 图片 / 34 语音 / ...) - \`_INNER_APPMSG_TYPE_LABEL\`: refer_type=49 时嵌套 appmsg/type → 标签 (5 链接 / 6 文件 / 19 聊天记录 / ...) - \`_extract_refer_info(appmsg)\`: 提取 refermsg 全字段返回 dict - \`_summarize_refer_content(refer_type, content)\`: 按 type 分支 - type=1: 取原文,截断到 max_len - type=3/34/43/47/...: 给标签,**不**展开 cdnurl/aeskey/md5 - type=49: 走 \`_parse_xml_root\` (经 \`_XML_UNSAFE_RE\` 过滤 DOCTYPE/ENTITY 防 XXE) 解一层 inner appmsg, 给 \`[链接] xxx\` - 未识别 type: 给 \`[type=N]\` 兜底 - \`_format_refer_message_text(appmsg, ...)\`: 渲染两行格式 \`<回复正文>\n ↳ 回复 <对方>: <摘要>\` 2. **\`_format_app_message_text\` 的 type=57 分支简化**: 19 行 inline → 3 行 dispatch 到 helper。 3. **新增 MCP 工具 \`decode_refer(chat_name, local_id, create_time=0)\`**: 输出结构化多行文本 (回复正文 / 被引用发送者 / 类型 / 摘要 / svrid / createtime), 错误文案分别指引 \`decode_file_message\` (type=6) / \`decode_record_item\` (type=19) / \`decode_transfer\` (type=2000), 不让用户在 4 个工具间盲猜。 新文件 \`tests/test_refer_message.py\`, 20 个新测试: - \`ReferInnerTypeLabelTests\` (2): 标签映射 spot-check - \`ExtractReferInfoTests\` (2): 全字段提取 / refermsg 缺失返回 None - \`SummarizeReferContentTests\` (11): 5 种 refer_type 标签 / type=1 文本截断 / type=49 嵌套链接卡 / type=49 聊天记录卡 / type=49 invalid XML 退化 / unknown type 兜底 / 空 content / XXE payload 拒绝 - \`FormatReferMessageTextTests\` (4): 1v1 文本引用渲染 / 图片引用不泄漏 PII (cdnurl/aeskey/md5) / refermsg 缺失退回 title / 空 reply 用占位符 - \`AppMessageDispatchReferTests\` (1): dispatcher 走新 helper 不走旧截断 合成 fixture (wxid_synth_a/b, 12345@chatroom, Sender A/B, svrid 1+0\*18), 无真实 PII。 基线 183 → 203 通过 (+20 新增), 0 回归。 - \`mcp_server.py\`: 替换 19 行 type=57 inline → 3 行 dispatch (净 -16 行); 新增 6 个 helpers + 1 个 MCP 工具 \`decode_refer\` (+275 行); 不改任何现有公开 API。 - \`tests/test_refer_message.py\`: 新增 (20 测试, 合成 fixture, 不依赖真实加密素材)。 - **本 PR 不包含 fork 里的 CLI 入口 (\`wxdec.cli.decode_refer\`) 和 \`export_chat\` / \`monitor_web\` 的对应改动** —— 那几处依赖 fork 私有的包结构 (\`wxdec/cli/\`), 不属于上游 scope。后续如有需要可单独提。 issue #44 #45 (引用回复渲染乱码) |
||
|
Belugary
|
5bc275b81c |
feat(mcp): get_chat_images/get_voice_messages 加 offset/time_range
\`get_chat_images\` 和 \`get_voice_messages\` 仅有 \`limit\`, 接口与 \`get_chat_history\` / \`search_messages\` (\`offset\` + \`start_time\` + \`end_time\`) 不对齐: 1. 查不了"某段时间内的图片/语音" 2. 不支持分页, 单次取 \`limit=1000\` 一次性拉 3. LLM 用同样模式调不同工具时签名不一致, 容易出错 两个工具各加 3 个可选参数: - \`offset: int = 0\` - \`start_time: str = ""\` - \`end_time: str = ""\` 复用上游已有的 \`_validate_pagination\` + \`_parse_time_range\` helpers。 - 输入校验失败立即报错 (offset 负数 / 时间格式错 / start > end) - 每 shard 拉 \`limit + offset\` 张候选, 合并后全局 \`create_time DESC\` 排序, 切片 \`[offset : offset + limit]\` 出本页 - 单 shard 凑得起本页, 避免某 shard 缺数据时本页变短 - header 显示 offset/limit 和时间范围 (传了才显示) - 加 \`start_ts=None\` / \`end_ts=None\` 参数 - SQL 动态拼 \`create_time >= ?\` / \`<= ?\` clause - 不传时序参数完全等价旧行为 (向后兼容) - 同样 3 个参数 + \`_validate_pagination\` + \`_parse_time_range\` - VoiceInfo 表 SQL 动态拼 \`chat_name_id = ? AND create_time ?...\` - 多 shard 各取 \`limit + offset\` 后合并切片 新增 \`tests/test_chat_images_query_align.py\` 8 个 case: - offset 负数报错 - start > end 报错 - candidate_limit = limit + offset (shard 调用确认) - 时间参数正确解析为 unix 秒并透传 - offset=2, limit=2 切到全局排序后第 3-4 张 - header 包含时间范围 - header 包含 offset/limit - 默认调用(不传新参)行为与旧接口一致 修改 \`tests/test_get_chat_images_multishard.py\` 的 fake_list 签名: - 旧: \`(db_path, table_name, username, lim)\` 位置参 - 新: \`(db_path, table_name, username, limit=20, start_ts=None, end_ts=None)\` - 既支持旧调用模式 (kwargs), 也兼容新签名 全量 \`pytest tests/\` 208/208 通过。 3 个可选参数全部带默认值 → 既有调用方零修改。 shard candidate=\`limit+offset\` 的成本: 大 offset 时单 shard 请求量 增大。但 image/voice 表每 chat 单 shard 一般 < 10K 条, 实际 cost 可 忽略。如果将来要做"翻 100 页"级深翻, 可以加 keyset pagination, 现 在 offset 模式与 \`get_chat_history\` 一致即可。 与 #103 / #104 触碰同一文件, 合并顺序无所谓 — 后合的 rebase 即可。 |
||
|
Belugary
|
6606122c86 |
feat(mcp): get_chat_history 加 msg_types 按类型过滤
LLM 用 \`get_chat_history\` 查"和 X 的所有图片消息"时, 只能拉 50 条 混合消息再客户端过滤 —— 大部分 token 浪费在不需要的文本上。同样 "只看转账记录" / "只看语音" 的场景, 没有原生过滤手段。 \`get_chat_history\` 加一个可选 kwarg \`msg_types: list[str] | None = None\`: - 接受 \`['text', 'image', 'voice', 'video', 'file', 'emoji', 'location', 'namecard', 'voip', 'system']\` 子集 - \`'file'\` 是 alias → 'app' (WeChat 把文件归到 \`local_type=49\`, 俗称 file) - 输入大小写不敏感, 自动 strip - 未知类型立即报错并列出可选值 (不偷偷过滤合法部分) - None 或 \`[]\` 表示不过滤, 完全等价于旧行为 (向后兼容) 实现上拆 3 件: 1. \`_MSG_TYPE_MAP\` 常量 (字符串 → \`local_type\` 整数列表) 2. \`_resolve_msg_types()\` helper 做输入校验 + 翻译 3. \`_build_message_filters\` / \`_query_messages\` / \`_collect_chat_history_lines\` 链路加 \`type_filter=None\` 透传, SQL 注入 \`local_type IN (?,?,...)\` clause \`tests/test_msg_types_filter.py\` 12 个 case: - None / 空 → 不过滤 - 单类型 / 多类型解析 - \`file\` alias → app - 大小写 + strip 不敏感 - 未知类型报错且不放过合法的 - SQL 生成: 无过滤时 clauses 不含 \`local_type\`, 单类型生成 \`IN (?)\`, 多类型生成 \`IN (?,?,?)\` - 与 time / keyword 组合时 param 顺序正确 全量 \`pytest tests/\` 212/212 通过。 新参数默认 None, **既有调用方零修改**。 类型映射表 (\`_MSG_TYPE_MAP\`) 命名是有立场的判断 (比如 \`'app'\` 这一 桶实际混了文件 / 分享卡 / 小程序 / 转账 / 引用回复), 如果维护者 不同意具体 label 或想拆细, 改 dict 就行, 不影响接口。 与 #103 (\`_pagination_hint\`) 触碰同一文件 \`mcp_server.py\`, 后合的 rebase 即可, 无逻辑冲突。 |
||
|
Belugary
|
9450e46ca5 |
feat(mcp): 加 _pagination_hint 帮 LLM 决定是否续翻 (#103)
## 问题 LLM 调用 \`get_chat_history(limit=50)\` 拿到 50 条消息后, 无法判断 "是真只有 50 条" 还是 "还有 150 条没拿"。LLM 缺少续翻信号, 容易 基于不完整数据回答。 类似问题影响所有分页工具: \`search_messages\` / \`get_chat_images\` / \`get_voice_messages\` / \`get_contacts\`。 ## 修复 加 \`_pagination_hint(count, limit, offset)\` helper: - \`count >= limit\` 时返回 \`(可能还有更多结果，可设 offset=N 继续查询)\` - \`count < limit\` 时返回空 (表示已读完当前条件全部结果) - \`limit == 0\` (理论非法, 上游有 \`_validate_pagination\` 兜底) 防御 性返回空 应用到 5 个工具返回字符串末尾: - \`get_chat_history\` (1 处) - \`search_messages\` 三个内部分发 \`_search_single_chat\` / \`_search_multiple_chats\` / \`_search_all_messages\` (3 处) - \`get_chat_images\` / \`get_voice_messages\` (各 1 处, 当前两者无 offset 参数, 使用 \`offset=0\` 占位; 后续接口对齐 PR 会把 \`offset\` 加进来) - \`get_contacts\` 单独用 \`total > limit\` 模式提示 "共 N 个匹配, 当前 仅显示前 limit 个, 可增大 limit" — 因为 \`get_contacts\` 当前无 pagination 语义, 仅有 limit, 文案语义不同 ## 测试 \`tests/test_pagination_hint.py\` 5 个 case 覆盖: - count < limit 不提示 - count == limit 提示且 offset 累加正确 - 连续翻页 offset 推进 (offset=100, limit=20 → 提示 offset=120) - limit=0 防御 - count > limit 边界 (理论不该发生) 全量 205/205 通过。 ## 范围 纯返回字符串末尾追加, 不改任何查询逻辑、不改函数签名、不改数据库 读路径。零破坏性, 调用方 100% 向后兼容。 提示文案如不合适可直接改, 不影响行为。 |
||
|
joshua-deng
|
70d44ef61f |
fix(export): strip group prefix before parsing appmsg in chat export (#101)
Issue #88: 群聊里的引用回复（appmsg type=57）/ 卡片 / 视频在 export_chat 和 export_all_chats 渲染成 type=link_or_file 且 content 为空。 根因：`_extract_content` 把数据库里带 `wxid_xxx:\n` 群前缀的原始 content 直接喂给 `_format_app_message_text`，XML 解析器在前缀文本上 ParseError， 返回 None。 修复： - 用 `chat_username.endswith('@chatroom')` 判定群聊 - 在 dispatch 前调 `mcp_server._parse_message_content(..., is_group=True)` 剥前缀；逻辑也对群里的 base=1 text 生效（之前同样带前缀） - 把 `is_group=True` 透传给 `_format_app_message_text`，让引用回复走 group 分支的发送者标签解析 - 用 `mcp_server.get_contact_names()` 代替之前硬编码的 `{}`，让 wxid 能 正确解出昵称 测试：新增 5 个测试覆盖群引用回复带前缀 / 1-on-1 不受影响 / 群 text 前缀剥离 / 1-on-1 text 不变 / names dict 正确解析。126/126 通过。 Belugary 在 #100 修了 `_format_app_message_text` 内部的 type=57 schema 渲染（对 get_chat_history 生效），本 PR 是补 export 这条路径上的群前缀 bug。两者互补。 Co-authored-by: ylytdeng <ylytdeng@users.noreply.github.com> |
||
|
Belugary
|
187d820bb0 |
feat(mcp): render voice messages with duration in chat history (#97)
## Problem
Voice messages in `_format_message_text` previously rendered as a bare
`[语音] (local_id=N, ts=T)` because msg_type=34 fell through to the generic
non-text branch with no schema-aware summarizer. LLMs reading chat history
had no way to judge whether a voice clip was worth calling `decode_voice`
on without first inspecting it.
## Fix
New helper `_format_voice_text(content)` parses the embedded
`<voicemsg voicelength="…">` and renders `[语音 N.Ns]` (duration to one
decimal, milliseconds → seconds). Type=34 dispatches through it, then
appends the existing `_id_suffix()` so the local_id annotation is
preserved end-to-end:
[语音 3.3s] (local_id=72481, ts=1700000000)
Falls back to `[语音]` (still with `_id_suffix()`) when content is empty,
`<voicemsg>` is absent, XML parse fails, or `voicelength` is missing /
zero / non-numeric.
XML parsing routes through the existing `_parse_xml_root` so the
`_XML_UNSAFE_RE` DOCTYPE/ENTITY filter and 200KB size cap are reused —
no new XXE surface.
## Tests
12 new cases in `tests/test_voice_format.py`: happy path, subsecond,
multi-second, missing / zero / non-numeric voicelength, empty / None
content, missing `<voicemsg>` tag, malformed XML, XXE payload, and two
end-to-end cases through `_format_message_text` (with and without
voicelength) to pin the full rendered output including `_id_suffix()`.
Baseline 183 → 195 passing, 0 regressions.
## Scope
- `mcp_server.py`: adds `_format_voice_text` helper and one branch in
`_format_message_text` (base_type == 34). No public surface change —
this only affects formatting of messages that previously rendered as
the bare `[语音]` fallback.
- `tests/test_voice_format.py`: new file, synthetic fixtures only (no
real PII).
|
||
|
Belugary
|
f03df51561 |
feat: parse WeChat transfer messages (appmsg type=2000) (#85)
Add structured parsing for transfer messages so they no longer fall through to the generic `[链接/文件]` fallback in chat history exports. Mirrors the dispatch + helper pattern PR #65 (merged-forward type=19) established for `base_type=49` appmsg sub-types. ## What is added **Helpers (mcp_server.py):** - `_TRANSFER_PAYSUBTYPE_LABEL` — maps the 6 community-consensus paysubtypes (1 发起 / 3 已收款 / 4 已退还 / 5 过期已退还 / 7 待领取 / 8 已领取); unknown values degrade to `未知(paysubtype=N)` so a new variant in a future WeChat build is visible rather than silently dropped. - `_extract_transfer_info(appmsg)` — pulls fields out of `<wcpayinfo>`, with snake/camelCase fallback (`feedesc`/`feeDesc`, `pay_memo`/`paymemo`) observed across WeChat versions. - `_format_transfer_message_text(appmsg, title)` — one-line render for chat history: `[转账·已收款] ¥100.00 备注: lunch`. **Dispatch (mcp_server.py):** - `_format_app_message_text` gains an `app_type == 2000` branch that routes to `_format_transfer_message_text`. `get_chat_history`, `export_chat`, `export_all_chats` and `monitor_web` all inherit automatically. **New MCP tool (mcp_server.py):** - `decode_transfer(chat_name, local_id, create_time=0)` — full structured view: direction, amount, memo, payer/receiver wxid, transfer id, transcation id, begin/invalid timestamps. Uses the same multi-shard scan + ambiguity-by-create_time pattern as `decode_file_message` / `decode_record_item`. **CLI wrapper:** - `decode_transfer.py` at the repo root — argparse wrapper that prints the same text as the MCP tool, returning non-zero exit when the message can't be decoded (script-friendly). **JSON export (chat_export_helpers.py + export_chat.py + export_all_chats.py):** - `_extract_content` now returns `(rendered, extras)`. `extras` carries structured fields when a message type has more signal than the human-readable string (currently: transfers → `type:"transfer" + transfer:{direction, fee_desc, pay_memo, ...}`). The channel is forward-compatible — future additions (video号 metadata, expanded merged-forward, etc.) flow through the same shape without changing the caller signature. JSON consumers that only read `content` are unaffected; the change is additive. **monitor_web (monitor_web.py):** - Backend dispatch branch + orange-yellow `.msg-transfer` card CSS + `renderRich` JS handler. ## Tests 12 new cases in `tests/test_record_decoders.py`: - `TransferPaysubTypeLabelTests` — locks the 6-value label table. - `ExtractTransferInfoTests` (6 cases) — full field round-trip, missing `<wcpayinfo>` fallback, snake/camelCase variants, unknown paysubtype degradation, empty paysubtype handling. - `FormatTransferMessageTextTests` (4 cases) — initiate / received-with-memo / missing-wcpayinfo / missing-fee-desc. - `AppMessageDispatchTransferTests` — `_format_app_message_text` routes type=2000 correctly so `get_chat_history` / `export_chat` both pick it up. All fixtures use synthetic placeholder values (`wxid_payer_synth`, `¥100.00`, `1` + 27×`0`); no real PII or transaction IDs. ## Scope 7 files, +546 / -15 (additions only — no behavior change for existing message types). All 180 tests pass locally (168 baseline + 12 new). |
||
|
Belugary
|
b2affdcf88 |
fix(image): scope local_id lookup by chat_id + use real column name (#82)
* fix(image): scope local_id lookup by chat_id + use real column name `ImageResolver.get_image_md5` made two wrong assumptions about the production `MessageResourceInfo` schema, which made the decode_image MCP tool always fail with "无法找到 local_id=X 的图片信息": 1. The column is `message_local_id`, not `local_id`. The current query throws `sqlite3.OperationalError: no such column: local_id`, but the exception is swallowed by `except Exception: pass`, masking the real failure as a silent miss. 2. `message_local_id` is not globally unique. In production it repeats across chats, and within an active chat the same local_id can recur up to 7 times (observed on a real DB). The production schema scopes by `chat_id`, resolved from `ChatName2Id.rowid WHERE user_name = ?`. Fix: `get_image_md5` now takes `(username, local_id)`: - Resolve `username -> chat_id` via `ChatName2Id`. - Query `MessageResourceInfo` filtered by `chat_id + message_local_id + message_local_type == 3` (image type; high bits are session flags, so use `% 2^32`), ordered by `message_create_time DESC LIMIT 1`. External callers (`mcp_server.decode_image_tool` / `list_chat_images_tool`) already pass `username` through `ImageResolver.decode_image()` / `list_chat_images()`, so the public API is unchanged. Only the internal helper signature shifts. The existing test fixture in `test_decode_image_v2` used the same wrong schema as the buggy code (`CREATE TABLE MessageResourceInfo (local_id INTEGER PRIMARY KEY, packed_info BLOB)`), so the tests passed against a self-consistent fiction. The fixture is rebuilt to match real columns plus `ChatName2Id`, and three regression tests are added: - cross-chat collision (same local_id in 3 chats; must pick the right one and not the type=43 video row) - same-chat reuse (same local_id, two timestamps; must pick the newer) - unknown chat (username not in ChatName2Id; structured error, no crash) All 154 tests pass locally (151 baseline + 3 new). * fix(image): surface get_image_md5 errors and use read-only DB open Two follow-ups on top of the chat_id scoping fix: 1. The bare `except Exception: pass` was the original failure mode: it silently swallowed `OperationalError: no such column: local_id` when the production schema diverged from the old `local_id` column name, masking the bug this PR fixes. Print the exception to stderr so future schema drift surfaces immediately instead of returning a misleading "image not found" error. 2. Open message_resource.db with `file:...?mode=ro` URI to match the rest of the project (monitor_web.py uses this idiom in 9 places). The DB is read-only for our purposes and a running WeChat may still hold it; using URI ro avoids any chance of lock contention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Belugary
|
cd329afa1b |
fix(mcp): scan all message DB shards in get_chat_images (#84)
WeChat rolls a chat's messages over to the next `message_N.db` shard once the current shard fills up (~100 MB), so any chat older than the current shard window has its history split across multiple shards. The other message-query tools — `get_chat_history`, `search_messages`, and `decode_image` — already iterate every matching shard via the plural helper `_find_msg_tables_for_user`. Only `get_chat_images` still used the singular `_find_msg_table_for_user`, which returns the first shard that contains the user's table. Effect: every image that lived in a non-first shard was silently dropped from `get_chat_images`. On a long-lived chat with many images, the tool would return only the most recent slice and pretend the rest did not exist. Fix: switch `get_chat_images` to `_find_msg_tables_for_user`, fetch `limit` images per shard, merge, sort by `create_time` DESC, and slice to `limit`. This mirrors how the other tools fan out across shards. Tests in `tests/test_get_chat_images_multishard.py`: - `test_collects_images_from_every_shard` — both shards' images appear in the output (the regression case) - `test_global_sort_by_create_time_desc` — newer image from an older shard still wins, output is globally sorted (not per-shard concat) - `test_limit_truncates_globally_across_shards` — limit=3 takes the 3 newest overall, not "first shard wins" - `test_no_shards_returns_not_found` — empty shard list path - `test_all_shards_empty_returns_no_images` — every shard empty path All 156 tests pass locally (151 baseline + 5 new). Public tool signature is unchanged; only the internal scanning loop is widened. |
||
|
Belugary
|
c162a9b92f |
fix(mcp): trim raw XML payload from namecard (type=42) chat output (#83)
When a chat history contains a name-card message (msg_type=42), the
dispatcher in `_format_message_text` had no case for `base_type == 42`,
so it fell through to the generic non-text branch:
elif base_type != 1:
type_label = format_msg_type(local_type)
text = f"[{type_label}] {text}" if text else f"[{type_label}]"
`text` for type=42 is the full raw `<msg ...>` element, so chat history
exports emitted `[名片] <msg username="..." antispamticket="v2_..."
brandIconUrl="https://wx.qlogo.cn/..." bigheadimgurl="..." ... />`.
That payload has two problems:
1. It leaks anti-spam tokens (`antispamticket`) and head-image CDN URLs
into chat logs that are routinely piped to LLMs and other downstream
tools.
2. The raw XML drowns out the actual signal — a human or an LLM reading
the chat just wants to know "X shared Y's contact".
This PR adds `_format_namecard_text(content)` that pulls only the three
useful attributes:
- `nickname` — display name
- `username` — wxid (annotated as "公众号" when prefixed `gh_`)
- `certinfo` — user-authored bio
and wires it into the dispatch chain via a new `elif base_type == 42:`
branch, sitting alongside the existing `49` (app message) handler. It
reuses `_parse_xml_root` and `_collapse_text` — no new helpers
introduced.
Tests: 7 cases in `tests/test_namecard_format.py` covering the realistic
shape (with antispamticket / brand URLs that must NOT appear in output),
official accounts (`gh_*`), missing certinfo, missing nickname, missing
both identifiers, and broken-XML fallthrough.
All 158 tests pass locally (151 baseline + 7 new).
|
||
|
Belugary
|
216f44a99f |
fix(image): reject corrupted V2 image when AES or XOR key is wrong (#81)
`v2_decrypt_file` previously wrote files to disk even when the keys were
wrong, producing garbage output with no way for the caller to detect the
failure:
1. Wrong AES key -> `detect_image_format` returns 'bin' (magic does not
match any known format) -> a `.bin` file of random bytes was written.
2. Wrong XOR key with correct AES key -> file header looks like a valid
jpg/png (the AES segment decrypts correctly) but the trailing XOR
segment is scrambled -> callers get a half-valid image file that
image viewers either render as truncated or fail to open.
Both cases now return `(None, None)`:
- `fmt == 'bin'` -> fail fast, no file written.
- `xor_size >= 2` -> validate trailer magic by format:
* jpg must end with FF D9 (EOI marker)
* png must contain IEND chunk in the last 12 bytes
Other formats (gif/bmp/tif/webp/hevc/wxgf) lack a mandatory trailer
signature, so they skip the check to avoid false rejection.
Also fixes a latent bug in `test_decode_image_v1_no_aes_key_uses_fixed_key`:
the test built the synthetic .dat with `TEST_XOR_KEY=0x37` but constructed
`ImageResolver` without `xor_key=`, defaulting to `0x88`. The XOR segment
was always scrambled — the test passed because the AES segment alone was
enough for `detect_image_format` to return 'png' from the header, and no
trailer validation existed to catch the corruption. The new trailer check
surfaces this, so the test now passes `xor_key=TEST_XOR_KEY` explicitly.
Tests: 5 new cases (wrong AES key / wrong XOR for jpg / wrong XOR for
png / xor_size=0 bypass / wxgf bypass). All 156 existing tests still pass.
|
||
|
jiangbowen
|
4be1ac4713 |
feat: 解析合并转发的聊天记录消息（appmsg type=19）+ 文件本地路径查找工具 (#65)
* feat: 解析合并转发的聊天记录消息（appmsg type=19）+ 新增文件路径查找工具
mcp_server.py:
- _format_app_message_text 增加 app_type=19 分支，解析 <recorditem> 内嵌
XML，把"[链接/文件] xxx的聊天记录"展开成多行 datalist 内容（含发送者/
时间/数据类型）。覆盖 datatype 1/2/3/4/5/6/7/8/17/19/22/23/29/36/37
共 14 种类型；超过 50 条自动截断；空 datalist fallback 到"（待加载）"
- 新增 decode_file_message 工具：从 type=49+sub=6 消息找本地副本路径
（~/Library/.../msg/file/{YYYY-MM}/原文件名.{ext}），返回精确路径
+ size 二次确认，处理同名 (1)(2) 后缀
- 新增 decode_record_item 工具：从 type=49+sub=19 合并记录的第 N 个
dataitem 找本地副本（msg/attach/{table_hash}/*/Rec/*/F/{idx}/{name}），
未下载时给精确"在 wechat 点击哪一项"指引
回归测试：在数千条真实合并转发消息上 ~83% 完美展开 datalist 内容，
剩余的 content 缺失/消息被撤回情况下行为与改前一致（fallback 到原 [链接/文件]）。
* fix: hoist subdir_map in decode_record_item to avoid UnboundLocalError
When the chat's attach directory does not exist (no merged-record
attachment ever downloaded for that chat), the if-block defining
`subdir_map` was skipped, so the not-found branch's reference to
`subdir_map.get(datatype, '?')` raised UnboundLocalError instead of
returning the intended guidance message.
Hoist the dict definition above the if-block so both branches can
safely reference it.
Caught by Codex review on PR #65.
* fix: address Codex P2 — large recorditem XML + glob escape
Two issues caught by Codex review on PR #65:
P2-1: _parse_xml_root rejects payloads >20KB, which silently dropped
~330 large merged-record cards (max observed 418KB with 99 dataitems)
back to "[链接/文件]" fallback. Add a dedicated _parse_record_xml with
a 500KB limit for embedded recorditem XML; routes both call sites in
_format_record_message_text and decode_record_item to it. Boosts
overall parse coverage from 83% to ~87% on real-world data.
P2-2: decode_record_item passed datatitle directly to glob, so file
names containing [ ] * ? would be treated as glob patterns rather
than literals — leading to wrong candidates or missed real files. Wrap
datatitle with glob.escape() before the exact-match query.
Full unit-test suite (35 tests) still passes.
* perf+style: speed up decode_file_message + minor consistency fixes
Self-review findings on top of the Codex P1+P2 fixes:
- perf: decode_file_message previously os.walk-ed `msg/file/` and
`msg/attach/` from scratch on every call, scanning ~185k files /
17GB on a real-world install (~6.3s per call). Now first reads
`create_time` from the message and globs only the matching
`msg/file/{YYYY-MM}/` (plus ±1 month for cross-month edge cases),
with the original walk preserved as a fallback. Measured speed-up
~10x on first call, ~750x on warm cache.
- decode_record_item: extend `type_label` to cover datatype 23
(视频号直播) and 36 (小程序/H5) so the not-found message matches
the labels emitted by _format_record_dataitem instead of falling
back to a raw `datatype=23` string.
- decode_record_item: replace unused `sub_type_packed` with `_` to
silence the "name assigned but unused" smell.
- _parse_record_xml: comment now states the empirically observed
~418KB upper bound (was "~50KB"), making the 500KB ceiling
obviously sufficient.
All 35 existing tests still pass.
* fix: address Codex round-3 P2 — multi-shard lookup + size-validate month scan
Two more issues caught by Codex review on PR #65 that I missed during
self-review:
P2-3 (multi-shard local_id lookup): both `decode_file_message` and
`decode_record_item` were resolving a single message-table via the
singular `_find_msg_table_for_user`, but a chat's messages can span
multiple message_N.db shards (search_messages and history-iteration
already use `_find_msg_tables_for_user`). When the requested local_id
lived in a different shard the tools incorrectly returned "找不到
local_id" or — if IDs collide across shards — picked the wrong row.
Now both tools iterate all shards and stop at the first hit; the
not-found message reports how many shards were scanned.
P2-4 (size validation in month-scan fast path): the perf-fix in the
prior commit collected `msg/file/{YYYY-MM}/` matches without verifying
size, so when a same-named-but-different-size copy existed in the
target month the candidate list was non-empty, the walk-fallback was
skipped, and the later `size_match` filter could end up empty —
returning a wrong-size file. Now the month-scan filters by `totallen`
upfront when known, so unmatched candidates don't poison the fallback.
Same one-shot size validation applied to `decode_record_item`'s
exact-name glob branch for symmetry.
These were both "I should have caught" issues — Codex did the
cross-tool consistency check (singular vs plural shard helper) that I
skipped, and stress-tested an edge case (month-scan finds same-name
wrong-size) that I didn't think through when writing the perf fix.
35/35 existing tests still pass. Real-data smoke: decode_file_message
0.96s end-to-end (multi-shard scan + size validation),
decode_record_item 0.03s.
* refactor: reuse _parse_message_content helper for group prefix stripping
Self-review found that decode_file_message and decode_record_item
hand-rolled their own heuristic for stripping group-chat sender
prefixes ("wxid_xxx:\n<xml...>") via a string-startswith check, while
the rest of the project already uses the canonical
`_parse_message_content(content, local_type, is_group)` helper for
exactly this purpose.
Wired both tools to that helper, deriving is_group from the username
suffix `@chatroom`. Existing edge cases (private chat content with
literal "<...>", group content with "wxid_xxx:\n", etc.) still pass.
35/35 tests still pass; 6/6 edge-case smokes still pass.
* fix: address Codex adversarial-review high+medium findings
Adversarial review caught four issues that the surface-level passes
missed. All four are now fixed end-to-end (validated against real
data, not just helper-level smoke):
[high] Large recorditem outer XML actually parsed:
Previous P2 fix added _parse_record_xml(500KB) for the inner CDATA
but the outer appmsg was still gated by _parse_xml_root(20KB), so
any merged-record card whose outer XML exceeded 20KB silently fell
back to "[链接/文件]" and never reached the inner expansion. Now
_parse_xml_root accepts a max_len kwarg, _format_app_message_text
retries with _RECORD_XML_PARSE_MAX_LEN when the default cap rejects
the outer XML, and _format_record_message_text passes the wider cap
for inner parses. Real-data check: a 34KB outer / 67-dataitem card
now expands fully via the get_chat_history → _format_message_text
→ _format_app_message_text → _format_record_message_text chain.
[high] Multi-shard local_id ambiguity:
decode_file_message and decode_record_item previously broke on the
first shard match. Empirically confirmed local_id 171 in the test
account exists in TWO shards as TWO different messages (one type=1
text, one type=6 file at different create_times). Now both tools
scan all shards, fail with an explicit ambiguity error when more
than one row matches, and accept an optional create_time arg from
the user to disambiguate uniquely.
[medium] History output now exposes (local_id, ts) for file and
record cards, and record dataitem rows are prefixed with their
0-based [item_index]. Without these, callers had no way to feed
decode_file_message / decode_record_item a stable identifier.
[medium] decode_file_message now requires appmsg type=6 and an
appattach node, refusing to search the local cache by title/size for
unrelated app messages (links, miniapps, record cards) that happen
to share a title with a real file.
35/35 existing tests still pass. Real-data smokes:
- 34KB outer XML / 67 dataitems expanded end-to-end
- multi-shard ambiguity correctly raised + resolved by ts kwarg
- history output now contains "(local_id=N, ts=T)" suffixes and
"[N]" dataitem prefixes
* fix: address Codex adversarial round-2 high findings
Round-2 adversarial review caught two issues my self-review missed
again. Both are now fixed end-to-end:
[high] decode_record_item also rejects large outer XML (mcp_server.py:2124-2126)
Round-1 high #1 was fixed by adding a wider-limit retry inside
_format_app_message_text, but decode_record_item itself still
parsed the outer appmsg with `_parse_xml_root(xml_text)` at the
default 20KB cap. Same root cause: I patched one caller, missed
the other — exactly the kind of cross-tool inconsistency that
cost two rounds already.
Extracted a shared `_parse_app_message_outer(content)` helper that
encapsulates the "try default cap, fall back to wider limit when
default rejects" pattern. Now used by all three call sites:
_format_app_message_text, decode_file_message, decode_record_item.
Real-data check: a 34KB outer (67 dataitems) parses through every
caller path, not just history rendering.
[high] Record attachment lookup silently picks wrong cached file
Previous lookup had three fallback tiers (filename+size → size only
→ cross-subdir size only) and on multiple matches sorted by mtime
and took newest. Two failure modes:
1. Different forwarded-record cards in the same chat may produce
paths with identical (filename, item_index, datasize), and the
mtime tiebreak lets the tool return another record's file while
reporting "找到本地文件: ✅".
2. Cross-subdir size-only fallback can match files belonging to
unrelated dataitem types entirely.
Now fail-closed:
- Strict filename + size match only when datatitle is known.
- Size-only fallback now ONLY when datatitle is missing
(e.g. datatype=2 thumbnails) AND scoped to the same sub-dir +
item_index — no more cross-Rec leakage.
- Removed the cross-subdir terminal fallback entirely.
- Multiple candidates after strict matching → ambiguity error
listing all candidates with mtime, no silent pick.
35/35 existing tests still pass. Real-data smokes:
- 大 outer 34KB 卡片 _parse_app_message_outer 解析成功
- decode_record_item(local_id, ts) 正确命中 Lec 4 PDF
- 多分片冲突 + 不传 ts → 报歧义错误并提示加 create_time
- 未下载 dataitem → 精确指引"在 wechat 点第 N 项"
* fix: address Codex round-3 adversarial high+medium findings
Round-3 caught two more cross-tool inconsistency issues, both in the
same family I keep missing (修一处忘另一处):
[high] decode_file_message also needs to fail-closed on ambiguity
Round-2 high #2 forced decode_record_item to fail-closed when
multiple cached candidates remain after strict matching, but I
forgot to apply the same change to decode_file_message — it still
silently sorted by mtime and returned candidates[0]. Same root
cause as round-1 high #1: Codex catches what I miss when the same
pattern needs fixing in two places.
Now decode_file_message: strict size filter when totallen is known,
and ambiguity error (not mtime sort) when more than one candidate
remains. Behavioral change: previously returned 逻辑审计论文(1).pdf
on a real test case; now reports both candidates and asks user to
disambiguate. UX regression but safety-correct.
[medium] decode_record_item rejects non-downloadable datatypes
upfront. Previously, dataitems with unknown datatype fell through
to a wildcard `sub='*'` glob over all attach subdirs (F/Img/V/A),
which could match unrelated files for links/locations/cards/
miniapps/nested-record dataitems that have only metadata, no
binary payload. Now reject non-{2,4,5,8} datatypes with a clear
"no local binary, look at history output instead" message before
any filesystem lookup.
35/35 existing tests still pass.
* fix: address Codex adversarial round-4 high findings
Round-4 found three security/correctness issues. All addressed:
[high] Path traversal via untrusted XML titles
title (decode_file_message) and datatitle (decode_record_item) come
from message XML — attacker-controlled in the "malicious chat
partner" threat model. glob.escape does NOT strip path separators
or normalize absolute paths, so e.g. title="/etc/passwd" makes
os.path.join(month_dir, "/etc/passwd") == "/etc/passwd" (POSIX
rule: join drops left when right is absolute), and glob then walks
outside msg/file. If size also matches, the tool returns an
arbitrary system path as a "found wechat file".
Added _safe_basename(name) helper with strict-reject semantics
(per Codex: reject, don't normalize) — any name containing path
separators, .. components, NUL, or absolute-path prefix is
rejected outright. Both decoders sanitize their XML-derived names
before any filesystem operation. Added _path_under_root realpath
check after candidate selection as a second-line defense against
symlink escapes.
[high] decode_file_message and decode_record_item can return cached
files belonging to a DIFFERENT message even when len(candidates)==1
Both tools rely on (filename + size + optional item_index)
heuristic matching against the cache — they have no way to derive
a record-bound or message-bound path from wechat metadata, so
exactly one matching cached file from an unrelated message looks
identical to a correct hit. This is a design limitation: wechat
does not expose record_hash or attach-uuid in the message XML in
any form derivable from outside the client.
Acknowledged in tool output with an explicit ⚠️ "this path is
heuristic, please verify mtime/context/content" warning attached
to every "found local file" response. The match itself is still
the same heuristic — closing this fully would require either
removing the tools or reverse-engineering wechat's path hashing.
Documented the limitation in the warning so callers can manually
verify before trusting downstream Read/PDF results.
35/35 tests still pass; 12/12 path-sanitize edge cases pass.
* fix: address Codex round-5 adversarial findings + md5-strong binding
Codex round 5 caught two more high issues plus a perf/correctness
concern. All real and addressed:
[high] decode_file_message scanned msg/attach in fallback, picking
up unrelated forwarded-record cached files. Outer files only ever
live in msg/file/{YYYY-MM}/; restricted the slow-path walk to that
subtree only. msg/attach holds merged-card and image attachments
whose presence here is a different message's payload, not ours.
[high] **真正根治** record/file 路径绑定问题：用 md5 强校验
Both decode_file_message (`<md5>` in appmsg) and decode_record_item
(`<fullmd5>` in dataitem) now extract the WeChat-supplied md5 and
hash candidate files locally to compare. If md5 doesn't match, the
tool fails closed with an explicit md5-mismatch error rather than
returning a path. The candidate that *does* match is uniquely
bound to the selected message — md5 collisions of distinct files
are cryptographically negligible. This fixes the heuristic-only
warning paths from rounds 3-4 with cryptographic evidence rather
than just user-facing notes.
As a side benefit, md5 dedup also lets decode_file_message return
a result when WeChat creates "(1)/(2)" copies of the same file:
same-md5 candidates are真同一文件副本 (user re-sent or auto-rename),
any one of them is correct.
When XML doesn't ship md5 (rare but possible), behavior reverts to
the previous fail-closed-on-multiple-candidates path with an
explicit "no md5 available, treating as heuristic" note.
[medium] _parse_app_message_outer was retrying every appmsg under
the 500K cap on initial 20K rejection, which made history rendering
O(content_size) on big non-record appmsgs. Added a substring
`<type>19</type>` short-circuit so only true type=19 records pay
the wider parser cost. Verified non-type=19 big XML now returns
None in <0.01ms instead of doing a 500K parse.
35/35 existing tests still pass. Real-data smokes:
- decode_record_item 142,1 → "✅ md5 校验通过，路径与 dataitem 唯一绑定"
- decode_file_message 171 (with same-name (1).pdf copy in cache) →
md5 dedup recognizes both as same content, returns one with
"✅ md5 校验通过"
- non-type=19 big appmsg parses in <1ms (substring short-circuit)
* fix: address Codex round-6 adversarial findings — strict md5 binding + chunked hash
[high] decode_file_message / decode_record_item now fail-closed when
the message XML has no md5/fullmd5 field — instead of returning a
heuristic single-candidate path with a warning. The previous
warning-only approach (rounds 4-5) didn't actually stop downstream
Read/PDF callers from using the wrong path. Now: no md5 = no path
returned, period. The error message lists the heuristic candidates
with mtime so the user can manually pick if absolutely needed,
but the tool itself does not commit to any of them.
Behavioral consequence: messages where wechat omits md5 (rare but
possible — e.g. some image/voice dataitems lack fullmd5) become
not-resolvable via these tools. Acceptable safety/utility tradeoff
per Codex's recommendation.
[medium] md5 verification was reading the entire candidate file into
memory via `_hashlib.md5(_f.read()).hexdigest()`. For 100MB+
attachments (videos in merged-record cards, large PDFs) this could
spike RSS or stall the MCP process. Replaced with a streaming
helper `_md5_file_chunked` (64KB chunks) plus a 500MB hard cap that
returns an explicit error rather than attempting verification on
oversized files.
35/35 existing tests still pass. Real-data smokes:
- decode_file_message 171 (with md5) → "✅ md5 校验通过"
- decode_record_item 142,1 (with fullmd5) → "✅ md5 校验通过"
- _md5_file_chunked size cap 1KB rejection works correctly
* fix: round-7 + revert round-6 over-strict — match real threat model
Two real bugs from Codex round-7 plus a partial revert of round-6
over-strictness that doesn't match this tool's actual threat model.
[high] Group type=19 with 'sender:<?xml...' (no newline) prefix not
stripped (Codex round-7 high #1)
_parse_message_content only split on ':\n', missing real-world
group rows where wechat writes 'wxid_xxx:<?xml ...' or
'wxid_xxx:<msg ...' inline. _format_app_message_text and
decode_record_item both received the prefixed content, parsed it
as raw XML, and failed. Now also strips on regex match against
'<?xml|<msg|<msglist|<voipmsg|<sysmsg' immediately after a sender
token. Verified with 5 prefix shape variants; legacy ':\n' still
works.
[medium] Record images use flat 'Img/0_t' filenames, not 'Img/0/*'
(Codex round-7 medium #2)
decode_record_item's datatype=2 (image) branch globbed for
'*/Rec/*/Img/{idx}/*' but real wechat caches store record images
as flat files: '*/Rec/<id>/Img/0_t', '*/Rec/<id>/Img/0', or
'*/Rec/<id>/Img/0.{ext}'. Added flat-pattern matching for
datatype=2 with the four observed filename shapes. File/voice/
video classes still use the F|A|V/{idx}/{filename} shape they
always did.
[revert] Round-6's "no md5 → fail-closed" is too strict for this
tool's actual usage
This MCP server is invoked locally by the user, paths surface
only in the local Claude conversation, and contacts are not
hostile. Codex round-6's hard fail-closed-on-missing-md5 broke
ergonomics for real wechat messages that lack md5 (some image
and voice dataitems) without a corresponding security gain in
this scenario. Reverted to round-5 behavior:
- md5 present → cryptographic verification, mismatch fails
- md5 absent → heuristic + ⚠️ warning, multiple-candidate
ambiguity still fails closed.
Kept all other round-6 hardening: streaming chunked md5, 500MB
cap, _safe_basename strict reject, _path_under_root realpath
check, multi-shard ambiguity, substring short-circuit for
non-type=19 big XML.
35/35 existing tests still pass; 5/5 group-prefix variants pass;
real-data smokes for both decoders still hit md5-verified paths.
* fix: round-8 — defer ambiguity until after md5 dedup + tighten file fallback
Two more findings, both real:
[high] decode_file_message no-md5 fallback was using `stem in f`
substring matching — `stem='论文'` would happily accept
`某老师论文.pdf`. Tightened to: exact match OR strict `(N)` copy
variant (`xxx(1).pdf`, `xxx (1).pdf`) per wechat's auto-rename
convention. 7/7 unit cases verify legitimate accept and false-
positive reject behavior.
[medium/P1 from GitHub Codex] decode_record_item had a stale early
`len(candidates) > 1 → ambiguity` check left over from round-7
refactor — it ran BEFORE the fullmd5 filter, making the md5
disambiguation block unreachable for the exact case where md5
could safely pick the right file. Removed the early check; md5
filter now runs first (and the post-md5 ambiguity check at line
~2467 still fails closed when md5 is missing AND multi-candidate).
35/35 tests pass. Real-data smokes (decode_file_message and
decode_record_item with their corresponding md5/fullmd5) still hit
the ✅ md5-verified path.
* test: add 29 helper-level regression tests for record-decoder helpers
Locks in the bugs fixed across PR #65's many review rounds so they
don't silently regress:
- _safe_basename (7 cases): strict reject of absolute paths,
parent-dir components, path separators, NUL — round-4 high #1.
- _md5_file_chunked (3 cases): streaming hash equals stdlib hashlib,
size cap rejects oversized files, missing file → error — round-6.
- _parse_message_content (5 cases): both legacy `:\n` and round-7
`:<?xml`/`:<msg` group-prefix shapes strip correctly; private
chat does not strip; bytes content returns the binary marker.
- _parse_app_message_outer (3 cases): small XML uses default cap,
oversized non-type=19 short-circuits (no 500K parse), oversized
type=19 retries successfully — round-5 medium #3 + round-2 P2-1.
- _format_record_dataitem (7 cases): text / file / image / 视频号 /
音乐 fall-through render correctly; unknown datatype falls back
to datadesc or [未知类型 N].
- _format_record_message_text (4 cases): >20KB outer XML expands
via _format_app_message_text end-to-end (regression for the
"P2-1 was a fake fix because I tested helper in isolation" miss);
empty datalist shows 待加载; chatroom marker appended; overflow
produces "…还有 N 条未显示" line.
The two MCP-tool wrappers (decode_file_message / decode_record_item)
lean on module globals + the real wechat cache layout. They are
exercised by real-data smoke runs in the PR description rather than
mocked here — mocking the entire wechat tree would dwarf the actual
logic under test.
64/64 total tests pass (35 existing + 29 new).
* refactor: simplify per /simplify code review (no behavior change)
Three review agents (reuse / quality / efficiency) flagged the
following high-confidence cleanups. All applied; all 64 tests still
pass; real-data smokes still hit md5-verified paths.
[quality] Remove PR-history references in comments
CLAUDE.md is explicit about this: comments should explain
non-obvious WHY, not narrate which Codex round caught what.
Cleared "round-2 high #2", "Codex round-3 medium #1", "round-5",
"round-6 强制", "round-7 实测", "round-8 high #1" from helper
docstrings and inline comments. Kept the substantive WHY (e.g.
"Reject 而不是 normalize because intent is suspicious").
[reuse + quality] Module-level datatype constants
Three places maintained their own copy of the datatype → label /
subdir mapping (_format_record_dataitem if-cascade, decode_record_
item type_label dict, subdir_map literal). Extracted
_RECORD_DATATYPE_LABEL and _RECORD_BINARY_SUBDIR to module top.
Single source of truth.
[quality] Hoist local imports to module top
Removed 7 inline `import glob as glob_mod` / `from datetime import
datetime as _dt` / `from datetime import datetime as _dt, timedelta
as _td` / `import hashlib as _hashlib` calls inside hot paths and
helpers. Aliases collapsed to plain names (datetime, timedelta,
glob, hashlib).
[efficiency] xpath: drop `.//` recursive descent for known-direct children
_format_record_dataitem was using `.//appbranditem/sourcedisplayname`
and `.//finderFeed/desc` even though both are direct children of
the dataitem. Changed to direct-child paths — meaningful for big
cards (50 items × subtree-walk per render).
[efficiency] md5 verification short-circuits on first match
Multiple candidates sharing the same md5 are wechat re-named copies
of the same file (e.g. `xxx (1).pdf`); any one is correct. Added
`break` after the first md5 match to skip hashing remaining
candidates (which can each be 100+ MB).
[quality] Compress _format_record_dataitem if-cascade
Datatypes that just emit `[label]` (2/3/4/5/7/23/37) and the link/
H5 pair (6/36) collapsed into membership checks against
_RECORD_DATATYPE_LABEL.
64/64 tests still pass.
---------
Co-authored-by: jiangbowen <robin@jiangbowendeMacBook-Air.local>
|
||
|
Belugary
|
c29e8dd868 |
fix: ImageResolver 支持微信 4.0+ V2 加密图片格式 (#61)
ImageResolver.decode_image 之前只调 xor_decrypt_file（老格式 XOR-only 路径），微信 4.0+（2025-08+）已经改用 V2 AES-128-ECB + XOR 混合加密， 导致 mcp_server.py 注册的 decode_image MCP 工具对 V2 .dat 文件返回的 "解密"内容是错的——Claude AI 通过 MCP 调用看不到 V2 时代的图片。 monitor_web.py 早已正确处理 V2（line 41-42, 791-795：从 _cfg 读 image_aes_key / image_xor_key 后调 decrypt_dat_file 自动 magic 分发）， 本次把 MCP 路径补齐，行为与 monitor_web.py 对齐。 改动： - ImageResolver.__init__ 增加 aes_key=None, xor_key=0x88 关键字参数 （默认值保持向后兼容，老调用方无需改动） - ImageResolver.decode_image 把 xor_decrypt_file 换成 decrypt_dat_file， 按 magic 自动分发 V2 / V1 / 老 XOR - V2 文件 + 缺 aes_key 时早期返回结构化错误信息，避免在 v2_decrypt_file 内静默失败成笼统的"解密失败" - v2_decrypt_file 入口接受 xor_key 字符串形式（int(_, 0) 解析）， 与 aes_key 已有的 str→bytes 处理对称，允许 config.json 写 "0x88" - mcp_server.py 实例化时从 _cfg 读 image_aes_key / image_xor_key 注入 兼容性： - ImageResolver 老调用方（不传 keys）继续走老 XOR 路径，零 breaking - V1 magic（\x07\x08V1）不会被 is_v2_format 拦截，走 decrypt_dat_file 内置固定 key，所以 aes_key=None 也能解 V1 文件 - 整 repo 只有 mcp_server.py 一处生产调用 ImageResolver(...)，已 grep 确认 测试覆盖（11 个新测试，tests/test_decode_image_v2.py）： - v2_decrypt_file 合成数据 round-trip 字节级相等 - decrypt_dat_file 按 magic 自动分发 V2 / V1 / 老 XOR 三条路径 - aes_key 接受 str（来自 config.json）和 bytes 两种形式 - xor_key 接受 str（如 "0x88"）和 int 两种形式 - V2 wxgf 裸流返回 fmt='hevc'（HEVC→JPEG 转换是 monitor_web 职责， 不在 ImageResolver 内做，保留 .hevc 输出） - ImageResolver 端到端：from local_id to decrypted file - ImageResolver(aes_key=None) + V1 文件走固定 key 路径 - ImageResolver(aes_key=None) + V2 文件返回 success=False + 友好错误 - ImageResolver 默认参数 + 老 XOR .dat 保持向后兼容 测试 46 个全部通过（11 新 + 35 旧）。 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Belugary
|
49356e1692 |
feat: macOS 图片 AES key 从磁盘 kvcomm 缓存派生（解决 #23） (#60)
* feat: macOS 图片 AES key 从磁盘 kvcomm 缓存派生（issue #23）
macOS 用户长期无法用 C 版 find_image_key_macos 从微信进程内存提取
V2 图片密钥（issue #23 报告 197K 候选全部失败）。新增
find_image_key_macos.py 走完全不同的路径：从磁盘 kvcomm 缓存
文件名派生密钥，无需扫描内存、无需 root、无需重签名。
派生算法
--------
- 扫 ~/.../app_data/net/kvcomm/key_<code>_*.statistic 文件名
- 对每个 (code, wxid) 候选：
xor_key = code & 0xFF
aes_key = MD5(str(code) + cleaned_wxid).hex()[:16] # ASCII 字符串
- 用 V2 _t.dat 文件 [0xF:0x1F] 16 字节做 AES-128-ECB 模板验证：
解出来必须是图像 magic（JPEG / PNG / GIF / WebP / wxgf）
- 为防短 magic 偶然命中，要求多个不同模板都通过验证才算成功
- 命中后写回 config.json 的 image_aes_key / image_xor_key，
monitor_web.py 自动加载
致谢
----
派生算法源自 @hicccc77 在 issue #23 的评论；参考实现见其 WeFlow
项目 (CC BY-NC-SA 4.0)。本模块是独立的 Python clean-room 实现，
未复制其 TypeScript 源码；函数边界与变量命名沿用算法的自然结构
（regex 模式 / MD5 调用顺序 / magic 字节表等不可避免地相同）。
健壮性细节
----------
- 多候选 kvcomm 路径：枚举 5 个不同的 macOS 微信版本路径布局
- 多模板交叉验证：默认收集 3 个不同密文，全部通过才算命中
- 已有 image_aes_key 仍有效时短路返回，不重写 config
- 原子写 config.json：tmp + os.replace + finally 清理 .tmp
- 多 wxid 候选：同时试 raw 和归一化后的 wxid（A_Hare_626a → A_Hare）
- print(flush=True) 逐次显式（与 find_image_key.py 风格一致）
测试
----
新增 tests/test_find_image_key_macos.py，53 个测试覆盖：
派生算法 / wxid 归一化 / kvcomm 路径推算（含多候选）/ 模板收集
（去重 / 子目录 / max_files 边界）/ AES 验证（5 种 magic / 短输入
/ 空 key）/ 多模板交叉验证 / 端到端集成（命中 / 各种失败分支）/
原子写 / main 短路（已有有效 key 不重写 / 已有错 key 落到派生）。
全部通过：python -m unittest discover tests → 88/88。
兼容性
------
- 无新增依赖（pycryptodome 已在 requirements.txt）
- 不改任何现有 Python 文件，零回归风险
- 现有 Windows / Linux 路径 (find_image_key.py / find_image_key_monitor.py) 不受影响
* feat: macOS 图片 AES key 加方案2 fallback (issue #68 思路)
PR #60 的方案1 (kvcomm 缓存派生) 在 kvcomm 缺失 / 多账号歧义 / 首次
启动等场景下会失败。@H3CoF6 在 issue #68 提出关键洞察:
wxid 目录后 4 位 hex == md5(str(uin))[:4]
意味着不需要 kvcomm，可以从 wxid 目录名 + 任意 V2 .dat 反推 uin。
本 commit 在保留 PR #60 方案1 不变的前提下，加方案2 作为 dispatcher
fallback。
方案2 算法
----------
1. 从 db_dir 提 wxid 后 4 位 hex 作为 md5 前缀目标
2. 扫多个 V2 .dat 末字节投票反推 xor_key (假设 JPG EOI 0xD9,
默认至少 3 个样本投票)
3. 枚举 0~2^32 中 (uin & 0xff == xor_key) 的 2^24 个候选,
md5(str(uin))[:4] 匹配 wxid 后缀 → 得 ~256 个 uin 候选
4. 对每个候选算 aes_key, 用 PR #60 的 verify_aes_key_against_all
做 AES 模板交叉验证, 唯一定位 uin
实现
----
- find_image_key_macos 重构为 dispatcher: 先方案1 (kvcomm),
失败 fallback 方案2 (候选搜索); 模板收集移到 dispatcher 共享
- 新增 helper: extract_wxid_parts, derive_xor_key_from_v2_dat,
bruteforce_uin_candidates
- 模块顶部 docstring 加方案2 算法说明 + @H3CoF6 致谢
(保留 PR #60 对 @hicccc77 的方案1 致谢)
clean-room 声明
---------------
方案2 按 issue #68 的算法描述独立实现，未引用 @H3CoF6 任何代码。
方案1 仍沿用 PR #60 实现 (其 clean-room 声明对 @hicccc77 / WeFlow
保持不变)。
健壮性细节
----------
- xor_key 反推默认 min_samples=3, 样本不足直接放弃方案2 (避免
1-2 个样本时一旦撞到非 JPG 就 lock 错 xor_key)
- wxid 后缀正则收紧为 [0-9a-fA-F]{4} (md5 hex), 非 hex 后缀直接
返回 None 而非误导用户跑空候选搜索
- 投票分歧时打印 warning, 但仍试取多数 (兼容 attach 含少量非 JPG)
- 删除重构后未用的 import glob; Counter 统一在模块顶部 import
测试
----
新增 17 个测试 (53 → 70), 全部 7.4s 内通过:
- ExtractWxidPartsTests (5)
- DeriveXorKeyFromV2DatTests (7, 含新增 below_min_samples 边界)
- BruteforceUinCandidatesTests (1, 真跑全空间金标准验证)
- FindViaBruteforceTests (3)
- DispatcherFallbackTests (1, mock 加速)
顺手修复 2 个 pre-existing 测试 fail
------------------------------------
test_account_with_4char_alnum_suffix_stripped 与
test_returns_raw_and_normalized_when_different 用 6-char 后缀
your_wxid_a1b2c3, 但 normalize_wxid 只去 4-char 后缀 (匹配真实
macOS 路径) → 测试期望与代码不一致, 长期 fail。统一改用 4-char
后缀让测试与 macOS 现实对齐。
兼容性
------
- API 不变: find_image_key_macos(db_dir) 签名 / 返回值不变
- 现有 53 个测试全部仍通过 (含 happy path / 各种返回 None 分支 /
main 短路 / 原子写)
- 真实数据验证: 在本地 macOS 微信 4.x 上方案2 端到端跑通, 结果
与方案1 完全一致
* fix: replace test fixture with synthetic uin/wxid (privacy hardening)
PR #60 测试 fixture 与 docstring 示例之前用了真实 uin (8 位十进制)
作为 golden value，并在 docstring 里把 wxid 后缀作为示例展示。虽然
单独的 uin/suffix 不直接 unlock 任何资产 (需要配合真实 wxid + 物理
访问加密文件)，但行业最佳实践 (yt-dlp / openssl / Linux kernel test
fixture) 都明确要求用合成确定性值, 不绑定任何真实账号。
合成方案
--------
- uin: 12345678 (8 位, 一目了然 placeholder)
- suffix: md5("12345678")[:4] = "25d5" (派生, self-consistent)
- wxid_full 示例: your_wxid_25d5
- wxid_norm 示例: your_wxid
- aes_key_test_value: a0c093edddc98490 = md5("12345678your_wxid")[:16]
- xor_key: 0x4E (= 12345678 & 0xFF)
改动范围
--------
- tests/test_find_image_key_macos.py: 全部 fixture 改用合成值,
bruteforce 测试的 xor 也对应更新 (0x7F → 0x4E)
- find_image_key_macos.py:260 docstring 示例: 真实 wxid 字符串
替换为 placeholder
- 长 kvcomm 缓存文件名 fixture 同步合成 (避免暴露真实时间戳 / 内部 ID)
测试
----
70/70 仍通过 (7.1s), 合成 fixture self-consistent。
非范围 (历史 commit
|
||
|
btc-z
|
66eddaff0e |
feat: transcribe_voice 新增 OpenAI Whisper API 后端 (#66)
默认 local，零行为变化。opt-in 双因素：transcription_backend=openai 且 openai_api_key 都齐才生效；任一缺失静默回退 local + stderr 一行警告。 首次进入云路径会 stderr 警告"语音将上传至 OpenAI 服务器"。 新增 config.json 字段： - transcription_backend: "local" (默认) | "openai" - local_whisper_model: "base" (替换 mcp_server.py 里硬编码 DEFAULT_WHISPER_MODEL) - openai_api_key: "" (默认空；openai 包为 optional，按需 pip install) 关键技术选择： - _transcribe(wav, backend) 单一 if/else 分发，不引入插件/工厂层 (Rule of Three —— 只有一个云后端时不值得抽象) - 文件 > 25MB 在 OpenAI() 实例化之前提前拒绝，避免无谓上传 - 错误分类清晰: 缺 key / 缺 openai 包 / 401 / 429 / APIError 各自的提示 - PR #58 缓存 schema 自然扩展: 条目加 backend 字段，命中需 backend+model_size 都匹配 - 旧条目缺 backend 字段视为 "local"，向前兼容 PR #58 已落盘的所有数据 - transcribe_chat.py 批量 CLI 与 MCP 工具共享同一份配置，保持一致 新增 2 个测试 (tests/test_openai_backend.py)，只覆盖回归风险最高的两条： - 文件 > 25MB 必须在 SDK 实例化前拒绝（隐私契约的防线） - backend 不匹配的旧条目不命中（避免切后端时返回错后端结果） 其余路径要么琐碎（默认值读取）、要么坏掉时声音很大（SDK 错误、ImportError）， 要么已被 PR #58 现有测试隐式覆盖（缺 backend 字段的旧条目），不再单独写测试。 顺手把 README 里 PR #53 漏掉的 voice 三件套（get_voice_messages / decode_voice / transcribe_voice）补进 MCP 工具表，并新增"⚠️ 语音转录隐私" 章节说清数据流向、成本（约 \$0.006/分钟）、25MB 上限、回退行为。 Closes ylytdeng/wechat-decrypt#59 |
||
|
Belugary
|
989badd14f |
feat: 给 transcribe_voice 工具加持久化缓存 (#58)
Whisper 本地推理在 CPU 下每条语音数秒到数十秒，且同一段 voice_data 产出相同 text，非常适合缓存。新增 voice_transcriptions.json 持久化 存储，命中时跳过 DB 查询、SILK 解码和 Whisper 推理全链路。 关键技术选择： - 缓存 key 用 json.dumps([username, local_id])，即使 username 含 分隔符也不冲突 - 写入走 tmp + os.replace 原子替换，进程中断不会损坏主文件 - 条目记录 model_size，Whisper 默认模型升级后旧条目自动失效 - 空转录也缓存（配合 model_size 失效），避免静音片段每次重跑 - threading.Lock 防御并发 load/save 竞态 - 首次 OSError 写 stderr 警告一次，后续静默避免刷屏 小的行为改进：resolve_username 移到 whisper/pysilk 导入探测之前， bad chat_name 情况下不再需要 whisper 已安装也能给出"找不到聊天对象" 的错误提示。 15 个新测试：持久化 roundtrip、UTF-8 保留、corrupt JSON 容错、原子 写、写前失败不污染主文件、并发 load/save、缓存命中跳过重活、model 不匹配视为 miss、key 对含分隔符 username 的防御。全部通过。 |
||
|
dsjzazs
|
7c42ff5d38 | Investigate get_chat_history limit | ||
|
dsjzazs
|
9ae558a31e | Fix global search pagination | ||
|
dsjzazs
|
b623711410 | Add MCP search unit tests |