Three new flags for export_all_chats.py:
- -i / --incremental: reads existing JSON, appends only new messages
(deduplicates by local_id, preserves transcription field on merge)
- --start / --end: filter messages by date range (YYYY-MM-DD or timestamp)
passes start_ts/end_ts directly to mcp_server._query_messages
- --dry-run: preview mode (shows counts without writing files)
Voice transcription in incremental mode only processes newly appended
voice messages — existing transcribed entries are untouched.
## Problem
Voice messages in `_format_message_text` previously rendered as a bare
`[语音] (local_id=N, ts=T)` because msg_type=34 fell through to the generic
non-text branch with no schema-aware summarizer. LLMs reading chat history
had no way to judge whether a voice clip was worth calling `decode_voice`
on without first inspecting it.
## Fix
New helper `_format_voice_text(content)` parses the embedded
`<voicemsg voicelength="…">` and renders `[语音 N.Ns]` (duration to one
decimal, milliseconds → seconds). Type=34 dispatches through it, then
appends the existing `_id_suffix()` so the local_id annotation is
preserved end-to-end:
[语音 3.3s] (local_id=72481, ts=1700000000)
Falls back to `[语音]` (still with `_id_suffix()`) when content is empty,
`<voicemsg>` is absent, XML parse fails, or `voicelength` is missing /
zero / non-numeric.
XML parsing routes through the existing `_parse_xml_root` so the
`_XML_UNSAFE_RE` DOCTYPE/ENTITY filter and 200KB size cap are reused —
no new XXE surface.
## Tests
12 new cases in `tests/test_voice_format.py`: happy path, subsecond,
multi-second, missing / zero / non-numeric voicelength, empty / None
content, missing `<voicemsg>` tag, malformed XML, XXE payload, and two
end-to-end cases through `_format_message_text` (with and without
voicelength) to pin the full rendered output including `_id_suffix()`.
Baseline 183 → 195 passing, 0 regressions.
## Scope
- `mcp_server.py`: adds `_format_voice_text` helper and one branch in
`_format_message_text` (base_type == 34). No public surface change —
this only affects formatting of messages that previously rendered as
the bare `[语音]` fallback.
- `tests/test_voice_format.py`: new file, synthetic fixtures only (no
real PII).
## Problem
`_contact_names`, `_contact_full`, `_contact_tags`, and `_self_username`
are populated lazily on first access and never invalidated for the
process lifetime. When `contact.db` is re-decrypted (new contact added,
remark or group name edited, etc.) the on-disk DB updates but the
running MCP server keeps serving stale data — newly-added contacts are
invisible to `resolve_username` and downstream tools until the server
is restarted.
## Fix
Track the mtime of the contact.db backing file. On every
`_get_contact_db_path()` call (which all contact accessors go through),
compare against `_contact_db_mtime`; if it changed, clear all four
caches and record the new mtime. Lookups that don't trigger a real
re-decryption pay only one `os.path.getmtime()` syscall.
The function is reorganized so `_get_contact_db_path()` is the single
source of truth for both "where is contact.db" and "do we need to
invalidate" — `get_contact_names` and `_load_contact_tags` consult it
unconditionally before the early-return on the populated cache.
Also reorders `_get_self_username` to call `get_contact_names()` first
(which now triggers the mtime check via the path lookup) before
returning a cached `_self_username` — otherwise the rename case would
still resolve to the stale name.
## Tests
Baseline 183 → 183 passing, 0 regressions.
The pattern (mtime-track + invalidate-on-change) mirrors the existing
behaviour of DBCache, which already re-decrypts contact.db when the
source mtime changes; this fix closes the symmetric gap on the
in-memory side.
## Scope
- `mcp_server.py` only.
- No public surface change. Affects the contact-cache layer's behaviour
on re-decryption — previously: stale until restart; now: refreshed
on next contact-related call.
Add structured parsing for transfer messages so they no longer fall
through to the generic `[链接/文件]` fallback in chat history exports.
Mirrors the dispatch + helper pattern PR #65 (merged-forward type=19)
established for `base_type=49` appmsg sub-types.
## What is added
**Helpers (mcp_server.py):**
- `_TRANSFER_PAYSUBTYPE_LABEL` — maps the 6 community-consensus paysubtypes
(1 发起 / 3 已收款 / 4 已退还 / 5 过期已退还 / 7 待领取 / 8 已领取);
unknown values degrade to `未知(paysubtype=N)` so a new variant in a
future WeChat build is visible rather than silently dropped.
- `_extract_transfer_info(appmsg)` — pulls fields out of `<wcpayinfo>`,
with snake/camelCase fallback (`feedesc`/`feeDesc`, `pay_memo`/`paymemo`)
observed across WeChat versions.
- `_format_transfer_message_text(appmsg, title)` — one-line render
for chat history: `[转账·已收款] ¥100.00 备注: lunch`.
**Dispatch (mcp_server.py):**
- `_format_app_message_text` gains an `app_type == 2000` branch that
routes to `_format_transfer_message_text`. `get_chat_history`,
`export_chat`, `export_all_chats` and `monitor_web` all inherit
automatically.
**New MCP tool (mcp_server.py):**
- `decode_transfer(chat_name, local_id, create_time=0)` — full
structured view: direction, amount, memo, payer/receiver wxid,
transfer id, transcation id, begin/invalid timestamps. Uses the
same multi-shard scan + ambiguity-by-create_time pattern as
`decode_file_message` / `decode_record_item`.
**CLI wrapper:**
- `decode_transfer.py` at the repo root — argparse wrapper that prints
the same text as the MCP tool, returning non-zero exit when the
message can't be decoded (script-friendly).
**JSON export (chat_export_helpers.py + export_chat.py + export_all_chats.py):**
- `_extract_content` now returns `(rendered, extras)`. `extras` carries
structured fields when a message type has more signal than the
human-readable string (currently: transfers → `type:"transfer" +
transfer:{direction, fee_desc, pay_memo, ...}`). The channel is
forward-compatible — future additions (video号 metadata, expanded
merged-forward, etc.) flow through the same shape without changing
the caller signature. JSON consumers that only read `content` are
unaffected; the change is additive.
**monitor_web (monitor_web.py):**
- Backend dispatch branch + orange-yellow `.msg-transfer` card CSS +
`renderRich` JS handler.
## Tests
12 new cases in `tests/test_record_decoders.py`:
- `TransferPaysubTypeLabelTests` — locks the 6-value label table.
- `ExtractTransferInfoTests` (6 cases) — full field round-trip, missing
`<wcpayinfo>` fallback, snake/camelCase variants, unknown paysubtype
degradation, empty paysubtype handling.
- `FormatTransferMessageTextTests` (4 cases) — initiate / received-with-memo /
missing-wcpayinfo / missing-fee-desc.
- `AppMessageDispatchTransferTests` — `_format_app_message_text` routes
type=2000 correctly so `get_chat_history` / `export_chat` both pick
it up.
All fixtures use synthetic placeholder values (`wxid_payer_synth`,
`¥100.00`, `1` + 27×`0`); no real PII or transaction IDs.
## Scope
7 files, +546 / -15 (additions only — no behavior change for existing
message types). All 180 tests pass locally (168 baseline + 12 new).
The post-decrypt verification step (sqlite3.connect(out_path) + table list, around line 163) opens the freshly-written .db in default journal mode. Even though the connection is closed cleanly, SQLite leaves behind empty <db>-shm and <db>-wal companion files in OUT_DIR.
Downstream tools that later open the same .db will see those companion files and try to roll the (empty / stale) WAL forward, producing "database disk image is malformed" or silently masking the most recent pages. The decrypted DB itself is fine — the residuals are pure noise from the verification connection.
Fix: after the verification block (success or failure), unconditionally os.remove() out_path + "-shm" and out_path + "-wal" if present. Errors during cleanup are swallowed.
Tests: existing tests/ pass (168/168). The cleanup is additive and only runs after the existing verification path; no behavior change for callers that do not inspect OUT_DIR for companion files.
Scope: 10 lines in decrypt_db.py. No public API change, no schema change, no new dependency.
## Problem
On macOS, the default `db_dir` template in `config.py` (line 20) points to
`~/Documents/xwechat_files/your_wxid/db_storage`, but WeChat 4.x on macOS
stores data inside the app sandbox container at
`~/Library/Containers/com.tencent.xinWeChat/Data/Documents/xwechat_files/<wxid>/db_storage`.
`_auto_detect_db_dir_macos()` (config.py:166) handles the common case, but
when auto-detect fails — fresh install with no scan results yet, permission
issues, atypical install location — the template fallback is what the user
sees in their generated `config.json`. Today that fallback is a Linux-style
path that does not exist on macOS, so the user has to manually correct it
before the first run can succeed.
## Fix
Update the darwin branch of `_DEFAULT_TEMPLATE_DIR` to the actual sandbox
container path. `your_wxid` remains a placeholder.
Linux and Windows templates are unchanged.
## Tests
Existing `tests/` pass (168 / 168). The change only affects a module-level
constant; no behavior change for users whose auto-detect already succeeds.
## Scope
3 lines in `config.py`. No public API change, no schema change, no
dependency change.
* fix(image): scope local_id lookup by chat_id + use real column name
`ImageResolver.get_image_md5` made two wrong assumptions about the
production `MessageResourceInfo` schema, which made the decode_image
MCP tool always fail with "无法找到 local_id=X 的图片信息":
1. The column is `message_local_id`, not `local_id`. The current query
throws `sqlite3.OperationalError: no such column: local_id`, but the
exception is swallowed by `except Exception: pass`, masking the real
failure as a silent miss.
2. `message_local_id` is not globally unique. In production it repeats
across chats, and within an active chat the same local_id can recur
up to 7 times (observed on a real DB). The production schema scopes
by `chat_id`, resolved from `ChatName2Id.rowid WHERE user_name = ?`.
Fix: `get_image_md5` now takes `(username, local_id)`:
- Resolve `username -> chat_id` via `ChatName2Id`.
- Query `MessageResourceInfo` filtered by `chat_id + message_local_id
+ message_local_type == 3` (image type; high bits are session flags,
so use `% 2^32`), ordered by `message_create_time DESC LIMIT 1`.
External callers (`mcp_server.decode_image_tool` /
`list_chat_images_tool`) already pass `username` through
`ImageResolver.decode_image()` / `list_chat_images()`, so the public
API is unchanged. Only the internal helper signature shifts.
The existing test fixture in `test_decode_image_v2` used the same wrong
schema as the buggy code (`CREATE TABLE MessageResourceInfo (local_id
INTEGER PRIMARY KEY, packed_info BLOB)`), so the tests passed against a
self-consistent fiction. The fixture is rebuilt to match real columns
plus `ChatName2Id`, and three regression tests are added:
- cross-chat collision (same local_id in 3 chats; must pick the right one
and not the type=43 video row)
- same-chat reuse (same local_id, two timestamps; must pick the newer)
- unknown chat (username not in ChatName2Id; structured error, no crash)
All 154 tests pass locally (151 baseline + 3 new).
* fix(image): surface get_image_md5 errors and use read-only DB open
Two follow-ups on top of the chat_id scoping fix:
1. The bare `except Exception: pass` was the original failure mode: it
silently swallowed `OperationalError: no such column: local_id` when
the production schema diverged from the old `local_id` column name,
masking the bug this PR fixes. Print the exception to stderr so
future schema drift surfaces immediately instead of returning a
misleading "image not found" error.
2. Open message_resource.db with `file:...?mode=ro` URI to match the
rest of the project (monitor_web.py uses this idiom in 9 places).
The DB is read-only for our purposes and a running WeChat may still
hold it; using URI ro avoids any chance of lock contention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WeChat rolls a chat's messages over to the next `message_N.db` shard
once the current shard fills up (~100 MB), so any chat older than the
current shard window has its history split across multiple shards. The
other message-query tools — `get_chat_history`, `search_messages`, and
`decode_image` — already iterate every matching shard via the plural
helper `_find_msg_tables_for_user`. Only `get_chat_images` still used
the singular `_find_msg_table_for_user`, which returns the first shard
that contains the user's table.
Effect: every image that lived in a non-first shard was silently
dropped from `get_chat_images`. On a long-lived chat with many images,
the tool would return only the most recent slice and pretend the rest
did not exist.
Fix: switch `get_chat_images` to `_find_msg_tables_for_user`, fetch
`limit` images per shard, merge, sort by `create_time` DESC, and slice
to `limit`. This mirrors how the other tools fan out across shards.
Tests in `tests/test_get_chat_images_multishard.py`:
- `test_collects_images_from_every_shard` — both shards' images appear
in the output (the regression case)
- `test_global_sort_by_create_time_desc` — newer image from an older
shard still wins, output is globally sorted (not per-shard concat)
- `test_limit_truncates_globally_across_shards` — limit=3 takes the 3
newest overall, not "first shard wins"
- `test_no_shards_returns_not_found` — empty shard list path
- `test_all_shards_empty_returns_no_images` — every shard empty path
All 156 tests pass locally (151 baseline + 5 new). Public tool
signature is unchanged; only the internal scanning loop is widened.
When a chat history contains a name-card message (msg_type=42), the
dispatcher in `_format_message_text` had no case for `base_type == 42`,
so it fell through to the generic non-text branch:
elif base_type != 1:
type_label = format_msg_type(local_type)
text = f"[{type_label}] {text}" if text else f"[{type_label}]"
`text` for type=42 is the full raw `<msg ...>` element, so chat history
exports emitted `[名片] <msg username="..." antispamticket="v2_..."
brandIconUrl="https://wx.qlogo.cn/..." bigheadimgurl="..." ... />`.
That payload has two problems:
1. It leaks anti-spam tokens (`antispamticket`) and head-image CDN URLs
into chat logs that are routinely piped to LLMs and other downstream
tools.
2. The raw XML drowns out the actual signal — a human or an LLM reading
the chat just wants to know "X shared Y's contact".
This PR adds `_format_namecard_text(content)` that pulls only the three
useful attributes:
- `nickname` — display name
- `username` — wxid (annotated as "公众号" when prefixed `gh_`)
- `certinfo` — user-authored bio
and wires it into the dispatch chain via a new `elif base_type == 42:`
branch, sitting alongside the existing `49` (app message) handler. It
reuses `_parse_xml_root` and `_collapse_text` — no new helpers
introduced.
Tests: 7 cases in `tests/test_namecard_format.py` covering the realistic
shape (with antispamticket / brand URLs that must NOT appear in output),
official accounts (`gh_*`), missing certinfo, missing nickname, missing
both identifiers, and broken-XML fallthrough.
All 158 tests pass locally (151 baseline + 7 new).
`v2_decrypt_file` previously wrote files to disk even when the keys were
wrong, producing garbage output with no way for the caller to detect the
failure:
1. Wrong AES key -> `detect_image_format` returns 'bin' (magic does not
match any known format) -> a `.bin` file of random bytes was written.
2. Wrong XOR key with correct AES key -> file header looks like a valid
jpg/png (the AES segment decrypts correctly) but the trailing XOR
segment is scrambled -> callers get a half-valid image file that
image viewers either render as truncated or fail to open.
Both cases now return `(None, None)`:
- `fmt == 'bin'` -> fail fast, no file written.
- `xor_size >= 2` -> validate trailer magic by format:
* jpg must end with FF D9 (EOI marker)
* png must contain IEND chunk in the last 12 bytes
Other formats (gif/bmp/tif/webp/hevc/wxgf) lack a mandatory trailer
signature, so they skip the check to avoid false rejection.
Also fixes a latent bug in `test_decode_image_v1_no_aes_key_uses_fixed_key`:
the test built the synthetic .dat with `TEST_XOR_KEY=0x37` but constructed
`ImageResolver` without `xor_key=`, defaulting to `0x88`. The XOR segment
was always scrambled — the test passed because the AES segment alone was
enough for `detect_image_format` to return 'png' from the header, and no
trailer validation existed to catch the corruption. The new trailer check
surfaces this, so the test now passes `xor_key=TEST_XOR_KEY` explicitly.
Tests: 5 new cases (wrong AES key / wrong XOR for jpg / wrong XOR for
png / xor_size=0 bypass / wxgf bypass). All 156 existing tests still pass.
* Add export_all_chats.py: batch export all WeChat chats to JSON
Mirrors export_chat.py functionality to export every chat in the
decrypted WeChat database at once. Output format is byte-for-byte
identical to export_chat.py.
Usage: python3 export_all_chats.py [output_dir]
- Reads all sessions from decrypted/session/session.db
- Uses the same content extraction pipeline as export_chat.py:
_resolve_sender, _extract_content, _msg_type_str, sticker/video/system
- Outputs group_<display>.json or single_<display>.json with same schema
- Progress reporting every 100 exports with summary at end
* refactor: 将 7 个重复 helper 函数提取到 chat_export_helpers.py
根据 PR #77 review 反馈，将 export_chat.py 和 export_all_chats.py 中
逐字复制的消息格式化函数提取到共享模块 chat_export_helpers.py。
提取的函数:
MSG_TYPE_MAP, _msg_type_str, _resolve_sender,
_decode_sticker_desc, _format_sticker_message,
_format_system_message, _format_video_message, _extract_content
两个导出脚本现在从 chat_export_helpers import 所需函数，
消除代码漂移风险。
* feat: 离线计算图片密钥
* fix(find_all_keys): address review feedback on #69
Apply 5 fixes per @ylytdeng's review:
- find_xor_key: return None when last-byte ^ 0xD9 doesn't match the
first-byte-derived xor_key (was returning xor_key in both branches,
so the validation was a no-op)
- multiprocessing cleanup: split single-line terminate, add
p.join(timeout=1) loop to avoid orphan workers
- replace 3 bare `except:` with `except Exception:` so KeyboardInterrupt
can break the brute-force loop
- add actionable hint ("请先在微信中查看 2-3 张图片") when xor_key or
ciphertext can't be derived from attach_dir
- drop try/except ImportError fallback on `from Crypto.Cipher import AES`
(and the now-dead `if not AES` guards); pycryptodome is already a hard
dependency elsewhere in the project
Original algorithm and multiprocessing implementation by @H3CoF6 in #69.
Review by @ylytdeng: https://github.com/ylytdeng/wechat-decrypt/pull/69
Co-authored-by: H3CoF6 <190114211+H3CoF6@users.noreply.github.com>
---------
Co-authored-by: Belugary <53219544+Belugary@users.noreply.github.com>
Co-authored-by: H3CoF6 <190114211+H3CoF6@users.noreply.github.com>
* feat: 解析合并转发的聊天记录消息（appmsg type=19）+ 新增文件路径查找工具
mcp_server.py:
- _format_app_message_text 增加 app_type=19 分支，解析 <recorditem> 内嵌
XML，把"[链接/文件] xxx的聊天记录"展开成多行 datalist 内容（含发送者/
时间/数据类型）。覆盖 datatype 1/2/3/4/5/6/7/8/17/19/22/23/29/36/37
共 14 种类型；超过 50 条自动截断；空 datalist fallback 到"（待加载）"
- 新增 decode_file_message 工具：从 type=49+sub=6 消息找本地副本路径
（~/Library/.../msg/file/{YYYY-MM}/原文件名.{ext}），返回精确路径
+ size 二次确认，处理同名 (1)(2) 后缀
- 新增 decode_record_item 工具：从 type=49+sub=19 合并记录的第 N 个
dataitem 找本地副本（msg/attach/{table_hash}/*/Rec/*/F/{idx}/{name}），
未下载时给精确"在 wechat 点击哪一项"指引
回归测试：在数千条真实合并转发消息上 ~83% 完美展开 datalist 内容，
剩余的 content 缺失/消息被撤回情况下行为与改前一致（fallback 到原 [链接/文件]）。
* fix: hoist subdir_map in decode_record_item to avoid UnboundLocalError
When the chat's attach directory does not exist (no merged-record
attachment ever downloaded for that chat), the if-block defining
`subdir_map` was skipped, so the not-found branch's reference to
`subdir_map.get(datatype, '?')` raised UnboundLocalError instead of
returning the intended guidance message.
Hoist the dict definition above the if-block so both branches can
safely reference it.
Caught by Codex review on PR #65.
* fix: address Codex P2 — large recorditem XML + glob escape
Two issues caught by Codex review on PR #65:
P2-1: _parse_xml_root rejects payloads >20KB, which silently dropped
~330 large merged-record cards (max observed 418KB with 99 dataitems)
back to "[链接/文件]" fallback. Add a dedicated _parse_record_xml with
a 500KB limit for embedded recorditem XML; routes both call sites in
_format_record_message_text and decode_record_item to it. Boosts
overall parse coverage from 83% to ~87% on real-world data.
P2-2: decode_record_item passed datatitle directly to glob, so file
names containing [ ] * ? would be treated as glob patterns rather
than literals — leading to wrong candidates or missed real files. Wrap
datatitle with glob.escape() before the exact-match query.
Full unit-test suite (35 tests) still passes.
* perf+style: speed up decode_file_message + minor consistency fixes
Self-review findings on top of the Codex P1+P2 fixes:
- perf: decode_file_message previously os.walk-ed `msg/file/` and
`msg/attach/` from scratch on every call, scanning ~185k files /
17GB on a real-world install (~6.3s per call). Now first reads
`create_time` from the message and globs only the matching
`msg/file/{YYYY-MM}/` (plus ±1 month for cross-month edge cases),
with the original walk preserved as a fallback. Measured speed-up
~10x on first call, ~750x on warm cache.
- decode_record_item: extend `type_label` to cover datatype 23
(视频号直播) and 36 (小程序/H5) so the not-found message matches
the labels emitted by _format_record_dataitem instead of falling
back to a raw `datatype=23` string.
- decode_record_item: replace unused `sub_type_packed` with `_` to
silence the "name assigned but unused" smell.
- _parse_record_xml: comment now states the empirically observed
~418KB upper bound (was "~50KB"), making the 500KB ceiling
obviously sufficient.
All 35 existing tests still pass.
* fix: address Codex round-3 P2 — multi-shard lookup + size-validate month scan
Two more issues caught by Codex review on PR #65 that I missed during
self-review:
P2-3 (multi-shard local_id lookup): both `decode_file_message` and
`decode_record_item` were resolving a single message-table via the
singular `_find_msg_table_for_user`, but a chat's messages can span
multiple message_N.db shards (search_messages and history-iteration
already use `_find_msg_tables_for_user`). When the requested local_id
lived in a different shard the tools incorrectly returned "找不到
local_id" or — if IDs collide across shards — picked the wrong row.
Now both tools iterate all shards and stop at the first hit; the
not-found message reports how many shards were scanned.
P2-4 (size validation in month-scan fast path): the perf-fix in the
prior commit collected `msg/file/{YYYY-MM}/` matches without verifying
size, so when a same-named-but-different-size copy existed in the
target month the candidate list was non-empty, the walk-fallback was
skipped, and the later `size_match` filter could end up empty —
returning a wrong-size file. Now the month-scan filters by `totallen`
upfront when known, so unmatched candidates don't poison the fallback.
Same one-shot size validation applied to `decode_record_item`'s
exact-name glob branch for symmetry.
These were both "I should have caught" issues — Codex did the
cross-tool consistency check (singular vs plural shard helper) that I
skipped, and stress-tested an edge case (month-scan finds same-name
wrong-size) that I didn't think through when writing the perf fix.
35/35 existing tests still pass. Real-data smoke: decode_file_message
0.96s end-to-end (multi-shard scan + size validation),
decode_record_item 0.03s.
* refactor: reuse _parse_message_content helper for group prefix stripping
Self-review found that decode_file_message and decode_record_item
hand-rolled their own heuristic for stripping group-chat sender
prefixes ("wxid_xxx:\n<xml...>") via a string-startswith check, while
the rest of the project already uses the canonical
`_parse_message_content(content, local_type, is_group)` helper for
exactly this purpose.
Wired both tools to that helper, deriving is_group from the username
suffix `@chatroom`. Existing edge cases (private chat content with
literal "<...>", group content with "wxid_xxx:\n", etc.) still pass.
35/35 tests still pass; 6/6 edge-case smokes still pass.
* fix: address Codex adversarial-review high+medium findings
Adversarial review caught four issues that the surface-level passes
missed. All four are now fixed end-to-end (validated against real
data, not just helper-level smoke):
[high] Large recorditem outer XML actually parsed:
Previous P2 fix added _parse_record_xml(500KB) for the inner CDATA
but the outer appmsg was still gated by _parse_xml_root(20KB), so
any merged-record card whose outer XML exceeded 20KB silently fell
back to "[链接/文件]" and never reached the inner expansion. Now
_parse_xml_root accepts a max_len kwarg, _format_app_message_text
retries with _RECORD_XML_PARSE_MAX_LEN when the default cap rejects
the outer XML, and _format_record_message_text passes the wider cap
for inner parses. Real-data check: a 34KB outer / 67-dataitem card
now expands fully via the get_chat_history → _format_message_text
→ _format_app_message_text → _format_record_message_text chain.
[high] Multi-shard local_id ambiguity:
decode_file_message and decode_record_item previously broke on the
first shard match. Empirically confirmed local_id 171 in the test
account exists in TWO shards as TWO different messages (one type=1
text, one type=6 file at different create_times). Now both tools
scan all shards, fail with an explicit ambiguity error when more
than one row matches, and accept an optional create_time arg from
the user to disambiguate uniquely.
[medium] History output now exposes (local_id, ts) for file and
record cards, and record dataitem rows are prefixed with their
0-based [item_index]. Without these, callers had no way to feed
decode_file_message / decode_record_item a stable identifier.
[medium] decode_file_message now requires appmsg type=6 and an
appattach node, refusing to search the local cache by title/size for
unrelated app messages (links, miniapps, record cards) that happen
to share a title with a real file.
35/35 existing tests still pass. Real-data smokes:
- 34KB outer XML / 67 dataitems expanded end-to-end
- multi-shard ambiguity correctly raised + resolved by ts kwarg
- history output now contains "(local_id=N, ts=T)" suffixes and
"[N]" dataitem prefixes
* fix: address Codex adversarial round-2 high findings
Round-2 adversarial review caught two issues my self-review missed
again. Both are now fixed end-to-end:
[high] decode_record_item also rejects large outer XML (mcp_server.py:2124-2126)
Round-1 high #1 was fixed by adding a wider-limit retry inside
_format_app_message_text, but decode_record_item itself still
parsed the outer appmsg with `_parse_xml_root(xml_text)` at the
default 20KB cap. Same root cause: I patched one caller, missed
the other — exactly the kind of cross-tool inconsistency that
cost two rounds already.
Extracted a shared `_parse_app_message_outer(content)` helper that
encapsulates the "try default cap, fall back to wider limit when
default rejects" pattern. Now used by all three call sites:
_format_app_message_text, decode_file_message, decode_record_item.
Real-data check: a 34KB outer (67 dataitems) parses through every
caller path, not just history rendering.
[high] Record attachment lookup silently picks wrong cached file
Previous lookup had three fallback tiers (filename+size → size only
→ cross-subdir size only) and on multiple matches sorted by mtime
and took newest. Two failure modes:
1. Different forwarded-record cards in the same chat may produce
paths with identical (filename, item_index, datasize), and the
mtime tiebreak lets the tool return another record's file while
reporting "找到本地文件: ✅".
2. Cross-subdir size-only fallback can match files belonging to
unrelated dataitem types entirely.
Now fail-closed:
- Strict filename + size match only when datatitle is known.
- Size-only fallback now ONLY when datatitle is missing
(e.g. datatype=2 thumbnails) AND scoped to the same sub-dir +
item_index — no more cross-Rec leakage.
- Removed the cross-subdir terminal fallback entirely.
- Multiple candidates after strict matching → ambiguity error
listing all candidates with mtime, no silent pick.
35/35 existing tests still pass. Real-data smokes:
- 大 outer 34KB 卡片 _parse_app_message_outer 解析成功
- decode_record_item(local_id, ts) 正确命中 Lec 4 PDF
- 多分片冲突 + 不传 ts → 报歧义错误并提示加 create_time
- 未下载 dataitem → 精确指引"在 wechat 点第 N 项"
* fix: address Codex round-3 adversarial high+medium findings
Round-3 caught two more cross-tool inconsistency issues, both in the
same family I keep missing (修一处忘另一处):
[high] decode_file_message also needs to fail-closed on ambiguity
Round-2 high #2 forced decode_record_item to fail-closed when
multiple cached candidates remain after strict matching, but I
forgot to apply the same change to decode_file_message — it still
silently sorted by mtime and returned candidates[0]. Same root
cause as round-1 high #1: Codex catches what I miss when the same
pattern needs fixing in two places.
Now decode_file_message: strict size filter when totallen is known,
and ambiguity error (not mtime sort) when more than one candidate
remains. Behavioral change: previously returned 逻辑审计论文(1).pdf
on a real test case; now reports both candidates and asks user to
disambiguate. UX regression but safety-correct.
[medium] decode_record_item rejects non-downloadable datatypes
upfront. Previously, dataitems with unknown datatype fell through
to a wildcard `sub='*'` glob over all attach subdirs (F/Img/V/A),
which could match unrelated files for links/locations/cards/
miniapps/nested-record dataitems that have only metadata, no
binary payload. Now reject non-{2,4,5,8} datatypes with a clear
"no local binary, look at history output instead" message before
any filesystem lookup.
35/35 existing tests still pass.
* fix: address Codex adversarial round-4 high findings
Round-4 found three security/correctness issues. All addressed:
[high] Path traversal via untrusted XML titles
title (decode_file_message) and datatitle (decode_record_item) come
from message XML — attacker-controlled in the "malicious chat
partner" threat model. glob.escape does NOT strip path separators
or normalize absolute paths, so e.g. title="/etc/passwd" makes
os.path.join(month_dir, "/etc/passwd") == "/etc/passwd" (POSIX
rule: join drops left when right is absolute), and glob then walks
outside msg/file. If size also matches, the tool returns an
arbitrary system path as a "found wechat file".
Added _safe_basename(name) helper with strict-reject semantics
(per Codex: reject, don't normalize) — any name containing path
separators, .. components, NUL, or absolute-path prefix is
rejected outright. Both decoders sanitize their XML-derived names
before any filesystem operation. Added _path_under_root realpath
check after candidate selection as a second-line defense against
symlink escapes.
[high] decode_file_message and decode_record_item can return cached
files belonging to a DIFFERENT message even when len(candidates)==1
Both tools rely on (filename + size + optional item_index)
heuristic matching against the cache — they have no way to derive
a record-bound or message-bound path from wechat metadata, so
exactly one matching cached file from an unrelated message looks
identical to a correct hit. This is a design limitation: wechat
does not expose record_hash or attach-uuid in the message XML in
any form derivable from outside the client.
Acknowledged in tool output with an explicit ⚠️ "this path is
heuristic, please verify mtime/context/content" warning attached
to every "found local file" response. The match itself is still
the same heuristic — closing this fully would require either
removing the tools or reverse-engineering wechat's path hashing.
Documented the limitation in the warning so callers can manually
verify before trusting downstream Read/PDF results.
35/35 tests still pass; 12/12 path-sanitize edge cases pass.
* fix: address Codex round-5 adversarial findings + md5-strong binding
Codex round 5 caught two more high issues plus a perf/correctness
concern. All real and addressed:
[high] decode_file_message scanned msg/attach in fallback, picking
up unrelated forwarded-record cached files. Outer files only ever
live in msg/file/{YYYY-MM}/; restricted the slow-path walk to that
subtree only. msg/attach holds merged-card and image attachments
whose presence here is a different message's payload, not ours.
[high] **真正根治** record/file 路径绑定问题：用 md5 强校验
Both decode_file_message (`<md5>` in appmsg) and decode_record_item
(`<fullmd5>` in dataitem) now extract the WeChat-supplied md5 and
hash candidate files locally to compare. If md5 doesn't match, the
tool fails closed with an explicit md5-mismatch error rather than
returning a path. The candidate that *does* match is uniquely
bound to the selected message — md5 collisions of distinct files
are cryptographically negligible. This fixes the heuristic-only
warning paths from rounds 3-4 with cryptographic evidence rather
than just user-facing notes.
As a side benefit, md5 dedup also lets decode_file_message return
a result when WeChat creates "(1)/(2)" copies of the same file:
same-md5 candidates are真同一文件副本 (user re-sent or auto-rename),
any one of them is correct.
When XML doesn't ship md5 (rare but possible), behavior reverts to
the previous fail-closed-on-multiple-candidates path with an
explicit "no md5 available, treating as heuristic" note.
[medium] _parse_app_message_outer was retrying every appmsg under
the 500K cap on initial 20K rejection, which made history rendering
O(content_size) on big non-record appmsgs. Added a substring
`<type>19</type>` short-circuit so only true type=19 records pay
the wider parser cost. Verified non-type=19 big XML now returns
None in <0.01ms instead of doing a 500K parse.
35/35 existing tests still pass. Real-data smokes:
- decode_record_item 142,1 → "✅ md5 校验通过，路径与 dataitem 唯一绑定"
- decode_file_message 171 (with same-name (1).pdf copy in cache) →
md5 dedup recognizes both as same content, returns one with
"✅ md5 校验通过"
- non-type=19 big appmsg parses in <1ms (substring short-circuit)
* fix: address Codex round-6 adversarial findings — strict md5 binding + chunked hash
[high] decode_file_message / decode_record_item now fail-closed when
the message XML has no md5/fullmd5 field — instead of returning a
heuristic single-candidate path with a warning. The previous
warning-only approach (rounds 4-5) didn't actually stop downstream
Read/PDF callers from using the wrong path. Now: no md5 = no path
returned, period. The error message lists the heuristic candidates
with mtime so the user can manually pick if absolutely needed,
but the tool itself does not commit to any of them.
Behavioral consequence: messages where wechat omits md5 (rare but
possible — e.g. some image/voice dataitems lack fullmd5) become
not-resolvable via these tools. Acceptable safety/utility tradeoff
per Codex's recommendation.
[medium] md5 verification was reading the entire candidate file into
memory via `_hashlib.md5(_f.read()).hexdigest()`. For 100MB+
attachments (videos in merged-record cards, large PDFs) this could
spike RSS or stall the MCP process. Replaced with a streaming
helper `_md5_file_chunked` (64KB chunks) plus a 500MB hard cap that
returns an explicit error rather than attempting verification on
oversized files.
35/35 existing tests still pass. Real-data smokes:
- decode_file_message 171 (with md5) → "✅ md5 校验通过"
- decode_record_item 142,1 (with fullmd5) → "✅ md5 校验通过"
- _md5_file_chunked size cap 1KB rejection works correctly
* fix: round-7 + revert round-6 over-strict — match real threat model
Two real bugs from Codex round-7 plus a partial revert of round-6
over-strictness that doesn't match this tool's actual threat model.
[high] Group type=19 with 'sender:<?xml...' (no newline) prefix not
stripped (Codex round-7 high #1)
_parse_message_content only split on ':\n', missing real-world
group rows where wechat writes 'wxid_xxx:<?xml ...' or
'wxid_xxx:<msg ...' inline. _format_app_message_text and
decode_record_item both received the prefixed content, parsed it
as raw XML, and failed. Now also strips on regex match against
'<?xml|<msg|<msglist|<voipmsg|<sysmsg' immediately after a sender
token. Verified with 5 prefix shape variants; legacy ':\n' still
works.
[medium] Record images use flat 'Img/0_t' filenames, not 'Img/0/*'
(Codex round-7 medium #2)
decode_record_item's datatype=2 (image) branch globbed for
'*/Rec/*/Img/{idx}/*' but real wechat caches store record images
as flat files: '*/Rec/<id>/Img/0_t', '*/Rec/<id>/Img/0', or
'*/Rec/<id>/Img/0.{ext}'. Added flat-pattern matching for
datatype=2 with the four observed filename shapes. File/voice/
video classes still use the F|A|V/{idx}/{filename} shape they
always did.
[revert] Round-6's "no md5 → fail-closed" is too strict for this
tool's actual usage
This MCP server is invoked locally by the user, paths surface
only in the local Claude conversation, and contacts are not
hostile. Codex round-6's hard fail-closed-on-missing-md5 broke
ergonomics for real wechat messages that lack md5 (some image
and voice dataitems) without a corresponding security gain in
this scenario. Reverted to round-5 behavior:
- md5 present → cryptographic verification, mismatch fails
- md5 absent → heuristic + ⚠️ warning, multiple-candidate
ambiguity still fails closed.
Kept all other round-6 hardening: streaming chunked md5, 500MB
cap, _safe_basename strict reject, _path_under_root realpath
check, multi-shard ambiguity, substring short-circuit for
non-type=19 big XML.
35/35 existing tests still pass; 5/5 group-prefix variants pass;
real-data smokes for both decoders still hit md5-verified paths.
* fix: round-8 — defer ambiguity until after md5 dedup + tighten file fallback
Two more findings, both real:
[high] decode_file_message no-md5 fallback was using `stem in f`
substring matching — `stem='论文'` would happily accept
`某老师论文.pdf`. Tightened to: exact match OR strict `(N)` copy
variant (`xxx(1).pdf`, `xxx (1).pdf`) per wechat's auto-rename
convention. 7/7 unit cases verify legitimate accept and false-
positive reject behavior.
[medium/P1 from GitHub Codex] decode_record_item had a stale early
`len(candidates) > 1 → ambiguity` check left over from round-7
refactor — it ran BEFORE the fullmd5 filter, making the md5
disambiguation block unreachable for the exact case where md5
could safely pick the right file. Removed the early check; md5
filter now runs first (and the post-md5 ambiguity check at line
~2467 still fails closed when md5 is missing AND multi-candidate).
35/35 tests pass. Real-data smokes (decode_file_message and
decode_record_item with their corresponding md5/fullmd5) still hit
the ✅ md5-verified path.
* test: add 29 helper-level regression tests for record-decoder helpers
Locks in the bugs fixed across PR #65's many review rounds so they
don't silently regress:
- _safe_basename (7 cases): strict reject of absolute paths,
parent-dir components, path separators, NUL — round-4 high #1.
- _md5_file_chunked (3 cases): streaming hash equals stdlib hashlib,
size cap rejects oversized files, missing file → error — round-6.
- _parse_message_content (5 cases): both legacy `:\n` and round-7
`:<?xml`/`:<msg` group-prefix shapes strip correctly; private
chat does not strip; bytes content returns the binary marker.
- _parse_app_message_outer (3 cases): small XML uses default cap,
oversized non-type=19 short-circuits (no 500K parse), oversized
type=19 retries successfully — round-5 medium #3 + round-2 P2-1.
- _format_record_dataitem (7 cases): text / file / image / 视频号 /
音乐 fall-through render correctly; unknown datatype falls back
to datadesc or [未知类型 N].
- _format_record_message_text (4 cases): >20KB outer XML expands
via _format_app_message_text end-to-end (regression for the
"P2-1 was a fake fix because I tested helper in isolation" miss);
empty datalist shows 待加载; chatroom marker appended; overflow
produces "…还有 N 条未显示" line.
The two MCP-tool wrappers (decode_file_message / decode_record_item)
lean on module globals + the real wechat cache layout. They are
exercised by real-data smoke runs in the PR description rather than
mocked here — mocking the entire wechat tree would dwarf the actual
logic under test.
64/64 total tests pass (35 existing + 29 new).
* refactor: simplify per /simplify code review (no behavior change)
Three review agents (reuse / quality / efficiency) flagged the
following high-confidence cleanups. All applied; all 64 tests still
pass; real-data smokes still hit md5-verified paths.
[quality] Remove PR-history references in comments
CLAUDE.md is explicit about this: comments should explain
non-obvious WHY, not narrate which Codex round caught what.
Cleared "round-2 high #2", "Codex round-3 medium #1", "round-5",
"round-6 强制", "round-7 实测", "round-8 high #1" from helper
docstrings and inline comments. Kept the substantive WHY (e.g.
"Reject 而不是 normalize because intent is suspicious").
[reuse + quality] Module-level datatype constants
Three places maintained their own copy of the datatype → label /
subdir mapping (_format_record_dataitem if-cascade, decode_record_
item type_label dict, subdir_map literal). Extracted
_RECORD_DATATYPE_LABEL and _RECORD_BINARY_SUBDIR to module top.
Single source of truth.
[quality] Hoist local imports to module top
Removed 7 inline `import glob as glob_mod` / `from datetime import
datetime as _dt` / `from datetime import datetime as _dt, timedelta
as _td` / `import hashlib as _hashlib` calls inside hot paths and
helpers. Aliases collapsed to plain names (datetime, timedelta,
glob, hashlib).
[efficiency] xpath: drop `.//` recursive descent for known-direct children
_format_record_dataitem was using `.//appbranditem/sourcedisplayname`
and `.//finderFeed/desc` even though both are direct children of
the dataitem. Changed to direct-child paths — meaningful for big
cards (50 items × subtree-walk per render).
[efficiency] md5 verification short-circuits on first match
Multiple candidates sharing the same md5 are wechat re-named copies
of the same file (e.g. `xxx (1).pdf`); any one is correct. Added
`break` after the first md5 match to skip hashing remaining
candidates (which can each be 100+ MB).
[quality] Compress _format_record_dataitem if-cascade
Datatypes that just emit `[label]` (2/3/4/5/7/23/37) and the link/
H5 pair (6/36) collapsed into membership checks against
_RECORD_DATATYPE_LABEL.
64/64 tests still pass.
---------
Co-authored-by: jiangbowen <robin@jiangbowendeMacBook-Air.local>
Introduce SNS (朋友圈) and batch image tooling and enhance the GUI export workflow. Added new scripts: decrypt_sns.py (decrypt WeChat SNS cache) and batch_decrypt_images.py (bulk .dat image decrypt). Update build scripts and PyInstaller spec to include the new files. app_gui.py: add contact discovery, contact selection/export options dialog, new buttons (find image key, SNS), orchestrate combined export/voice/SNS tasks via subprocesses and env flags, and auto-run export after decryption. config.py: add output_base_dir, auto-detect WeChat Files path, and expose msgattach/xwechat cache dirs. export_messages.py: switch to output_base_dir, add image resource lookup and .dat locating/decryption helpers, and support environment-driven contact/format/image filters. Misc: update build.bat and packaging datas to include the new modules.
Co-Authored-By: Copilot <198982749+Copilot@users.noreply.github.com>
