_load_contacts_from() loaded all rows from the contact table without
filtering, causing ~8000+ chat room member records (local_type=3) to
appear as contacts. This inflated the contact count and polluted search
results. Add WHERE local_type != 3 to exclude chat room members.
## Problem
In `--web` mode, messages longer than ~90 chars get truncated in the
SSE feed. `SessionMonitor.check_updates()` pushes `SessionTable.summary`
to clients, but `summary` is WeChat's own ~80-char preview kept for the
client-side chat list — not the full message body.
## Fix
`_lookup_latest_local_id` already hits `Msg_<md5(username)>` for the
row at `(username, create_time)` to obtain `local_id` for #79's
dedup. Extend the same query to also return `message_content` and
`WCDB_CT_message_content`, and use it to replace `summary` when the
DB body is longer:
SELECT local_id, message_content, WCDB_CT_message_content
FROM [Msg_<md5>]
WHERE create_time = ?
ORDER BY local_id DESC LIMIT 1
Same row, same query — zero additional IO vs. the prior `MAX(local_id)`.
Renamed to `_lookup_latest_message` to reflect the new
`(local_id, content)` return shape. zstd handling and `wxid_xxx:\n`
group-prefix stripping mirror the existing `SessionTable.summary`
logic in `check_updates`, so the SSE `content` field stays in the same
format clients already render.
Replacement is conservative — only swaps in `full_content` when
strictly longer than `summary`. This never shortens existing behavior
and degrades cleanly if the message-DB write hasn't landed yet (the
SessionTable-vs-message_N.db timing race that #79 already documented).
## Scope
- `monitor_web.py`: one helper extended + one call site adjusted. No
schema change, no new dependency, no client/UI change.
- `_check_hidden_messages` cold path is untouched — its same-second
multi-message coverage still runs as before.
Three new flags for export_all_chats.py:
- -i / --incremental: reads existing JSON, appends only new messages
(deduplicates by local_id, preserves transcription field on merge)
- --start / --end: filter messages by date range (YYYY-MM-DD or timestamp)
passes start_ts/end_ts directly to mcp_server._query_messages
- --dry-run: preview mode (shows counts without writing files)
Voice transcription in incremental mode only processes newly appended
voice messages — existing transcribed entries are untouched.
## Problem
Voice messages in `_format_message_text` previously rendered as a bare
`[语音] (local_id=N, ts=T)` because msg_type=34 fell through to the generic
non-text branch with no schema-aware summarizer. LLMs reading chat history
had no way to judge whether a voice clip was worth calling `decode_voice`
on without first inspecting it.
## Fix
New helper `_format_voice_text(content)` parses the embedded
`<voicemsg voicelength="…">` and renders `[语音 N.Ns]` (duration to one
decimal, milliseconds → seconds). Type=34 dispatches through it, then
appends the existing `_id_suffix()` so the local_id annotation is
preserved end-to-end:
[语音 3.3s] (local_id=72481, ts=1700000000)
Falls back to `[语音]` (still with `_id_suffix()`) when content is empty,
`<voicemsg>` is absent, XML parse fails, or `voicelength` is missing /
zero / non-numeric.
XML parsing routes through the existing `_parse_xml_root` so the
`_XML_UNSAFE_RE` DOCTYPE/ENTITY filter and 200KB size cap are reused —
no new XXE surface.
## Tests
12 new cases in `tests/test_voice_format.py`: happy path, subsecond,
multi-second, missing / zero / non-numeric voicelength, empty / None
content, missing `<voicemsg>` tag, malformed XML, XXE payload, and two
end-to-end cases through `_format_message_text` (with and without
voicelength) to pin the full rendered output including `_id_suffix()`.
Baseline 183 → 195 passing, 0 regressions.
## Scope
- `mcp_server.py`: adds `_format_voice_text` helper and one branch in
`_format_message_text` (base_type == 34). No public surface change —
this only affects formatting of messages that previously rendered as
the bare `[语音]` fallback.
- `tests/test_voice_format.py`: new file, synthetic fixtures only (no
real PII).
## Problem
`_contact_names`, `_contact_full`, `_contact_tags`, and `_self_username`
are populated lazily on first access and never invalidated for the
process lifetime. When `contact.db` is re-decrypted (new contact added,
remark or group name edited, etc.) the on-disk DB updates but the
running MCP server keeps serving stale data — newly-added contacts are
invisible to `resolve_username` and downstream tools until the server
is restarted.
## Fix
Track the mtime of the contact.db backing file. On every
`_get_contact_db_path()` call (which all contact accessors go through),
compare against `_contact_db_mtime`; if it changed, clear all four
caches and record the new mtime. Lookups that don't trigger a real
re-decryption pay only one `os.path.getmtime()` syscall.
The function is reorganized so `_get_contact_db_path()` is the single
source of truth for both "where is contact.db" and "do we need to
invalidate" — `get_contact_names` and `_load_contact_tags` consult it
unconditionally before the early-return on the populated cache.
Also reorders `_get_self_username` to call `get_contact_names()` first
(which now triggers the mtime check via the path lookup) before
returning a cached `_self_username` — otherwise the rename case would
still resolve to the stale name.
## Tests
Baseline 183 → 183 passing, 0 regressions.
The pattern (mtime-track + invalidate-on-change) mirrors the existing
behaviour of DBCache, which already re-decrypts contact.db when the
source mtime changes; this fix closes the symmetric gap on the
in-memory side.
## Scope
- `mcp_server.py` only.
- No public surface change. Affects the contact-cache layer's behaviour
on re-decryption — previously: stale until restart; now: refreshed
on next contact-related call.
Add structured parsing for transfer messages so they no longer fall
through to the generic `[链接/文件]` fallback in chat history exports.
Mirrors the dispatch + helper pattern PR #65 (merged-forward type=19)
established for `base_type=49` appmsg sub-types.
## What is added
**Helpers (mcp_server.py):**
- `_TRANSFER_PAYSUBTYPE_LABEL` — maps the 6 community-consensus paysubtypes
(1 发起 / 3 已收款 / 4 已退还 / 5 过期已退还 / 7 待领取 / 8 已领取);
unknown values degrade to `未知(paysubtype=N)` so a new variant in a
future WeChat build is visible rather than silently dropped.
- `_extract_transfer_info(appmsg)` — pulls fields out of `<wcpayinfo>`,
with snake/camelCase fallback (`feedesc`/`feeDesc`, `pay_memo`/`paymemo`)
observed across WeChat versions.
- `_format_transfer_message_text(appmsg, title)` — one-line render
for chat history: `[转账·已收款] ¥100.00 备注: lunch`.
**Dispatch (mcp_server.py):**
- `_format_app_message_text` gains an `app_type == 2000` branch that
routes to `_format_transfer_message_text`. `get_chat_history`,
`export_chat`, `export_all_chats` and `monitor_web` all inherit
automatically.
**New MCP tool (mcp_server.py):**
- `decode_transfer(chat_name, local_id, create_time=0)` — full
structured view: direction, amount, memo, payer/receiver wxid,
transfer id, transcation id, begin/invalid timestamps. Uses the
same multi-shard scan + ambiguity-by-create_time pattern as
`decode_file_message` / `decode_record_item`.
**CLI wrapper:**
- `decode_transfer.py` at the repo root — argparse wrapper that prints
the same text as the MCP tool, returning non-zero exit when the
message can't be decoded (script-friendly).
**JSON export (chat_export_helpers.py + export_chat.py + export_all_chats.py):**
- `_extract_content` now returns `(rendered, extras)`. `extras` carries
structured fields when a message type has more signal than the
human-readable string (currently: transfers → `type:"transfer" +
transfer:{direction, fee_desc, pay_memo, ...}`). The channel is
forward-compatible — future additions (video号 metadata, expanded
merged-forward, etc.) flow through the same shape without changing
the caller signature. JSON consumers that only read `content` are
unaffected; the change is additive.
**monitor_web (monitor_web.py):**
- Backend dispatch branch + orange-yellow `.msg-transfer` card CSS +
`renderRich` JS handler.
## Tests
12 new cases in `tests/test_record_decoders.py`:
- `TransferPaysubTypeLabelTests` — locks the 6-value label table.
- `ExtractTransferInfoTests` (6 cases) — full field round-trip, missing
`<wcpayinfo>` fallback, snake/camelCase variants, unknown paysubtype
degradation, empty paysubtype handling.
- `FormatTransferMessageTextTests` (4 cases) — initiate / received-with-memo /
missing-wcpayinfo / missing-fee-desc.
- `AppMessageDispatchTransferTests` — `_format_app_message_text` routes
type=2000 correctly so `get_chat_history` / `export_chat` both pick
it up.
All fixtures use synthetic placeholder values (`wxid_payer_synth`,
`¥100.00`, `1` + 27×`0`); no real PII or transaction IDs.
## Scope
7 files, +546 / -15 (additions only — no behavior change for existing
message types). All 180 tests pass locally (168 baseline + 12 new).
The post-decrypt verification step (sqlite3.connect(out_path) + table list, around line 163) opens the freshly-written .db in default journal mode. Even though the connection is closed cleanly, SQLite leaves behind empty <db>-shm and <db>-wal companion files in OUT_DIR.
Downstream tools that later open the same .db will see those companion files and try to roll the (empty / stale) WAL forward, producing "database disk image is malformed" or silently masking the most recent pages. The decrypted DB itself is fine — the residuals are pure noise from the verification connection.
Fix: after the verification block (success or failure), unconditionally os.remove() out_path + "-shm" and out_path + "-wal" if present. Errors during cleanup are swallowed.
Tests: existing tests/ pass (168/168). The cleanup is additive and only runs after the existing verification path; no behavior change for callers that do not inspect OUT_DIR for companion files.
Scope: 10 lines in decrypt_db.py. No public API change, no schema change, no new dependency.
## Problem
On macOS, the default `db_dir` template in `config.py` (line 20) points to
`~/Documents/xwechat_files/your_wxid/db_storage`, but WeChat 4.x on macOS
stores data inside the app sandbox container at
`~/Library/Containers/com.tencent.xinWeChat/Data/Documents/xwechat_files/<wxid>/db_storage`.
`_auto_detect_db_dir_macos()` (config.py:166) handles the common case, but
when auto-detect fails — fresh install with no scan results yet, permission
issues, atypical install location — the template fallback is what the user
sees in their generated `config.json`. Today that fallback is a Linux-style
path that does not exist on macOS, so the user has to manually correct it
before the first run can succeed.
## Fix
Update the darwin branch of `_DEFAULT_TEMPLATE_DIR` to the actual sandbox
container path. `your_wxid` remains a placeholder.
Linux and Windows templates are unchanged.
## Tests
Existing `tests/` pass (168 / 168). The change only affects a module-level
constant; no behavior change for users whose auto-detect already succeeds.
## Scope
3 lines in `config.py`. No public API change, no schema change, no
dependency change.
* fix(image): scope local_id lookup by chat_id + use real column name
`ImageResolver.get_image_md5` made two wrong assumptions about the
production `MessageResourceInfo` schema, which made the decode_image
MCP tool always fail with "无法找到 local_id=X 的图片信息":
1. The column is `message_local_id`, not `local_id`. The current query
throws `sqlite3.OperationalError: no such column: local_id`, but the
exception is swallowed by `except Exception: pass`, masking the real
failure as a silent miss.
2. `message_local_id` is not globally unique. In production it repeats
across chats, and within an active chat the same local_id can recur
up to 7 times (observed on a real DB). The production schema scopes
by `chat_id`, resolved from `ChatName2Id.rowid WHERE user_name = ?`.
Fix: `get_image_md5` now takes `(username, local_id)`:
- Resolve `username -> chat_id` via `ChatName2Id`.
- Query `MessageResourceInfo` filtered by `chat_id + message_local_id
+ message_local_type == 3` (image type; high bits are session flags,
so use `% 2^32`), ordered by `message_create_time DESC LIMIT 1`.
External callers (`mcp_server.decode_image_tool` /
`list_chat_images_tool`) already pass `username` through
`ImageResolver.decode_image()` / `list_chat_images()`, so the public
API is unchanged. Only the internal helper signature shifts.
The existing test fixture in `test_decode_image_v2` used the same wrong
schema as the buggy code (`CREATE TABLE MessageResourceInfo (local_id
INTEGER PRIMARY KEY, packed_info BLOB)`), so the tests passed against a
self-consistent fiction. The fixture is rebuilt to match real columns
plus `ChatName2Id`, and three regression tests are added:
- cross-chat collision (same local_id in 3 chats; must pick the right one
and not the type=43 video row)
- same-chat reuse (same local_id, two timestamps; must pick the newer)
- unknown chat (username not in ChatName2Id; structured error, no crash)
All 154 tests pass locally (151 baseline + 3 new).
* fix(image): surface get_image_md5 errors and use read-only DB open
Two follow-ups on top of the chat_id scoping fix:
1. The bare `except Exception: pass` was the original failure mode: it
silently swallowed `OperationalError: no such column: local_id` when
the production schema diverged from the old `local_id` column name,
masking the bug this PR fixes. Print the exception to stderr so
future schema drift surfaces immediately instead of returning a
misleading "image not found" error.
2. Open message_resource.db with `file:...?mode=ro` URI to match the
rest of the project (monitor_web.py uses this idiom in 9 places).
The DB is read-only for our purposes and a running WeChat may still
hold it; using URI ro avoids any chance of lock contention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WeChat rolls a chat's messages over to the next `message_N.db` shard
once the current shard fills up (~100 MB), so any chat older than the
current shard window has its history split across multiple shards. The
other message-query tools — `get_chat_history`, `search_messages`, and
`decode_image` — already iterate every matching shard via the plural
helper `_find_msg_tables_for_user`. Only `get_chat_images` still used
the singular `_find_msg_table_for_user`, which returns the first shard
that contains the user's table.
Effect: every image that lived in a non-first shard was silently
dropped from `get_chat_images`. On a long-lived chat with many images,
the tool would return only the most recent slice and pretend the rest
did not exist.
Fix: switch `get_chat_images` to `_find_msg_tables_for_user`, fetch
`limit` images per shard, merge, sort by `create_time` DESC, and slice
to `limit`. This mirrors how the other tools fan out across shards.
Tests in `tests/test_get_chat_images_multishard.py`:
- `test_collects_images_from_every_shard` — both shards' images appear
in the output (the regression case)
- `test_global_sort_by_create_time_desc` — newer image from an older
shard still wins, output is globally sorted (not per-shard concat)
- `test_limit_truncates_globally_across_shards` — limit=3 takes the 3
newest overall, not "first shard wins"
- `test_no_shards_returns_not_found` — empty shard list path
- `test_all_shards_empty_returns_no_images` — every shard empty path
All 156 tests pass locally (151 baseline + 5 new). Public tool
signature is unchanged; only the internal scanning loop is widened.
When a chat history contains a name-card message (msg_type=42), the
dispatcher in `_format_message_text` had no case for `base_type == 42`,
so it fell through to the generic non-text branch:
elif base_type != 1:
type_label = format_msg_type(local_type)
text = f"[{type_label}] {text}" if text else f"[{type_label}]"
`text` for type=42 is the full raw `<msg ...>` element, so chat history
exports emitted `[名片] <msg username="..." antispamticket="v2_..."
brandIconUrl="https://wx.qlogo.cn/..." bigheadimgurl="..." ... />`.
That payload has two problems:
1. It leaks anti-spam tokens (`antispamticket`) and head-image CDN URLs
into chat logs that are routinely piped to LLMs and other downstream
tools.
2. The raw XML drowns out the actual signal — a human or an LLM reading
the chat just wants to know "X shared Y's contact".
This PR adds `_format_namecard_text(content)` that pulls only the three
useful attributes:
- `nickname` — display name
- `username` — wxid (annotated as "公众号" when prefixed `gh_`)
- `certinfo` — user-authored bio
and wires it into the dispatch chain via a new `elif base_type == 42:`
branch, sitting alongside the existing `49` (app message) handler. It
reuses `_parse_xml_root` and `_collapse_text` — no new helpers
introduced.
Tests: 7 cases in `tests/test_namecard_format.py` covering the realistic
shape (with antispamticket / brand URLs that must NOT appear in output),
official accounts (`gh_*`), missing certinfo, missing nickname, missing
both identifiers, and broken-XML fallthrough.
All 158 tests pass locally (151 baseline + 7 new).
`v2_decrypt_file` previously wrote files to disk even when the keys were
wrong, producing garbage output with no way for the caller to detect the
failure:
1. Wrong AES key -> `detect_image_format` returns 'bin' (magic does not
match any known format) -> a `.bin` file of random bytes was written.
2. Wrong XOR key with correct AES key -> file header looks like a valid
jpg/png (the AES segment decrypts correctly) but the trailing XOR
segment is scrambled -> callers get a half-valid image file that
image viewers either render as truncated or fail to open.
Both cases now return `(None, None)`:
- `fmt == 'bin'` -> fail fast, no file written.
- `xor_size >= 2` -> validate trailer magic by format:
* jpg must end with FF D9 (EOI marker)
* png must contain IEND chunk in the last 12 bytes
Other formats (gif/bmp/tif/webp/hevc/wxgf) lack a mandatory trailer
signature, so they skip the check to avoid false rejection.
Also fixes a latent bug in `test_decode_image_v1_no_aes_key_uses_fixed_key`:
the test built the synthetic .dat with `TEST_XOR_KEY=0x37` but constructed
`ImageResolver` without `xor_key=`, defaulting to `0x88`. The XOR segment
was always scrambled — the test passed because the AES segment alone was
enough for `detect_image_format` to return 'png' from the header, and no
trailer validation existed to catch the corruption. The new trailer check
surfaces this, so the test now passes `xor_key=TEST_XOR_KEY` explicitly.
Tests: 5 new cases (wrong AES key / wrong XOR for jpg / wrong XOR for
png / xor_size=0 bypass / wxgf bypass). All 156 existing tests still pass.
