zikai/zWorkFlow
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(export): 增加微信聊天增量 JSON 导出 (#122)

Browse Source 
新增 export_all_chats --delta-only 按时间窗口导出每个会话的增量 JSON.

- --delta-only 写入 deltas/<run_id>/manifest.json + 有消息的会话文件, 便于后续按批次消费
- 沿用现有 CSV plan 选择逻辑, 可通过 --from-plan-csv 控制范围
- 空时间窗口会话直接跳过, 不再生成 message_count=0 的 delta 文件
- 增量 JSON 保留会话级 metadata, 不改变现有 CSV 导出默认流程
- manifest.json 只记录实际有消息的会话, 无法解析的留在 errors 中便于定位

测试: tests/test_export_all_chats_delta.py 7 个 case (run_id 时间戳形状 / 文件名安全化 / msg_uid 哈希组成 / 写窗口不覆盖全量 JSON / 空窗口跳过 / CLI 参数校验 / manifest 记录文件).

跟 #114 (CSV plan) 互补 — CSV plan 选 "哪些会话" 而 delta 选 "哪个时间窗口".
This commit is contained in:
phoenixray2000
2026-05-29 13:18:53 +08:00
committed by GitHub
parent eba9a9d4cd
commit 5e0eaa33fa
5 changed files with 559 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
EXE_USAGE.md
View File

@@ -16,9 +16,14 @@
.\WeChatDecrypt.exe status
.\WeChatDecrypt.exe decrypt
.\WeChatDecrypt.exe export --from-plan-csv export_plan.csv
.\WeChatDecrypt.exe export-all output_dir --delta-only --start "2026-05-26 00:00:00" --end "2026-05-26 08:00:00"
.\WeChatDecrypt.exe export-all --write-plan-csv export_plan.csv
```
`--delta-only` 只写入时间窗口内的 delta JSON，不读取或覆盖已有完整 JSON。
输出里每条消息包含 `msg_uid`，下游可以用它做去重；时间窗口内没有消息的会话会跳过，
不生成空的 `*.delta.json` 文件。
无参数或 `web` 子命令仍启动 Web UI:
```powershell
@@ -99,6 +104,7 @@ wechat_files/<wxid>/ ← 导出的聊天记录 (按 wxid + 联系人分
朋友圈图片/ ← 朋友圈缓存图片解密后
data/ ← 语音转 MP3 输出 (有的话)
exported_chats/ ← 用 ③ 导出全部聊天 (JSON) 时的输出 (export_all_chats.py)
deltas/<run_id>/ ← --delta-only 输出 manifest.json + 非空 chats/*.delta.json
wxwork_export/ ← 企微聊天导出
```

7
README.md
View File

@@ -222,6 +222,7 @@ py -m pip install --user -r requirements.txt
| 生成导出计划 CSV（白名单） | `python export_all_chats.py --write-plan-csv export_plan.csv --plan-mode whitelist` |
| 按计划 CSV 导出（黑名单，默认） | `python export_all_chats.py output_dir --from-plan-csv export_plan.csv` |
| 按计划 CSV 导出（白名单） | `python export_all_chats.py output_dir --from-plan-csv export_plan.csv --plan-mode whitelist` |
| 导出时间窗口 delta JSON（不覆盖完整 JSON） | `python export_all_chats.py output_dir --delta-only --start "2026-05-26 00:00:00" --end "2026-05-26 08:00:00"` |
| 批量导出 + 语音转录 | `python export_all_chats.py --with-transcriptions` |
| 转录单个文件语音 | `python transcribe_chat.py input.json [output.json]` |
| 注册 MCP Server（Claude） | `claude mcp add wechat -- python /path/to/mcp_server.py` |
@@ -235,6 +236,10 @@ py -m pip install --user -r requirements.txt
的行会被跳过，空值、`1` 或没有 `export` 列都会导出；`whitelist`
模式下只有明确 `export=1` 的行会导出。
`--delta-only` 是无状态时间窗口导出，必须显式传 `--start`。它不会读取或
覆盖 `exported_chats/*.json`，适合由外部调度器维护游标后定时拉取新增消息。
时间窗口内没有消息的会话会跳过，不生成空的 `*.delta.json` 文件。
### Web UI
`python main.py` 启动后打开 http://localhost:5678 查看实时消息流。
@@ -398,7 +403,7 @@ make help # 列出所有命令
| 文件 | 说明 |
|---|---|
| `export_all_chats.py` | 批量导出全部聊天为 JSON (含 CSV 计划选择、`-t` 转录、`-i` 增量、日期范围、`--dry-run`) |
| `export_all_chats.py` | 批量导出全部聊天为 JSON (含 CSV 计划选择、`--delta-only` 时间窗口、`-t` 转录、`-i` 增量、日期范围、`--dry-run`) |
| `export_chat.py` | 单会话 JSON 导出 (供 `export_all_chats` 调用) |
| `chat_export_helpers.py` | JSON 导出共享格式化函数 (避免漂移) |
| `export_messages.py` | CSV / HTML / JSON 三种格式导出, 图片可内联 (PR #107) |

21
docs/chat_export_format.md
View File

@@ -110,3 +110,24 @@ pending = [m for m in data["messages"]
`[图片]` 等。原始媒体仍在 WeChat DB 中，可用 `mcp_server.py` 中的
辅助函数（`decode_image`、`decode_voice`）取出。
- **群聊**中的 `sender` 是群成员解析后的显示名；当前登录用户仍为 `"me"`。
## Delta JSON 输出
`export_all_chats.py --delta-only --start <time> [--end <time>]` 会写入：
```text
deltas/<run_id>/manifest.json
deltas/<run_id>/chats/*.delta.json
```
时间窗口内没有消息的会话会跳过，不生成空的 `*.delta.json` 文件。
Delta 文件顶层字段与完整聊天 JSON 类似，但包含：
- `schema_version`
- `export_kind: "wechat_delta"`
- `range.start` / `range.end`
- `message_count`
- `messages[].msg_uid`
`msg_uid` 是下游去重用的稳定哈希，不要求调用方只依赖 `local_id`。

300
export_all_chats.py
View File

@@ -30,6 +30,7 @@ import sys
import time
from contextlib import closing
from datetime import datetime
from pathlib import Path
import mcp_server
@@ -59,6 +60,7 @@ PLAN_CSV_FIELDS = [
EXPORT_INDEX_FILE = "_export_index.json"
EXPORT_INDEX_VERSION = 1
DELTA_SCHEMA_VERSION = 1
PLAN_MODE_BLACKLIST = "blacklist"
PLAN_MODE_WHITELIST = "whitelist"
_UNSAFE_FILENAME_RE = re.compile(r'[\\/:*?"<>|]')
@@ -116,6 +118,33 @@ def _safe_export_filename_part(value):
return cleaned or "unknown"
def _delta_run_id(now_ts=None):
ts = int(now_ts if now_ts is not None else time.time())
return datetime.fromtimestamp(ts).strftime("%Y%m%dT%H%M%S")
def _delta_filename(display_name, is_group, username):
prefix = "group" if is_group else "single"
label = _safe_export_filename_part(display_name or username or "unknown")
user_part = _safe_export_filename_part(username)
return f"{prefix}_{label}__{user_part}.delta.json"
def _content_hash_for_uid(content):
if content is None:
return ""
return hashlib.sha256(str(content).encode("utf-8")).hexdigest()
def _delta_msg_uid(username, db_path, local_id, timestamp, msg_type, content):
db_name = os.path.basename(str(db_path or ""))
payload = (
f"{username}|{db_name}|{int(local_id)}|{int(timestamp)}|"
f"{msg_type or 'text'}|{_content_hash_for_uid(content)}"
)
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
def _export_filename(display_name, is_group, username=None):
prefix = "group" if is_group else "single"
label = display_name or username or "unknown"
@@ -1177,6 +1206,208 @@ def export_one(username, output_dir, names, transcribe=False,
return True, len(messages), new_count, None
def _transcribe_delta_voice_messages(username, messages):
for msg in messages:
if msg.get("type") != "voice" or msg.get("transcription"):
continue
lid = msg["local_id"]
try:
row = mcp_server._fetch_voice_row(username, lid)
if row is None:
continue
voice_data, create_time = row
wav_path, _ = mcp_server._silk_to_wav(
voice_data, create_time, username, lid
)
backend = _resolve_backend()
result = mcp_server._transcribe(wav_path, backend)
if result and result.get("text"):
msg["transcription"] = result["text"]
os.unlink(wav_path)
except Exception:
continue
def export_delta_one(username, delta_root, names, run_id, start_ts,
end_ts=None, transcribe=False):
if start_ts is None:
raise ValueError("delta export requires start_ts")
ctx = mcp_server._resolve_chat_context(username)
if ctx is None:
return {
"success": False,
"username": username,
"message_count": 0,
"reason": f"Cannot resolve: {username}",
}
display_name = ctx["display_name"]
message_tables = ctx["message_tables"]
if not message_tables:
return {
"success": False,
"username": username,
"message_count": 0,
"reason": "no tables",
}
rows_for_delta = []
for table_info in message_tables:
db_path = table_info["db_path"]
table_name = table_info["table_name"]
try:
with closing(sqlite3.connect(db_path)) as conn:
id_to_username = mcp_server._load_name2id_maps(conn)
rows = mcp_server._query_messages(
conn,
table_name,
start_ts=start_ts,
end_ts=end_ts,
limit=None,
oldest_first=True,
)
for row in rows:
rows_for_delta.append((row, table_info, id_to_username))
except Exception as e:
return {
"success": False,
"username": username,
"message_count": 0,
"reason": f"DB query error: {e}",
}
rows_for_delta.sort(key=lambda item: item[0][2] or 0)
messages = []
for row, table_info, id_to_username in rows_for_delta:
local_id, local_type, create_time, real_sender_id, content, ct = row
raw_content = content
sender = _resolve_sender(row, ctx, names, id_to_username)
type_str = _msg_type_str(local_type)
rendered, extras = _extract_content(
local_id, local_type, raw_content, ct, username, display_name
)
effective_type = (extras or {}).get("type") or type_str
msg = {
"msg_uid": _delta_msg_uid(
username=username,
db_path=table_info.get("db_path"),
local_id=local_id,
timestamp=create_time,
msg_type=effective_type,
content=raw_content,
),
"local_id": local_id,
"timestamp": create_time,
"sender": sender,
}
if effective_type != "text":
msg["type"] = effective_type
if rendered is not None:
msg["content"] = rendered
if extras:
for key, value in extras.items():
if key != "type":
msg[key] = value
messages.append(msg)
if not messages:
return {
"success": True,
"skipped": True,
"username": username,
"chat": display_name,
"message_count": 0,
"reason": "no messages in delta window",
}
if transcribe:
_transcribe_delta_voice_messages(username, messages)
delta_rel_path = os.path.join(
"chats",
_delta_filename(display_name, ctx["is_group"], username),
)
delta_path = os.path.join(delta_root, "deltas", run_id, delta_rel_path)
output = {
"schema_version": DELTA_SCHEMA_VERSION,
"export_kind": "wechat_delta",
"chat": display_name,
"username": username,
"exported_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
"range": {
"start": _date_from_message_ts(start_ts),
"end": _date_from_message_ts(end_ts) if end_ts is not None else "",
},
}
if messages:
output["date_first_msg"] = _date_from_message_ts(
messages[0].get("timestamp")
)
output["date_last_msg"] = _date_from_message_ts(
messages[-1].get("timestamp")
)
if ctx["is_group"]:
output["is_group"] = True
else:
output.update(_contact_metadata_for_export(username, ctx["is_group"]))
output["message_count"] = len(messages)
output["messages"] = messages
os.makedirs(os.path.dirname(delta_path), exist_ok=True)
with open(delta_path, "w", encoding="utf-8") as f:
json.dump(output, f, ensure_ascii=False, indent=2)
return {
"success": True,
"username": username,
"chat": display_name,
"path": delta_rel_path.replace("\\", "/"),
"message_count": len(messages),
}
def _write_delta_manifest(delta_root, run_id, start_ts, end_ts,
chats_checked, results):
files = []
errors = []
for result in results:
if result.get("success") and result.get("message_count", 0) > 0:
files.append({
"username": result["username"],
"chat": result.get("chat", result["username"]),
"path": result["path"],
"message_count": result.get("message_count", 0),
})
elif not result.get("success"):
errors.append({
"username": result.get("username", ""),
"reason": result.get("reason", "unknown"),
})
manifest = {
"schema_version": DELTA_SCHEMA_VERSION,
"export_kind": "wechat_delta_run",
"run_id": run_id,
"generated_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
"range": {
"start": _date_from_message_ts(start_ts),
"end": _date_from_message_ts(end_ts) if end_ts is not None else "",
},
"chats_checked": chats_checked,
"chats_with_messages": len(files),
"messages_exported": sum(item["message_count"] for item in files),
"files": files,
"errors": errors,
}
manifest_path = Path(delta_root) / "deltas" / run_id / "manifest.json"
manifest_path.parent.mkdir(parents=True, exist_ok=True)
manifest_path.write_text(
json.dumps(manifest, ensure_ascii=False, indent=2),
encoding="utf-8",
)
return manifest_path
_BACKEND_CACHE = None
@@ -1253,6 +1484,11 @@ def main(argv=None):
action="store_true",
help="增量导出：只追加新消息到已有 JSON 文件",
)
parser.add_argument(
"--delta-only",
action="store_true",
help="只导出 --start/--end 时间窗口内的增量 delta JSON，不读取或覆盖已有完整 JSON",
)
parser.add_argument(
"--start",
default=None,
@@ -1292,6 +1528,8 @@ def main(argv=None):
print(f"错误: 无法解析结束时间: {args.end}", file=sys.stderr)
print("支持格式: 2025-01-01, 2025-01-01 14:30, 2025-01-01T14:30:00", file=sys.stderr)
sys.exit(1)
if args.delta_only and start_ts is None:
parser.error("--delta-only requires --start to avoid accidental full export")
if args.with_transcriptions:
try:
@@ -1330,7 +1568,9 @@ def main(argv=None):
# 显示模式信息
mode = ""
if args.incremental:
if args.delta_only:
mode = "Delta 增量导出"
elif args.incremental:
mode = "增量模式"
if start_ts:
start_dt = datetime.fromtimestamp(start_ts).strftime("%Y-%m-%d %H:%M")
@@ -1401,18 +1641,48 @@ def main(argv=None):
total_new = 0
total_sessions = len(sessions)
run_id = _delta_run_id()
delta_results = []
for i, username in enumerate(sessions, 1):
display = names.get(username, username)
chat_t0 = time.time()
print(f"[{i}/{total_sessions}] 开始导出: {display} ({username})", flush=True)
success, total_msgs, new_msgs, reason = export_one(
username, output_dir, names,
transcribe=args.with_transcriptions,
start_ts=start_ts,
end_ts=end_ts,
incremental=args.incremental,
)
if args.delta_only:
result = export_delta_one(
username=username,
delta_root=output_dir,
names=names,
run_id=run_id,
start_ts=start_ts,
end_ts=end_ts,
transcribe=args.with_transcriptions,
)
delta_results.append(result)
success = result.get("success", False)
total_msgs = result.get("message_count", 0)
new_msgs = total_msgs
reason = result.get("reason")
else:
success, total_msgs, new_msgs, reason = export_one(
username, output_dir, names,
transcribe=args.with_transcriptions,
start_ts=start_ts,
end_ts=end_ts,
incremental=args.incremental,
)
if success:
elapsed = time.time() - t0
chat_elapsed = time.time() - chat_t0
eta = (elapsed / i) * (total_sessions - i) if i > 0 else 0
if args.delta_only and result.get("skipped"):
skip += 1
print(
f"[{i}/{total_sessions}] 跳过: {display} ({reason}) "
f"(本会话 {chat_elapsed:.1f}s, ETA {eta/60:.1f}分)",
flush=True,
)
continue
ok += 1
total += total_msgs
total_new += new_msgs
@@ -1420,9 +1690,6 @@ def main(argv=None):
label = f"+{new_msgs} new" if args.incremental else f"{total_msgs} msgs"
else:
label = f"{total_msgs} msgs"
elapsed = time.time() - t0
chat_elapsed = time.time() - chat_t0
eta = (elapsed / i) * (total_sessions - i) if i > 0 else 0
print(
f"[{i}/{total_sessions}] 完成导出: {display} - {label} "
f"(本会话 {chat_elapsed:.1f}s, ETA {eta/60:.1f}分)",
@@ -1450,6 +1717,17 @@ def main(argv=None):
flush=True,
)
if args.delta_only:
manifest_path = _write_delta_manifest(
delta_root=output_dir,
run_id=run_id,
start_ts=start_ts,
end_ts=end_ts,
chats_checked=total_sessions,
results=delta_results,
)
print(f"Delta manifest: {manifest_path}")
elapsed = time.time() - t0
print()
print("=" * 60)

237
tests/test_export_all_chats_delta.py Normal file
View File

@@ -0,0 +1,237 @@
import hashlib
import io
import json
import os
import sqlite3
import tempfile
import unittest
from contextlib import redirect_stderr
from unittest.mock import patch
import export_all_chats
class DeltaHelperTests(unittest.TestCase):
def test_run_id_uses_timestamp_shape(self):
self.assertRegex(
export_all_chats._delta_run_id(1779724800),
r"\d{8}T\d{6}",
)
def test_filename_contains_safe_display_and_username(self):
filename = export_all_chats._delta_filename(
display_name="张三:/客户",
is_group=False,
username="wxid_abc",
)
self.assertEqual(filename, "single_张三__客户__wxid_abc.delta.json")
def test_msg_uid_includes_username_db_local_id_timestamp_type_and_content(self):
actual = export_all_chats._delta_msg_uid(
username="wxid_abc",
db_path=r"D:\wechat\message_1.db",
local_id=7,
timestamp=1778889601,
msg_type="text",
content="hello",
)
expected = hashlib.sha256((
"wxid_abc|message_1.db|7|1778889601|text|"
+ hashlib.sha256("hello".encode("utf-8")).hexdigest()
).encode("utf-8")).hexdigest()
self.assertEqual(actual, expected)
class ExportDeltaOneTests(unittest.TestCase):
def test_writes_only_window_and_does_not_rewrite_full_json(self):
username = "wxid_delta"
with tempfile.TemporaryDirectory() as tmp:
db_path = os.path.join(tmp, "message_1.db")
table_name = _create_message_db(db_path, username)
full_json = os.path.join(tmp, "single_张三.json")
with open(full_json, "w", encoding="utf-8") as f:
json.dump(
{"chat": "张三", "username": username, "messages": []},
f,
ensure_ascii=False,
)
with open(full_json, encoding="utf-8") as f:
before = f.read()
ctx = {
"username": username,
"display_name": "张三",
"message_tables": [{"db_path": db_path, "table_name": table_name}],
"is_group": False,
}
with patch.object(export_all_chats.mcp_server, "_resolve_chat_context",
return_value=ctx), \
patch.object(export_all_chats, "_resolve_sender",
return_value="me"), \
patch.object(export_all_chats, "_extract_content",
return_value=("rendered in range", None)), \
patch.object(export_all_chats, "_contact_metadata_for_export",
return_value={"contact_remark": "张三"}):
result = export_all_chats.export_delta_one(
username=username,
delta_root=tmp,
names={username: "张三"},
run_id="20260526T080000",
start_ts=1778889600,
end_ts=1778890000,
transcribe=False,
)
self.assertTrue(result["success"])
self.assertEqual(result["message_count"], 1)
with open(full_json, encoding="utf-8") as f:
self.assertEqual(f.read(), before)
delta_path = os.path.join(
tmp,
"deltas",
"20260526T080000",
result["path"],
)
with open(delta_path, encoding="utf-8") as f:
data = json.load(f)
self.assertEqual(data["export_kind"], "wechat_delta")
self.assertEqual(data["username"], username)
self.assertEqual(data["message_count"], 1)
self.assertEqual(data["messages"][0]["content"], "rendered in range")
self.assertEqual(
data["messages"][0]["msg_uid"],
export_all_chats._delta_msg_uid(
username=username,
db_path=db_path,
local_id=2,
timestamp=1778889601,
msg_type="text",
content="in range",
),
)
def test_skips_file_when_window_is_empty(self):
username = "wxid_delta"
with tempfile.TemporaryDirectory() as tmp:
db_path = os.path.join(tmp, "message_1.db")
table_name = _create_message_db(db_path, username)
ctx = {
"username": username,
"display_name": "张三",
"message_tables": [{"db_path": db_path, "table_name": table_name}],
"is_group": False,
}
with patch.object(export_all_chats.mcp_server, "_resolve_chat_context",
return_value=ctx):
result = export_all_chats.export_delta_one(
username=username,
delta_root=tmp,
names={username: "张三"},
run_id="20260526T080000",
start_ts=1778890001,
end_ts=1778890100,
transcribe=False,
)
self.assertTrue(result["success"])
self.assertTrue(result["skipped"])
self.assertEqual(result["message_count"], 0)
self.assertNotIn("path", result)
self.assertFalse(os.path.exists(os.path.join(
tmp,
"deltas",
"20260526T080000",
"chats",
)))
class DeltaOnlyCliTests(unittest.TestCase):
def test_requires_start_before_touching_decrypted_dir(self):
err = io.StringIO()
with redirect_stderr(err):
with self.assertRaises(SystemExit) as cm:
export_all_chats.main(["--delta-only", r"D:\tmp\out"])
self.assertEqual(cm.exception.code, 2)
self.assertIn("--delta-only requires --start", err.getvalue())
class DeltaManifestTests(unittest.TestCase):
def test_records_files(self):
with tempfile.TemporaryDirectory() as tmp:
manifest_path = export_all_chats._write_delta_manifest(
delta_root=tmp,
run_id="20260526T080000",
start_ts=1778889600,
end_ts=1778890000,
chats_checked=3,
results=[
{
"success": True,
"username": "wxid_delta",
"chat": "张三",
"path": "chats/single_张三__wxid_delta.delta.json",
"message_count": 2,
},
{
"success": False,
"username": "wxid_empty",
"message_count": 0,
"reason": "no tables",
},
{
"success": True,
"skipped": True,
"username": "wxid_window_empty",
"message_count": 0,
"reason": "no messages in delta window",
},
],
)
with open(manifest_path, encoding="utf-8") as f:
data = json.load(f)
self.assertEqual(data["export_kind"], "wechat_delta_run")
self.assertEqual(data["chats_checked"], 3)
self.assertEqual(data["chats_with_messages"], 1)
self.assertEqual(data["messages_exported"], 2)
self.assertEqual(data["files"][0]["username"], "wxid_delta")
self.assertEqual(data["errors"][0]["username"], "wxid_empty")
def _msg_table_name(username):
return "Msg_" + hashlib.md5(username.encode("utf-8")).hexdigest()
def _create_message_db(path, username):
table = _msg_table_name(username)
conn = sqlite3.connect(path)
conn.execute(
f"""
CREATE TABLE [{table}] (
local_id INTEGER,
local_type INTEGER,
create_time INTEGER,
real_sender_id INTEGER,
message_content TEXT,
WCDB_CT_message_content INTEGER
)
"""
)
conn.executemany(
f"INSERT INTO [{table}] VALUES (?, ?, ?, ?, ?, ?)",
[
(1, 1, 1778889500, 0, "too early", 0),
(2, 1, 1778889601, 0, "in range", 0),
],
)
conn.commit()
conn.close()
return table
if __name__ == "__main__":
unittest.main()