feat: add pocketbase dry-run report

This commit is contained in:
Codex
2026-06-29 11:06:31 +08:00
parent e587a0c955
commit f74db433cc
14 changed files with 811 additions and 13 deletions

View File

@@ -157,8 +157,10 @@ npm run check:refactor
npm run check:api
npm run check:worker
npm run check:importer
npm run pb:import:dry-run
npm run pb:import:validate
npm run test:readiness
npm run test:pb:dry-run
npm run test:api
npm run test:worker:crm
npm run test:worker:commerce
@@ -183,6 +185,22 @@ npm run readiness:production:db
这个检查会阻断默认弱密钥、`CORS=*`、mock 短信、legacy 身份头、local_dev 存储、对象存储未配置、CRM insecure localhost 等生产风险;带 `:db` 的版本还会检查租户 provider 公开配置是否混入密钥、活跃短信/OAuth/支付 provider 是否缺少 `app_private.tenant_secrets`、域名是否未验证。
## PocketBase 迁移 Dry-Run
把旧 PocketBase 导出的集合 JSON 放到仓库根目录 `pb_export/` 后,先执行不写数据库的静态 dry-run
```bash
npm run pb:import:dry-run
```
需要给 CI 或脚本读取时:
```bash
npm run pb:import:dry-run -- --json
```
dry-run 会检查导出目录、JSON 形态、核心集合缺失、重复/缺失旧 ID、敏感字段、旧 schema 关系断裂和未映射集合。存在 blocker 时命令返回非 0所有 blocker 处理完后,再执行 `npm run pb:import:json``npm run pb:import:validate`
## API 模块
当前 API 目录:

View File

@@ -158,7 +158,8 @@
| 能力 | 状态 | 说明 |
| --- | --- | --- |
| PocketBase schema 分析 | 可联调 | `scripts/import-pocketbase` |
| PocketBase schema/导出分析 | 可联调 | `scripts/import-pocketbase` 支持 schema summary/risk、`npm run pb:import:dry-run` 导出目录静态迁移报告 |
| PocketBase JSON dry-run | 可联调 | 不写数据库检查导出目录、JSON 形态、核心集合、旧 ID、敏感字段、schema relation、未映射集合和关键业务计数 |
| 题目 JSON preview/import | 可联调 | 后端负责规范化、issue、幂等、审计 |
| 公共题库采纳、手动同步和自动同步 | 可联调 | 平台授权后,租户可采纳公共题库并复制已发布题目快照;同步 API 和 `public-banks` worker 支持新增/更新题目、重新校验授权、跨租户拒绝、审计记录和租户自改冲突保护;已覆盖跨租户、重复采纳、采纳后组卷、同步新增题、冲突不覆盖和 worker 自动同步测试 |
| 单词 JSON preview/import | 可联调 | 兼容旧模板 |
@@ -179,6 +180,7 @@ npx supabase db reset
npm audit --audit-level=high
npm run check:refactor
npm run test:readiness
npm run test:pb:dry-run
npm run test:worker:imports
npm run test:worker:crm
npm run test:worker:commerce

View File

@@ -32,7 +32,7 @@
| 会员与订单 | 可联调 | 下单、订单详情/状态轮询、优惠券领取/抵扣、零元订单自动开通、手工确认权限保护、激活码预检查/兑换、微信支付、支付宝、微信/支付宝发起退款、微信/支付宝退款查询确认、微信/支付宝退款通知 webhook、支付/退款补偿 worker、权益发放 | 完整资金流水对账、异常订单运营台 |
| 登录认证 | 可联调 | 短信 mock、阿里云/腾讯云短信 adapter、迁移期 session、Supabase Auth JWT、微信小程序登录、微信网页登录、QQ 登录、手机号绑定/换绑、OAuth 配置表 | 真实生产账号和回调域名联调 |
| 销售/代理/CRM | 基础完成 | 邀请码、首绑保护、团队关系、销售统计、CRM 入队 | 小程序码真实生成、分佣结算、钉钉/飞书/企微 worker |
| 内容导入 | 可联调 | 题目、单词、知识手册、分数线、视频 JSON/CSV/Excel preview/import、issue、job、审计、幂等、`executionMode=async`、imports worker、导入后复检、模板下载字段映射 API | 字段映射 UI、真实数据 dry-run 和导入性能压测 |
| 内容导入 | 可联调 | 题目、单词、知识手册、分数线、视频 JSON/CSV/Excel preview/import、issue、job、审计、幂等、`executionMode=async`、imports worker、导入后复检、模板下载字段映射 API、PocketBase JSON dry-run 报告 | 字段映射 UI、真实数据 dry-run 执行验收和导入性能压测 |
| 数据看板 | 可联调 | 租户 dashboard 聚合接口收益、注册、学习、内容、激活码、反馈、趋势、24h 活跃、套餐销量和运营动态 | 预聚合 worker、缓存、慢 SQL 监控和销售转化看板 |
| AI 择校推荐 | 未开始 | 暂无 | 数据上下文、AI JSON schema、报告渲染、PDF 生成 |
| Taro 前端 | 未开始 | 旧 Web 已有新 API 适配雏形 | `apps/taro`、跨端 API client、H5/小程序页面和端到端测试 |
@@ -77,7 +77,7 @@
- 生产鉴权API 已支持 Supabase Auth JWT继续做真实云端 Auth/JWKS 回归、RLS 深测,并在生产关闭 `x-user-id``x-platform-admin-key` 兼容入口。
- 对象存储:上传/下载签名已接入阿里云 OSS、腾讯云 COS、Supabase Storage上传确认、PDF/图片预览签名和 assets worker 复检已完成,继续补 PDF 渲染、视频播放防盗链、杀毒扫描和水印。
- 真实数据 dry-run导出 PocketBase 用户、题库、单词、知识手册、分数线、订单、权益,跑迁移和校验报告。
- 真实数据 dry-run导出 PocketBase 用户、题库、单词、知识手册、分数线、订单、权益,先跑 `npm run pb:import:dry-run`,再跑迁移和校验报告。
- 生产环境配置:`.env.example``npm run readiness:production` / `npm run readiness:production:db` 已补;继续补数据库迁移流程、备份恢复、日志、告警和 API 容器部署说明。
- Taro scaffold建立 `apps/taro`,先完成租户解析、首页、题库、背单词、知识手册、个人中心主链路。

View File

@@ -378,6 +378,6 @@ GET /api/tenant-content/imports/post-check?jobId=<jobId>
## 下一步
- 真实 PocketBase 全量导出数据做多轮 dry-run并把复检报告作为上线验收材料。
- 已补 `npm run pb:import:dry-run` 静态报告工具;拿到真实 PocketBase 全量导出数据做多轮 dry-run并把 dry-run 报告、正式导入报告和复检报告作为上线验收材料。
- 前端补可视化字段映射、模板下载按钮、导入 job 轮询、逐行 issue 展示和复检结果面板。
- 后续按大租户数据量补导入任务分页预览、抽样校验和导入性能压测。

View File

@@ -15,7 +15,7 @@
| 旧 PocketBase/React 项目 | `F:\project\参考\旧题库项目` | 保留作为功能参照和迁移来源 |
| 新 Node API | `F:\project\apps\api` | 已按 `core/features` 分层重构 |
| Supabase/PostgreSQL 迁移 | `F:\project\supabase\migrations` | 已建立多租户和业务域表 |
| PocketBase 数据导入 | `F:\project\scripts\import-pocketbase` | 已支持多类旧数据归一化导入与校验 |
| PocketBase 数据导入 | `F:\project\scripts\import-pocketbase` | 已支持多类旧数据归一化导入、dry-run 报告与校验 |
| 共享包 | `F:\project\packages\config``F:\project\packages\db``F:\project\packages\domain` | 已建立基础共享层 |
| 旧 Web 到新 API 适配 | `F:\project\参考\旧题库项目\src\services\supabaseApi.ts` | 仅作参考;后续应在 `apps/taro` 或共享包中建立新的跨端 API client |
@@ -39,7 +39,7 @@
| 租户后台 | 已建品牌、域名、设置、支付账户、登录 provider、私密密钥表、成员、审计日志、资源台账、导入台账、内容导航台账 | 不适用 | 概览、品牌、设置、域名、支付账户、登录配置、密钥掩码、活动内容、兑换码/优惠券、成员管理、权限矩阵、审计查询、内容入口/分类树/题目集合/练习蓝图维护、资源管理、题目/单词/知识手册/分数线/视频 JSON/CSV/Excel 同步/异步导入已实现 | 核心 API 集成测试含角色/权限/租户隔离/密钥不泄露/导航/组卷/资源与导入断言 | 租户配置与运营闭环可用,前端权限 UI、字段映射操作台和导入复检结果面板待补 |
| 平台后台 | 已建 SaaS 套餐、订阅、账单、服务费、用量 | 不适用 | 租户管理、账单、收款确认、用量记录、平台管理员 Supabase JWT 鉴权已实现 | API 集成测试 | 平台收费链路骨架可用,平台审计报表/自动计费待补 |
| 登录认证 | 已建短信验证码、会话、OAuth provider 配置表,并支持 `auth_user_id` 映射 | 旧用户映射已预留 | 短信 mock 登录、迁移期 session、Supabase JWT 验签映射、微信小程序登录主链路、微信网页登录、QQ 登录、手机号绑定/换绑已实现 | API 集成测试 | H5 Supabase Auth 可联调;真实短信/OAuth 生产账号和回调域名联调待补 |
| 数据导入 | 已建立 importer、risk report、validate | 已覆盖多类旧集合 | 命令行导入/校验 | `pb:import:validate` | 基础工具可用,需真实完整数据多轮 dry-run |
| 数据导入 | 已建立 importer、risk report、dry-run report、validate | 已覆盖多类旧集合 | 命令行 dry-run/导入/校验 | `pb:import:dry-run``pb:import:validate` | 基础工具可用,需真实完整数据执行多轮 dry-run 和导入回归 |
| 测试体系 | 不适用 | 不适用 | 不适用 | 已新增核心 API 集成测试、租户隔离测试、权限矩阵测试、资源/题目导入测试、导入校验 | 还不是完整覆盖,支付幂等、真实导入回归、前端端到端测试仍需补 |
## 已实现 API 范围
@@ -284,7 +284,7 @@ platform-admin:
为了先把旧项目核心业务补齐,再进入支付/短信等商用关键模块,建议按下面顺序继续:
1. 完善内容导入和文件上传:字段映射 UI、真实数据 dry-run、CDN 防盗链、杀毒扫描。
1. 完善内容导入和文件上传:字段映射 UI、真实数据 dry-run 执行验收、CDN 防盗链、杀毒扫描。
2. 补公共题库版本通知/冲突操作台、租户套餐地区/科目/题库范围限制、主题模板系统。
3. 补学习统计增强:排行榜防刷/预聚合、断点续练、专项练习策略和更细题型分析。
4. 补视频商用控制:深度防盗链、动态水印和播放统计。

View File

@@ -98,7 +98,7 @@
1. 排行榜增强:刷题、模考、背单词、积分排行榜主接口已有;还需防刷、日/周榜预聚合、运营后台排名看板。
2. 账号设置完整流:绑定/更换手机号基础 API 已完成;仍缺头像上传、微信/QQ 账号合并、密码/邮箱能力。
3. 题库导出:服务端 JSON/试卷 payload 导出、权限审计和答案脱敏已补;仍缺 PDF/Word 二进制生成、水印、资料发布和后台导出操作台。
4. 导入扩展:题目/单词/知识手册/分数线/视频已支持 JSON、CSV 和 Excel 预览导入,并可用 `executionMode=async` 进入 imports worker导入后复检、模板下载字段映射 API 已补,仍缺前端字段映射 UI 和真实数据 dry-run
4. 导入扩展:题目/单词/知识手册/分数线/视频已支持 JSON、CSV 和 Excel 预览导入,并可用 `executionMode=async` 进入 imports worker导入后复检、模板下载字段映射 API 和 PocketBase JSON dry-run 报告已补,仍缺前端字段映射 UI 和真实数据执行验收
5. 公共题库商业化:平台公共/地区题库授权、租户快照采纳、手动同步、自动同步 worker、冲突查询和租户自改冲突保护已完成基础闭环还需版本通知、冲突处理操作台和运营后台 UI。
6. CRM/销售结算CRM worker、分佣规则、结算单、审核和打款状态基础闭环已完成仍缺轮询/定向分配、打款导出、凭证和销售结算看板。
7. 题目反馈增强:处理通知、消息提醒、问题聚合统计和内容修复闭环。
@@ -117,7 +117,7 @@
1. 微信/支付宝支付和 webhook 幂等。
2. 对象存储 PDF 预览、视频深度防盗链、动态水印。
3. 字段映射 UI、真实数据 dry-run 和导入复检结果操作台。
3. 字段映射 UI、真实数据 dry-run 执行验收和导入复检结果操作台。
4. 数据看板预聚合 worker、销售/代理转化看板和分佣结算。
5. 公共题库版本通知、冲突处理操作台和租户确认/跳过策略。

View File

@@ -109,10 +109,13 @@ PORT=8787
把 PocketBase 导出的集合 JSON 放到仓库根目录的 `pb_export` 文件夹后执行:
```bash
npm run pb:import:dry-run
npm run pb:import:json
npm run pb:import:validate
```
`pb:import:dry-run` 不写数据库,会先检查 JSON 格式、核心集合、旧 ID、敏感字段、关系断裂和未映射集合。存在 blocker 时先处理问题,不要直接正式导入。
导入真实密钥时必须明确打开开关,且只允许进入 `app_private.tenant_secrets`
```bash

View File

@@ -52,7 +52,7 @@
3. 真实导入 dry-run
- 从 PocketBase 导出现有用户、题库、单词、知识手册、分数线、订单、权益数据。
- `scripts/import-pocketbase`,生成迁移报告
- 已补 `npm run pb:import:dry-run` 静态迁移报告工具;拿到真实导出后先跑 dry-run再跑 `pb:import:json``pb:import:validate`
- 对题目 JSON、单词、知识手册、分数线、视频走后端 preview/import API 做二次验证。
4. 部署配置
@@ -78,7 +78,7 @@
3. 导入体系扩展
- 已完成题目、单词、知识手册、分数线、视频的 CSV/Excel 到规范 JSON 解析适配。
- 已完成大批量异步导入 worker、重试、导入后复检和审计。
- 模板下载、字段映射 API导入后复检已补;继续补字段映射 UI、真实数据 dry-run、导入前抽样校验和复检结果操作台。
- 模板下载、字段映射 API导入后复检和 PocketBase JSON dry-run 报告工具已补;继续补字段映射 UI、真实数据执行验收、导入前抽样校验和复检结果操作台。
4. 题库导出增强
- 已完成服务端 JSON、`paper_json``print_payload` 基础导出和导出 job 台账。

View File

@@ -67,7 +67,30 @@ pb_export/
questions.json
```
执行导入
执行不写数据库的 dry-run
```bash
npm run import:dry-run
```
根目录也可以执行:
```bash
npm run pb:import:dry-run
```
dry-run 会检查:
- 导出目录和 JSON 文件格式。
- 核心集合是否缺失。
- 旧记录 `id` 是否缺失或重复。
-`pb_schema.json` 中的 relation 是否能在导出数据里找到目标记录。
- `settings``crm_config`、用户资料等位置是否存在敏感字段。
- 哪些集合当前已有规范化 mapper哪些集合仍未映射。
存在 `blocker` 时不要执行正式导入;先修导出、补 mapper 或记录业务处理结论。
执行正式导入:
```bash
npm run import:json
@@ -82,6 +105,7 @@ npm run import:validate
根目录也可以执行:
```bash
npm run pb:import:dry-run
npm run pb:import:json
npm run pb:import:validate
```

View File

@@ -22,7 +22,7 @@
"check:api": "npm --workspace @tiku-saas/api run check",
"check:importer": "npm --workspace @tiku-saas/import-pocketbase run check",
"check:worker": "npm --workspace @tiku-saas/worker run check",
"check:refactor": "npm run check:api && npm run check:worker && npm run check:importer && npm run pb:import:validate && npm run test:readiness && npm run test:api",
"check:refactor": "npm run check:api && npm run check:worker && npm run check:importer && npm run pb:import:validate && npm run test:readiness && npm run test:pb:dry-run && npm run test:api",
"docker:api:build": "docker compose -f docker-compose.api.yml build",
"docker:api:up": "docker compose -f docker-compose.api.yml up api",
"docker:api:down": "docker compose -f docker-compose.api.yml down",
@@ -39,11 +39,13 @@
"test:worker:imports": "npm run db:smoke-seed && npm run build:worker && node scripts/import-worker-integration-test.js",
"test:worker:public-banks": "npm run db:smoke-seed && npm run build:worker && node scripts/public-bank-worker-integration-test.js",
"test:readiness": "node scripts/production-readiness-check-test.js",
"test:pb:dry-run": "node scripts/pb-dry-run-report-test.js",
"readiness:production": "node scripts/production-readiness-check.js --skip-db",
"readiness:production:db": "node scripts/production-readiness-check.js --check-db",
"test:api:remote": "node scripts/api-integration-test.js",
"pb:schema:summary": "npm --workspace @tiku-saas/import-pocketbase run schema:summary",
"pb:schema:risk": "npm --workspace @tiku-saas/import-pocketbase run schema:risk",
"pb:import:dry-run": "npm --workspace @tiku-saas/import-pocketbase run import:dry-run --",
"pb:import:json": "npm --workspace @tiku-saas/import-pocketbase run import:json",
"pb:import:validate": "npm --workspace @tiku-saas/import-pocketbase run import:validate"
}

View File

@@ -6,6 +6,7 @@
"scripts": {
"schema:summary": "tsx src/analyze-schema.ts",
"schema:risk": "tsx src/risk-report.ts",
"import:dry-run": "tsx src/dry-run-report.ts",
"import:json": "tsx src/import-json.ts",
"import:validate": "tsx src/validate-import.ts",
"check": "tsc -p tsconfig.json --noEmit"

View File

@@ -0,0 +1,523 @@
import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { fieldsOf, readPocketBaseSchema, type PocketBaseCollection, type PocketBaseField } from './pb-schema.js';
import { loadEnv } from './env.js';
loadEnv();
type JsonRecord = Record<string, unknown> & { id?: unknown };
type Severity = 'blocker' | 'warning' | 'info';
interface DryRunIssue {
severity: Severity;
code: string;
collection?: string;
file?: string;
fieldPath?: string;
message: string;
sample?: string;
count?: number;
}
interface CollectionReport {
collection: string;
file: string;
recordCount: number;
missingIdCount: number;
duplicateIdCount: number;
fieldCount: number;
sampleIds: string[];
supported: boolean;
}
interface DryRunReport {
exportDir: string;
schemaPath?: string;
generatedAt: string;
summary: {
files: number;
collections: number;
records: number;
supportedCollections: number;
unsupportedCollections: number;
blockers: number;
warnings: number;
infos: number;
};
businessCounts: Record<string, number>;
collections: CollectionReport[];
issues: DryRunIssue[];
}
const args = new Set(process.argv.slice(2));
const jsonOutput = args.has('--json');
const failOnWarnings = args.has('--fail-on-warnings') || process.env.PB_DRY_RUN_FAIL_ON_WARNINGS === 'true';
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../..');
const exportDir = path.resolve(repoRoot, process.env.PB_EXPORT_DIR || 'pb_export');
const supportedCollections = new Set([
'announcements',
'app_assets',
'audit_logs',
'badges',
'banners',
'categories',
'code_batches',
'codes',
'coupons',
'coupon_redemptions',
'crm_config',
'crm_webhook_log',
'crm_webhook_queue',
'dashboard_daily_stats',
'exam_dates',
'faqs',
'handbook_chapters',
'handbook_entries',
'handbook_subjects',
'images',
'majors',
'module_nodes',
'orders',
'products',
'question_type_groups',
'question_videos',
'questions',
'recent_practices',
'referral_tracks',
'region_modules',
'regions',
'reports',
'revenue_daily_stats',
'schools',
'scoreline_fields',
'scoreline_majors',
'scoreline_records',
'scoreline_schools',
'settings',
'subject_shares',
'subjects',
'svip_plans',
'timelines',
'user_badges',
'user_word_favorites',
'user_word_progress',
'users',
'video_explanations',
'vocabulary',
'vocabulary_units',
]);
const coreCollections = [
'users',
'questions',
'subjects',
'categories',
'orders',
'svip_plans',
'codes',
'vocabulary_units',
'vocabulary',
'handbook_subjects',
'handbook_chapters',
'handbook_entries',
];
const businessCollections = [
'users',
'questions',
'orders',
'svip_plans',
'codes',
'vocabulary_units',
'vocabulary',
'handbook_subjects',
'handbook_chapters',
'handbook_entries',
'scoreline_schools',
'scoreline_majors',
'scoreline_records',
'video_explanations',
'question_videos',
'referral_tracks',
'coupons',
'badges',
];
const sensitiveKeyPattern =
/(password|token|secret|privatekey|sessionkey|accesskey|appkey|apikey|api_v3_key|notifytoken|aeskey|openid|unionid|wxaccesstoken|wechatsessionkey|smscode|verifycode|verificationcode|captcha)/i;
const highRiskCollections = new Set(['settings', 'crm_config', 'smscodes']);
function issue(issues: DryRunIssue[], nextIssue: DryRunIssue) {
issues.push(nextIssue);
}
function asRecords(input: unknown): JsonRecord[] | null {
if (Array.isArray(input)) return input as JsonRecord[];
if (input && typeof input === 'object') {
const candidate = input as { items?: unknown; records?: unknown };
if (Array.isArray(candidate.items)) return candidate.items as JsonRecord[];
if (Array.isArray(candidate.records)) return candidate.records as JsonRecord[];
}
return null;
}
function collectionNameFromFile(fileName: string) {
return fileName.replace(/\.json$/i, '');
}
function text(value: unknown) {
if (value === null || value === undefined) return '';
return String(value).trim();
}
function readJsonFile(filePath: string) {
return JSON.parse(fs.readFileSync(filePath, 'utf8')) as unknown;
}
function scalarSample(value: unknown) {
if (value === null || value === undefined) return '';
if (typeof value === 'object') return JSON.stringify(value).slice(0, 160);
return String(value).slice(0, 160);
}
function walkSensitiveFields(
issues: DryRunIssue[],
collection: string,
value: unknown,
pathParts: string[] = [],
depth = 0,
) {
if (!value || typeof value !== 'object' || depth > 8) return;
if (Array.isArray(value)) {
value.slice(0, 10).forEach((item, index) => walkSensitiveFields(issues, collection, item, [...pathParts, String(index)], depth + 1));
return;
}
for (const [key, childValue] of Object.entries(value as Record<string, unknown>)) {
const fieldPath = [...pathParts, key].join('.');
if (sensitiveKeyPattern.test(key) && childValue !== null && childValue !== undefined && childValue !== '') {
issue(issues, {
severity: highRiskCollections.has(collection) ? 'blocker' : 'warning',
code: 'sensitive_field_in_export',
collection,
fieldPath,
message: `Legacy export contains sensitive-looking field "${fieldPath}". It must not be copied into public normalized tables.`,
sample: scalarSample(childValue),
});
}
walkSensitiveFields(issues, collection, childValue, [...pathParts, key], depth + 1);
}
}
function fieldValues(record: JsonRecord, field: PocketBaseField) {
const value = record[field.name];
if (value === null || value === undefined || value === '') return [];
if (Array.isArray(value)) return value.map(item => text(item)).filter(Boolean);
return [text(value)].filter(Boolean);
}
function relationTargetName(field: PocketBaseField, byId: Map<string, PocketBaseCollection>) {
if (!field.collectionId) return '';
return byId.get(field.collectionId)?.name || '';
}
function validateRelations(
issues: DryRunIssue[],
collection: PocketBaseCollection,
records: JsonRecord[],
idIndex: Map<string, Set<string>>,
byId: Map<string, PocketBaseCollection>,
) {
const relationFields = fieldsOf(collection).filter(field => field.type === 'relation');
for (const field of relationFields) {
const targetName = relationTargetName(field, byId);
if (!targetName) {
issue(issues, {
severity: 'warning',
code: 'relation_target_schema_missing',
collection: collection.name,
fieldPath: field.name,
message: `Relation field "${field.name}" points to an unknown collection id in pb_schema.json.`,
});
continue;
}
const targetIds = idIndex.get(targetName);
if (!targetIds) {
const hasValues = records.some(record => fieldValues(record, field).length > 0);
if (hasValues) {
issue(issues, {
severity: coreCollections.includes(collection.name) ? 'blocker' : 'warning',
code: 'relation_target_export_missing',
collection: collection.name,
fieldPath: field.name,
message: `Relation field "${field.name}" references "${targetName}", but ${targetName}.json is not present in export.`,
});
}
continue;
}
let unresolved = 0;
let sample = '';
for (const record of records) {
for (const value of fieldValues(record, field)) {
if (!targetIds.has(value)) {
unresolved += 1;
if (!sample) sample = `${text(record.id) || '(missing id)'} -> ${value}`;
}
}
}
if (unresolved > 0) {
issue(issues, {
severity: coreCollections.includes(collection.name) ? 'blocker' : 'warning',
code: 'relation_unresolved',
collection: collection.name,
fieldPath: field.name,
message: `Relation field "${field.name}" has ${unresolved} unresolved reference(s) to "${targetName}".`,
count: unresolved,
sample,
});
}
}
}
function loadSchema(issues: DryRunIssue[]) {
try {
return readPocketBaseSchema();
} catch (error) {
issue(issues, {
severity: 'warning',
code: 'schema_unavailable',
message: `pb_schema.json could not be loaded; relation checks will be skipped. ${error instanceof Error ? error.message : ''}`.trim(),
});
return null;
}
}
function collectFieldNames(records: JsonRecord[]) {
const fields = new Set<string>();
for (const record of records.slice(0, 500)) {
Object.keys(record).forEach(key => fields.add(key));
}
return fields;
}
function buildReport(): DryRunReport {
const issues: DryRunIssue[] = [];
const schema = loadSchema(issues);
const collectionByName = schema?.byName || new Map<string, PocketBaseCollection>();
const files = fs.existsSync(exportDir) ? fs.readdirSync(exportDir).filter(file => file.toLowerCase().endsWith('.json')) : [];
const collections: CollectionReport[] = [];
const recordsByCollection = new Map<string, JsonRecord[]>();
const idIndex = new Map<string, Set<string>>();
if (!fs.existsSync(exportDir)) {
issue(issues, {
severity: 'blocker',
code: 'export_dir_missing',
message: `Export dir not found: ${exportDir}`,
});
}
if (files.length === 0) {
issue(issues, {
severity: 'blocker',
code: 'export_files_missing',
message: `No PocketBase collection JSON files found in ${exportDir}`,
});
}
for (const file of files.sort()) {
const collection = collectionNameFromFile(file);
const filePath = path.join(exportDir, file);
let parsed: unknown;
try {
parsed = readJsonFile(filePath);
} catch (error) {
issue(issues, {
severity: 'blocker',
code: 'json_parse_failed',
collection,
file,
message: `Failed to parse ${file}: ${error instanceof Error ? error.message : String(error)}`,
});
continue;
}
const records = asRecords(parsed);
if (!records) {
issue(issues, {
severity: 'blocker',
code: 'collection_shape_invalid',
collection,
file,
message: `${file} must be an array or an object with items/records array.`,
});
continue;
}
const ids = new Set<string>();
const duplicateIds = new Set<string>();
let missingIdCount = 0;
for (const record of records) {
const id = text(record.id);
if (!id) {
missingIdCount += 1;
continue;
}
if (ids.has(id)) duplicateIds.add(id);
ids.add(id);
}
recordsByCollection.set(collection, records);
idIndex.set(collection, ids);
const fields = collectFieldNames(records);
collections.push({
collection,
file,
recordCount: records.length,
missingIdCount,
duplicateIdCount: duplicateIds.size,
fieldCount: fields.size,
sampleIds: [...ids].slice(0, 5),
supported: supportedCollections.has(collection),
});
if (!supportedCollections.has(collection) && !collection.startsWith('_')) {
issue(issues, {
severity: 'warning',
code: 'unsupported_collection',
collection,
file,
message: `${collection}.json is not currently normalized by import-json.ts. It will be preserved only if a future mapper is added.`,
count: records.length,
});
}
if (missingIdCount > 0) {
issue(issues, {
severity: coreCollections.includes(collection) ? 'blocker' : 'warning',
code: 'record_id_missing',
collection,
file,
message: `${collection} has ${missingIdCount} record(s) without id.`,
count: missingIdCount,
});
}
if (duplicateIds.size > 0) {
issue(issues, {
severity: 'blocker',
code: 'record_id_duplicate',
collection,
file,
message: `${collection} has duplicate legacy id(s).`,
count: duplicateIds.size,
sample: [...duplicateIds].slice(0, 5).join(', '),
});
}
records.slice(0, 200).forEach(record => walkSensitiveFields(issues, collection, record));
}
for (const collection of coreCollections) {
if (!recordsByCollection.has(collection)) {
issue(issues, {
severity: 'warning',
code: 'core_collection_missing',
collection,
message: `${collection}.json is not present. If old production has this collection, export it before migration.`,
});
}
}
if (schema) {
for (const [collectionName, records] of recordsByCollection.entries()) {
const collection = collectionByName.get(collectionName);
if (collection) validateRelations(issues, collection, records, idIndex, schema.byId);
}
}
const businessCounts = Object.fromEntries(
businessCollections.map(collection => [collection, recordsByCollection.get(collection)?.length || 0]),
);
const supportedCount = collections.filter(item => item.supported).length;
const unsupportedCount = collections.length - supportedCount;
const recordCount = collections.reduce((sum, item) => sum + item.recordCount, 0);
const blockers = issues.filter(item => item.severity === 'blocker').length;
const warnings = issues.filter(item => item.severity === 'warning').length;
const infos = issues.filter(item => item.severity === 'info').length;
return {
exportDir,
schemaPath: schema?.absPath,
generatedAt: new Date().toISOString(),
summary: {
files: files.length,
collections: collections.length,
records: recordCount,
supportedCollections: supportedCount,
unsupportedCollections: unsupportedCount,
blockers,
warnings,
infos,
},
businessCounts,
collections: collections.sort((a, b) => a.collection.localeCompare(b.collection)),
issues: issues.sort((a, b) => {
const order: Record<Severity, number> = { blocker: 0, warning: 1, info: 2 };
return order[a.severity] - order[b.severity] || (a.collection || '').localeCompare(b.collection || '') || a.code.localeCompare(b.code);
}),
};
}
function printHuman(report: DryRunReport) {
console.log('PocketBase dry-run migration report');
console.log(`Export dir: ${report.exportDir}`);
if (report.schemaPath) console.log(`Schema: ${report.schemaPath}`);
console.log(
`Summary: ${report.summary.records} records across ${report.summary.collections} collections; ` +
`${report.summary.blockers} blocker(s), ${report.summary.warnings} warning(s)`,
);
console.log('');
console.log('Business counts:');
for (const [name, count] of Object.entries(report.businessCounts)) {
console.log(` ${name.padEnd(24)} ${count}`);
}
console.log('');
console.log('Collections:');
for (const item of report.collections) {
const marker = item.supported ? 'mapped' : 'unmapped';
console.log(
` ${item.collection.padEnd(24)} records=${String(item.recordCount).padStart(6)} ` +
`fields=${String(item.fieldCount).padStart(3)} ${marker}`,
);
}
console.log('');
console.log('Issues:');
if (report.issues.length === 0) {
console.log(' none');
} else {
for (const item of report.issues.slice(0, 200)) {
const location = [item.collection, item.fieldPath].filter(Boolean).join('.') || item.file || 'export';
const count = item.count === undefined ? '' : ` (${item.count})`;
const sample = item.sample ? ` sample=${item.sample}` : '';
console.log(` [${item.severity.toUpperCase()}] ${item.code} ${location}: ${item.message}${count}${sample}`);
}
if (report.issues.length > 200) console.log(` ... ${report.issues.length - 200} more issue(s) omitted`);
}
}
function main() {
const report = buildReport();
if (jsonOutput) console.log(JSON.stringify(report, null, 2));
else printHuman(report);
if (report.summary.blockers > 0 || (failOnWarnings && report.summary.warnings > 0)) {
process.exitCode = 1;
}
}
main();

View File

@@ -0,0 +1,122 @@
import assert from 'node:assert/strict';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { spawnSync } from 'node:child_process';
const repoRoot = process.cwd();
const dryRunScript = path.join(repoRoot, 'scripts', 'import-pocketbase', 'src', 'dry-run-report.ts');
function writeJson(filePath, value) {
fs.writeFileSync(filePath, JSON.stringify(value, null, 2), 'utf8');
}
function runDryRun(exportDir) {
const result = spawnSync(process.execPath, ['--import', 'tsx', dryRunScript, '--json'], {
cwd: repoRoot,
encoding: 'utf8',
env: {
...process.env,
PB_EXPORT_DIR: exportDir,
},
});
if (result.error) throw result.error;
const stdout = result.stdout || '';
const jsonStart = stdout.indexOf('{');
const payload = JSON.parse(jsonStart >= 0 ? stdout.slice(jsonStart) : '{}');
return { ...result, payload };
}
function createSafeExport() {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'tiku-pb-dry-safe-'));
writeJson(path.join(dir, 'users.json'), [
{ id: 'u1', phone: '13800000000', role: 'student', nickname: '学生A' },
]);
writeJson(path.join(dir, 'subjects.json'), [
{ id: 's1', name: '英语' },
]);
writeJson(path.join(dir, 'categories.json'), [
{ id: 'c1', name: '阅读理解', subjectId: 's1' },
]);
writeJson(path.join(dir, 'questions.json'), [
{ id: 'q1', subjectId: 's1', categoryId: 'c1', type: 'choice', content: '题干', answer: 'A' },
]);
writeJson(path.join(dir, 'orders.json'), [
{ id: 'o1', userId: 'u1', amount: 99, status: 'paid' },
]);
writeJson(path.join(dir, 'svip_plans.json'), [
{ id: 'p1', name: '题库会员', price: 99 },
]);
writeJson(path.join(dir, 'codes.json'), [
{ id: 'code1', code: 'SAFE-001', status: 'unused' },
]);
writeJson(path.join(dir, 'vocabulary_units.json'), [
{ id: 'vu1', name: 'Unit 1' },
]);
writeJson(path.join(dir, 'vocabulary.json'), [
{ id: 'vw1', unitId: 'vu1', word: 'abandon', meaning: '放弃' },
]);
writeJson(path.join(dir, 'handbook_subjects.json'), [
{ id: 'hs1', name: '政治' },
]);
writeJson(path.join(dir, 'handbook_chapters.json'), [
{ id: 'hc1', subjectId: 'hs1', title: '第一章' },
]);
writeJson(path.join(dir, 'handbook_entries.json'), [
{ id: 'he1', chapterId: 'hc1', title: '知识点', content: '内容' },
]);
return dir;
}
function createUnsafeExport() {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'tiku-pb-dry-unsafe-'));
writeJson(path.join(dir, 'users.json'), [
{ id: 'u1', phone: '13800000000', password: 'plaintext-should-not-exist' },
{ phone: '13900000000' },
]);
writeJson(path.join(dir, 'questions.json'), [
{ id: 'q1', subjectId: 'missing-subject', content: '题干' },
{ id: 'q1', subjectId: 'missing-subject', content: '重复题' },
]);
writeJson(path.join(dir, 'settings.json'), [
{ id: 'settings1', appSecret: 'secret-in-public-export' },
]);
writeJson(path.join(dir, 'unknown_business.json'), [
{ id: 'x1', name: '尚未映射集合' },
]);
return dir;
}
const safeDir = createSafeExport();
const safe = runDryRun(safeDir);
assert.equal(safe.status, 0, `safe dry-run should pass: ${safe.stdout} ${safe.stderr}`);
assert.equal(safe.payload.summary?.blockers, 0, 'safe dry-run should have no blockers');
assert.ok(safe.payload.summary?.records >= 12, 'safe dry-run should count fixture records');
assert.equal(safe.payload.businessCounts?.questions, 1, 'safe dry-run should report question count');
const unsafeDir = createUnsafeExport();
const unsafe = runDryRun(unsafeDir);
assert.notEqual(unsafe.status, 0, 'unsafe dry-run should fail');
assert.ok(unsafe.payload.summary?.blockers > 0, 'unsafe dry-run should report blockers');
assert.ok(
unsafe.payload.issues?.some(item => item.code === 'record_id_missing' && item.collection === 'users'),
'unsafe dry-run should detect missing user id',
);
assert.ok(
unsafe.payload.issues?.some(item => item.code === 'record_id_duplicate' && item.collection === 'questions'),
'unsafe dry-run should detect duplicate question id',
);
assert.ok(
unsafe.payload.issues?.some(item => item.code === 'sensitive_field_in_export'),
'unsafe dry-run should detect sensitive fields',
);
assert.ok(
unsafe.payload.issues?.some(item => item.code === 'unsupported_collection'),
'unsafe dry-run should warn about unsupported collections',
);
fs.rmSync(safeDir, { recursive: true, force: true });
fs.rmSync(unsafeDir, { recursive: true, force: true });
console.log('[PASS] PocketBase dry-run report');

View File

@@ -87,6 +87,109 @@ async function main() {
try {
await client.query('begin');
await client.query(
`
create temporary table smoke_seed_transient_questions (
id uuid primary key
) on commit drop
`,
);
await client.query(
`
insert into smoke_seed_transient_questions (id)
select distinct q.id
from public.questions q
left join public.question_collection_items ci
on ci.tenant_id = q.tenant_id
and ci.question_id = q.id
where q.tenant_id = $1
and q.id <> all($3::uuid[])
and (
q.primary_collection_id = $2::uuid
or ci.collection_id = $2::uuid
or coalesce(q.legacy_id, '') like any($4::text[])
)
`,
[
tenantId,
ids.questionCollection,
[ids.question, ids.questionTwo, ids.questionThree],
['integration-import-%', 'worker-import-question-%', 'public-sync-source-%', 'content-import:%'],
],
);
await client.query(
`
delete from public.question_collection_items
where tenant_id = $1
and (
question_id in (select id from smoke_seed_transient_questions)
or (
collection_id = $2::uuid
and question_id <> all($3::uuid[])
)
)
`,
[tenantId, ids.questionCollection, [ids.question, ids.questionTwo, ids.questionThree]],
);
await client.query(
`
delete from public.question_versions
where tenant_id = $1
and question_id in (select id from smoke_seed_transient_questions)
`,
[tenantId],
);
await client.query(
`
delete from public.questions
where tenant_id = $1
and id in (select id from smoke_seed_transient_questions)
`,
[tenantId],
);
await client.query(
`
delete from public.content_export_jobs
where tenant_id = $1
and scope_id = $2::uuid
`,
[tenantId, ids.questionCollection],
);
await client.query(
`
delete from public.content_import_jobs
where tenant_id = $1
and (
source_name = any($2::text[])
or source_name like 'worker-import-%'
)
`,
[
tenantId,
[
'large-but-allowed-question-import.json',
'invalid-question-import.json',
'valid-question-import.json',
'questions.csv',
'async-question-import.json',
'invalid-vocabulary-import.json',
'legacy-vocabulary-import.json',
'vocabulary.csv',
'handbook-nested-import.json',
'scoreline-batch-import.json',
'scoreline-mixed-items.json',
'scoreline.xlsx',
'video-batch-import.json',
],
],
);
await client.query(
`
delete from public.report_status_events