diff --git a/README.md b/README.md index 18af68ed..53217a44 100644 --- a/README.md +++ b/README.md @@ -157,8 +157,10 @@ npm run check:refactor npm run check:api npm run check:worker npm run check:importer +npm run pb:import:dry-run npm run pb:import:validate npm run test:readiness +npm run test:pb:dry-run npm run test:api npm run test:worker:crm npm run test:worker:commerce @@ -183,6 +185,22 @@ npm run readiness:production:db 这个检查会阻断默认弱密钥、`CORS=*`、mock 短信、legacy 身份头、local_dev 存储、对象存储未配置、CRM insecure localhost 等生产风险；带 `:db` 的版本还会检查租户 provider 公开配置是否混入密钥、活跃短信/OAuth/支付 provider 是否缺少 `app_private.tenant_secrets`、域名是否未验证。 +## PocketBase 迁移 Dry-Run + +把旧 PocketBase 导出的集合 JSON 放到仓库根目录 `pb_export/` 后，先执行不写数据库的静态 dry-run： + +```bash +npm run pb:import:dry-run +``` + +需要给 CI 或脚本读取时： + +```bash +npm run pb:import:dry-run -- --json +``` + +dry-run 会检查导出目录、JSON 形态、核心集合缺失、重复/缺失旧 ID、敏感字段、旧 schema 关系断裂和未映射集合。存在 blocker 时命令返回非 0；所有 blocker 处理完后，再执行 `npm run pb:import:json` 和 `npm run pb:import:validate`。 + ## API 模块 当前 API 目录： diff --git a/docs/refactor/backend-capability-status.md b/docs/refactor/backend-capability-status.md index fec8591f..d077b37e 100644 --- a/docs/refactor/backend-capability-status.md +++ b/docs/refactor/backend-capability-status.md @@ -158,7 +158,8 @@ | 能力 | 状态 | 说明 | | --- | --- | --- | -| PocketBase schema 分析 | 可联调 | `scripts/import-pocketbase` | +| PocketBase schema/导出分析 | 可联调 | `scripts/import-pocketbase` 支持 schema summary/risk、`npm run pb:import:dry-run` 导出目录静态迁移报告 | +| PocketBase JSON dry-run | 可联调 | 不写数据库，检查导出目录、JSON 形态、核心集合、旧 ID、敏感字段、schema relation、未映射集合和关键业务计数 | | 题目 JSON preview/import | 可联调 | 后端负责规范化、issue、幂等、审计 | | 公共题库采纳、手动同步和自动同步 | 可联调 | 平台授权后，租户可采纳公共题库并复制已发布题目快照；同步 API 和 `public-banks` worker 支持新增/更新题目、重新校验授权、跨租户拒绝、审计记录和租户自改冲突保护；已覆盖跨租户、重复采纳、采纳后组卷、同步新增题、冲突不覆盖和 worker 自动同步测试 | | 单词 JSON preview/import | 可联调 | 兼容旧模板 | @@ -179,6 +180,7 @@ npx supabase db reset npm audit --audit-level=high npm run check:refactor npm run test:readiness +npm run test:pb:dry-run npm run test:worker:imports npm run test:worker:crm npm run test:worker:commerce diff --git a/docs/refactor/backend-handoff-roadmap.md b/docs/refactor/backend-handoff-roadmap.md index 2fe385d4..3c294474 100644 --- a/docs/refactor/backend-handoff-roadmap.md +++ b/docs/refactor/backend-handoff-roadmap.md @@ -32,7 +32,7 @@ | 会员与订单 | 可联调 | 下单、订单详情/状态轮询、优惠券领取/抵扣、零元订单自动开通、手工确认权限保护、激活码预检查/兑换、微信支付、支付宝、微信/支付宝发起退款、微信/支付宝退款查询确认、微信/支付宝退款通知 webhook、支付/退款补偿 worker、权益发放 | 完整资金流水对账、异常订单运营台 | | 登录认证 | 可联调 | 短信 mock、阿里云/腾讯云短信 adapter、迁移期 session、Supabase Auth JWT、微信小程序登录、微信网页登录、QQ 登录、手机号绑定/换绑、OAuth 配置表 | 真实生产账号和回调域名联调 | | 销售/代理/CRM | 基础完成 | 邀请码、首绑保护、团队关系、销售统计、CRM 入队 | 小程序码真实生成、分佣结算、钉钉/飞书/企微 worker | -| 内容导入 | 可联调 | 题目、单词、知识手册、分数线、视频 JSON/CSV/Excel preview/import、issue、job、审计、幂等、`executionMode=async`、imports worker、导入后复检、模板下载和字段映射 API | 字段映射 UI、真实数据 dry-run 和导入性能压测 | +| 内容导入 | 可联调 | 题目、单词、知识手册、分数线、视频 JSON/CSV/Excel preview/import、issue、job、审计、幂等、`executionMode=async`、imports worker、导入后复检、模板下载、字段映射 API、PocketBase JSON dry-run 报告 | 字段映射 UI、真实数据 dry-run 执行验收和导入性能压测 | | 数据看板 | 可联调 | 租户 dashboard 聚合接口，收益、注册、学习、内容、激活码、反馈、趋势、24h 活跃、套餐销量和运营动态 | 预聚合 worker、缓存、慢 SQL 监控和销售转化看板 | | AI 择校推荐 | 未开始 | 暂无 | 数据上下文、AI JSON schema、报告渲染、PDF 生成 | | Taro 前端 | 未开始 | 旧 Web 已有新 API 适配雏形 | `apps/taro`、跨端 API client、H5/小程序页面和端到端测试 | @@ -77,7 +77,7 @@ - 生产鉴权：API 已支持 Supabase Auth JWT；继续做真实云端 Auth/JWKS 回归、RLS 深测，并在生产关闭 `x-user-id` 与 `x-platform-admin-key` 兼容入口。 - 对象存储：上传/下载签名已接入阿里云 OSS、腾讯云 COS、Supabase Storage；上传确认、PDF/图片预览签名和 assets worker 复检已完成，继续补 PDF 渲染、视频播放防盗链、杀毒扫描和水印。 -- 真实数据 dry-run：导出 PocketBase 用户、题库、单词、知识手册、分数线、订单、权益，跑迁移和校验报告。 +- 真实数据 dry-run：导出 PocketBase 用户、题库、单词、知识手册、分数线、订单、权益，先跑 `npm run pb:import:dry-run`，再跑迁移和校验报告。 - 生产环境配置：`.env.example` 和 `npm run readiness:production` / `npm run readiness:production:db` 已补；继续补数据库迁移流程、备份恢复、日志、告警和 API 容器部署说明。 - Taro scaffold：建立 `apps/taro`，先完成租户解析、首页、题库、背单词、知识手册、个人中心主链路。 diff --git a/docs/refactor/content-import-contract.md b/docs/refactor/content-import-contract.md index 99fdf573..cece4370 100644 --- a/docs/refactor/content-import-contract.md +++ b/docs/refactor/content-import-contract.md @@ -378,6 +378,6 @@ GET /api/tenant-content/imports/post-check?jobId= ## 下一步 -- 用真实 PocketBase 全量导出数据做多轮 dry-run，并把复检报告作为上线验收材料。 +- 已补 `npm run pb:import:dry-run` 静态报告工具；拿到真实 PocketBase 全量导出数据后做多轮 dry-run，并把 dry-run 报告、正式导入报告和复检报告作为上线验收材料。 - 前端补可视化字段映射、模板下载按钮、导入 job 轮询、逐行 issue 展示和复检结果面板。 - 后续按大租户数据量补导入任务分页预览、抽样校验和导入性能压测。 diff --git a/docs/refactor/implementation-status.md b/docs/refactor/implementation-status.md index 61e7df32..8c3884e8 100644 --- a/docs/refactor/implementation-status.md +++ b/docs/refactor/implementation-status.md @@ -15,7 +15,7 @@ | 旧 PocketBase/React 项目 | `F:\project\参考\旧题库项目` | 保留作为功能参照和迁移来源 | | 新 Node API | `F:\project\apps\api` | 已按 `core/features` 分层重构 | | Supabase/PostgreSQL 迁移 | `F:\project\supabase\migrations` | 已建立多租户和业务域表 | -| PocketBase 数据导入 | `F:\project\scripts\import-pocketbase` | 已支持多类旧数据归一化导入与校验 | +| PocketBase 数据导入 | `F:\project\scripts\import-pocketbase` | 已支持多类旧数据归一化导入、dry-run 报告与校验 | | 共享包 | `F:\project\packages\config`、`F:\project\packages\db`、`F:\project\packages\domain` | 已建立基础共享层 | | 旧 Web 到新 API 适配 | `F:\project\参考\旧题库项目\src\services\supabaseApi.ts` | 仅作参考；后续应在 `apps/taro` 或共享包中建立新的跨端 API client | @@ -39,7 +39,7 @@ | 租户后台 | 已建品牌、域名、设置、支付账户、登录 provider、私密密钥表、成员、审计日志、资源台账、导入台账、内容导航台账 | 不适用 | 概览、品牌、设置、域名、支付账户、登录配置、密钥掩码、活动内容、兑换码/优惠券、成员管理、权限矩阵、审计查询、内容入口/分类树/题目集合/练习蓝图维护、资源管理、题目/单词/知识手册/分数线/视频 JSON/CSV/Excel 同步/异步导入已实现 | 核心 API 集成测试含角色/权限/租户隔离/密钥不泄露/导航/组卷/资源与导入断言 | 租户配置与运营闭环可用，前端权限 UI、字段映射操作台和导入复检结果面板待补 | | 平台后台 | 已建 SaaS 套餐、订阅、账单、服务费、用量 | 不适用 | 租户管理、账单、收款确认、用量记录、平台管理员 Supabase JWT 鉴权已实现 | API 集成测试 | 平台收费链路骨架可用，平台审计报表/自动计费待补 | | 登录认证 | 已建短信验证码、会话、OAuth provider 配置表，并支持 `auth_user_id` 映射 | 旧用户映射已预留 | 短信 mock 登录、迁移期 session、Supabase JWT 验签映射、微信小程序登录主链路、微信网页登录、QQ 登录、手机号绑定/换绑已实现 | API 集成测试 | H5 Supabase Auth 可联调；真实短信/OAuth 生产账号和回调域名联调待补 | -| 数据导入 | 已建立 importer、risk report、validate | 已覆盖多类旧集合 | 命令行导入/校验 | `pb:import:validate` | 基础工具可用，需用真实完整数据做多轮 dry-run | +| 数据导入 | 已建立 importer、risk report、dry-run report、validate | 已覆盖多类旧集合 | 命令行 dry-run/导入/校验 | `pb:import:dry-run`、`pb:import:validate` | 基础工具可用，需拿真实完整数据执行多轮 dry-run 和导入回归 | | 测试体系 | 不适用 | 不适用 | 不适用 | 已新增核心 API 集成测试、租户隔离测试、权限矩阵测试、资源/题目导入测试、导入校验 | 还不是完整覆盖，支付幂等、真实导入回归、前端端到端测试仍需补 | ## 已实现 API 范围 @@ -284,7 +284,7 @@ platform-admin: 为了先把旧项目核心业务补齐，再进入支付/短信等商用关键模块，建议按下面顺序继续： -1. 完善内容导入和文件上传：字段映射 UI、真实数据 dry-run、CDN 防盗链、杀毒扫描。 +1. 完善内容导入和文件上传：字段映射 UI、真实数据 dry-run 执行验收、CDN 防盗链、杀毒扫描。 2. 补公共题库版本通知/冲突操作台、租户套餐地区/科目/题库范围限制、主题模板系统。 3. 补学习统计增强：排行榜防刷/预聚合、断点续练、专项练习策略和更细题型分析。 4. 补视频商用控制：深度防盗链、动态水印和播放统计。 diff --git a/docs/refactor/legacy-feature-gap-matrix.md b/docs/refactor/legacy-feature-gap-matrix.md index 98111685..4d7907a8 100644 --- a/docs/refactor/legacy-feature-gap-matrix.md +++ b/docs/refactor/legacy-feature-gap-matrix.md @@ -98,7 +98,7 @@ 1. 排行榜增强：刷题、模考、背单词、积分排行榜主接口已有；还需防刷、日/周榜预聚合、运营后台排名看板。 2. 账号设置完整流：绑定/更换手机号基础 API 已完成；仍缺头像上传、微信/QQ 账号合并、密码/邮箱能力。 3. 题库导出：服务端 JSON/试卷 payload 导出、权限审计和答案脱敏已补；仍缺 PDF/Word 二进制生成、水印、资料发布和后台导出操作台。 -4. 导入扩展：题目/单词/知识手册/分数线/视频已支持 JSON、CSV 和 Excel 预览导入，并可用 `executionMode=async` 进入 imports worker；导入后复检、模板下载和字段映射 API 已补，仍缺前端字段映射 UI 和真实数据 dry-run。 +4. 导入扩展：题目/单词/知识手册/分数线/视频已支持 JSON、CSV 和 Excel 预览导入，并可用 `executionMode=async` 进入 imports worker；导入后复检、模板下载、字段映射 API 和 PocketBase JSON dry-run 报告已补，仍缺前端字段映射 UI 和真实数据执行验收。 5. 公共题库商业化：平台公共/地区题库授权、租户快照采纳、手动同步、自动同步 worker、冲突查询和租户自改冲突保护已完成基础闭环；还需版本通知、冲突处理操作台和运营后台 UI。 6. CRM/销售结算：CRM worker、分佣规则、结算单、审核和打款状态基础闭环已完成；仍缺轮询/定向分配、打款导出、凭证和销售结算看板。 7. 题目反馈增强：处理通知、消息提醒、问题聚合统计和内容修复闭环。 @@ -117,7 +117,7 @@ 1. 微信/支付宝支付和 webhook 幂等。 2. 对象存储 PDF 预览、视频深度防盗链、动态水印。 -3. 字段映射 UI、真实数据 dry-run 和导入复检结果操作台。 +3. 字段映射 UI、真实数据 dry-run 执行验收和导入复检结果操作台。 4. 数据看板预聚合 worker、销售/代理转化看板和分佣结算。 5. 公共题库版本通知、冲突处理操作台和租户确认/跳过策略。 diff --git a/docs/refactor/local-supabase.md b/docs/refactor/local-supabase.md index 773c9f21..2ce79740 100644 --- a/docs/refactor/local-supabase.md +++ b/docs/refactor/local-supabase.md @@ -109,10 +109,13 @@ PORT=8787 把 PocketBase 导出的集合 JSON 放到仓库根目录的 `pb_export` 文件夹后执行： ```bash +npm run pb:import:dry-run npm run pb:import:json npm run pb:import:validate ``` +`pb:import:dry-run` 不写数据库，会先检查 JSON 格式、核心集合、旧 ID、敏感字段、关系断裂和未映射集合。存在 blocker 时先处理问题，不要直接正式导入。 + 导入真实密钥时必须明确打开开关，且只允许进入 `app_private.tenant_secrets`： ```bash diff --git a/docs/refactor/next-development-todo.md b/docs/refactor/next-development-todo.md index 6ee867b6..b464dc80 100644 --- a/docs/refactor/next-development-todo.md +++ b/docs/refactor/next-development-todo.md @@ -52,7 +52,7 @@ 3. 真实导入 dry-run - 从 PocketBase 导出现有用户、题库、单词、知识手册、分数线、订单、权益数据。 - - 跑 `scripts/import-pocketbase`，生成迁移报告。 + - 已补 `npm run pb:import:dry-run` 静态迁移报告工具；拿到真实导出后先跑 dry-run，再跑 `pb:import:json` 和 `pb:import:validate`。 - 对题目 JSON、单词、知识手册、分数线、视频走后端 preview/import API 做二次验证。 4. 部署配置 @@ -78,7 +78,7 @@ 3. 导入体系扩展 - 已完成题目、单词、知识手册、分数线、视频的 CSV/Excel 到规范 JSON 解析适配。 - 已完成大批量异步导入 worker、重试、导入后复检和审计。 - - 模板下载、字段映射 API 和导入后复检已补；继续补字段映射 UI、真实数据 dry-run、导入前抽样校验和复检结果操作台。 + - 模板下载、字段映射 API、导入后复检和 PocketBase JSON dry-run 报告工具已补；继续补字段映射 UI、真实数据执行验收、导入前抽样校验和复检结果操作台。 4. 题库导出增强 - 已完成服务端 JSON、`paper_json`、`print_payload` 基础导出和导出 job 台账。 diff --git a/docs/refactor/pocketbase-to-supabase-mapping.md b/docs/refactor/pocketbase-to-supabase-mapping.md index 498be064..b72c8f59 100644 --- a/docs/refactor/pocketbase-to-supabase-mapping.md +++ b/docs/refactor/pocketbase-to-supabase-mapping.md @@ -67,7 +67,30 @@ pb_export/ questions.json ``` -执行导入： +先执行不写数据库的 dry-run： + +```bash +npm run import:dry-run +``` + +根目录也可以执行： + +```bash +npm run pb:import:dry-run +``` + +dry-run 会检查： + +- 导出目录和 JSON 文件格式。 +- 核心集合是否缺失。 +- 旧记录 `id` 是否缺失或重复。 +- 旧 `pb_schema.json` 中的 relation 是否能在导出数据里找到目标记录。 +- `settings`、`crm_config`、用户资料等位置是否存在敏感字段。 +- 哪些集合当前已有规范化 mapper，哪些集合仍未映射。 + +存在 `blocker` 时不要执行正式导入；先修导出、补 mapper 或记录业务处理结论。 + +执行正式导入： ```bash npm run import:json @@ -82,6 +105,7 @@ npm run import:validate 根目录也可以执行： ```bash +npm run pb:import:dry-run npm run pb:import:json npm run pb:import:validate ``` diff --git a/package.json b/package.json index 187f2294..73c33869 100644 --- a/package.json +++ b/package.json @@ -22,7 +22,7 @@ "check:api": "npm --workspace @tiku-saas/api run check", "check:importer": "npm --workspace @tiku-saas/import-pocketbase run check", "check:worker": "npm --workspace @tiku-saas/worker run check", - "check:refactor": "npm run check:api && npm run check:worker && npm run check:importer && npm run pb:import:validate && npm run test:readiness && npm run test:api", + "check:refactor": "npm run check:api && npm run check:worker && npm run check:importer && npm run pb:import:validate && npm run test:readiness && npm run test:pb:dry-run && npm run test:api", "docker:api:build": "docker compose -f docker-compose.api.yml build", "docker:api:up": "docker compose -f docker-compose.api.yml up api", "docker:api:down": "docker compose -f docker-compose.api.yml down", @@ -39,11 +39,13 @@ "test:worker:imports": "npm run db:smoke-seed && npm run build:worker && node scripts/import-worker-integration-test.js", "test:worker:public-banks": "npm run db:smoke-seed && npm run build:worker && node scripts/public-bank-worker-integration-test.js", "test:readiness": "node scripts/production-readiness-check-test.js", + "test:pb:dry-run": "node scripts/pb-dry-run-report-test.js", "readiness:production": "node scripts/production-readiness-check.js --skip-db", "readiness:production:db": "node scripts/production-readiness-check.js --check-db", "test:api:remote": "node scripts/api-integration-test.js", "pb:schema:summary": "npm --workspace @tiku-saas/import-pocketbase run schema:summary", "pb:schema:risk": "npm --workspace @tiku-saas/import-pocketbase run schema:risk", + "pb:import:dry-run": "npm --workspace @tiku-saas/import-pocketbase run import:dry-run --", "pb:import:json": "npm --workspace @tiku-saas/import-pocketbase run import:json", "pb:import:validate": "npm --workspace @tiku-saas/import-pocketbase run import:validate" } diff --git a/scripts/import-pocketbase/package.json b/scripts/import-pocketbase/package.json index d3ffa8a8..ed62a892 100644 --- a/scripts/import-pocketbase/package.json +++ b/scripts/import-pocketbase/package.json @@ -6,6 +6,7 @@ "scripts": { "schema:summary": "tsx src/analyze-schema.ts", "schema:risk": "tsx src/risk-report.ts", + "import:dry-run": "tsx src/dry-run-report.ts", "import:json": "tsx src/import-json.ts", "import:validate": "tsx src/validate-import.ts", "check": "tsc -p tsconfig.json --noEmit" diff --git a/scripts/import-pocketbase/src/dry-run-report.ts b/scripts/import-pocketbase/src/dry-run-report.ts new file mode 100644 index 00000000..881e0061 --- /dev/null +++ b/scripts/import-pocketbase/src/dry-run-report.ts @@ -0,0 +1,523 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { fieldsOf, readPocketBaseSchema, type PocketBaseCollection, type PocketBaseField } from './pb-schema.js'; +import { loadEnv } from './env.js'; + +loadEnv(); + +type JsonRecord = Record & { id?: unknown }; +type Severity = 'blocker' | 'warning' | 'info'; + +interface DryRunIssue { + severity: Severity; + code: string; + collection?: string; + file?: string; + fieldPath?: string; + message: string; + sample?: string; + count?: number; +} + +interface CollectionReport { + collection: string; + file: string; + recordCount: number; + missingIdCount: number; + duplicateIdCount: number; + fieldCount: number; + sampleIds: string[]; + supported: boolean; +} + +interface DryRunReport { + exportDir: string; + schemaPath?: string; + generatedAt: string; + summary: { + files: number; + collections: number; + records: number; + supportedCollections: number; + unsupportedCollections: number; + blockers: number; + warnings: number; + infos: number; + }; + businessCounts: Record; + collections: CollectionReport[]; + issues: DryRunIssue[]; +} + +const args = new Set(process.argv.slice(2)); +const jsonOutput = args.has('--json'); +const failOnWarnings = args.has('--fail-on-warnings') || process.env.PB_DRY_RUN_FAIL_ON_WARNINGS === 'true'; +const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../..'); +const exportDir = path.resolve(repoRoot, process.env.PB_EXPORT_DIR || 'pb_export'); + +const supportedCollections = new Set([ + 'announcements', + 'app_assets', + 'audit_logs', + 'badges', + 'banners', + 'categories', + 'code_batches', + 'codes', + 'coupons', + 'coupon_redemptions', + 'crm_config', + 'crm_webhook_log', + 'crm_webhook_queue', + 'dashboard_daily_stats', + 'exam_dates', + 'faqs', + 'handbook_chapters', + 'handbook_entries', + 'handbook_subjects', + 'images', + 'majors', + 'module_nodes', + 'orders', + 'products', + 'question_type_groups', + 'question_videos', + 'questions', + 'recent_practices', + 'referral_tracks', + 'region_modules', + 'regions', + 'reports', + 'revenue_daily_stats', + 'schools', + 'scoreline_fields', + 'scoreline_majors', + 'scoreline_records', + 'scoreline_schools', + 'settings', + 'subject_shares', + 'subjects', + 'svip_plans', + 'timelines', + 'user_badges', + 'user_word_favorites', + 'user_word_progress', + 'users', + 'video_explanations', + 'vocabulary', + 'vocabulary_units', +]); + +const coreCollections = [ + 'users', + 'questions', + 'subjects', + 'categories', + 'orders', + 'svip_plans', + 'codes', + 'vocabulary_units', + 'vocabulary', + 'handbook_subjects', + 'handbook_chapters', + 'handbook_entries', +]; + +const businessCollections = [ + 'users', + 'questions', + 'orders', + 'svip_plans', + 'codes', + 'vocabulary_units', + 'vocabulary', + 'handbook_subjects', + 'handbook_chapters', + 'handbook_entries', + 'scoreline_schools', + 'scoreline_majors', + 'scoreline_records', + 'video_explanations', + 'question_videos', + 'referral_tracks', + 'coupons', + 'badges', +]; + +const sensitiveKeyPattern = + /(password|token|secret|privatekey|sessionkey|accesskey|appkey|apikey|api_v3_key|notifytoken|aeskey|openid|unionid|wxaccesstoken|wechatsessionkey|smscode|verifycode|verificationcode|captcha)/i; +const highRiskCollections = new Set(['settings', 'crm_config', 'smscodes']); + +function issue(issues: DryRunIssue[], nextIssue: DryRunIssue) { + issues.push(nextIssue); +} + +function asRecords(input: unknown): JsonRecord[] | null { + if (Array.isArray(input)) return input as JsonRecord[]; + if (input && typeof input === 'object') { + const candidate = input as { items?: unknown; records?: unknown }; + if (Array.isArray(candidate.items)) return candidate.items as JsonRecord[]; + if (Array.isArray(candidate.records)) return candidate.records as JsonRecord[]; + } + return null; +} + +function collectionNameFromFile(fileName: string) { + return fileName.replace(/\.json$/i, ''); +} + +function text(value: unknown) { + if (value === null || value === undefined) return ''; + return String(value).trim(); +} + +function readJsonFile(filePath: string) { + return JSON.parse(fs.readFileSync(filePath, 'utf8')) as unknown; +} + +function scalarSample(value: unknown) { + if (value === null || value === undefined) return ''; + if (typeof value === 'object') return JSON.stringify(value).slice(0, 160); + return String(value).slice(0, 160); +} + +function walkSensitiveFields( + issues: DryRunIssue[], + collection: string, + value: unknown, + pathParts: string[] = [], + depth = 0, +) { + if (!value || typeof value !== 'object' || depth > 8) return; + if (Array.isArray(value)) { + value.slice(0, 10).forEach((item, index) => walkSensitiveFields(issues, collection, item, [...pathParts, String(index)], depth + 1)); + return; + } + + for (const [key, childValue] of Object.entries(value as Record)) { + const fieldPath = [...pathParts, key].join('.'); + if (sensitiveKeyPattern.test(key) && childValue !== null && childValue !== undefined && childValue !== '') { + issue(issues, { + severity: highRiskCollections.has(collection) ? 'blocker' : 'warning', + code: 'sensitive_field_in_export', + collection, + fieldPath, + message: `Legacy export contains sensitive-looking field "${fieldPath}". It must not be copied into public normalized tables.`, + sample: scalarSample(childValue), + }); + } + walkSensitiveFields(issues, collection, childValue, [...pathParts, key], depth + 1); + } +} + +function fieldValues(record: JsonRecord, field: PocketBaseField) { + const value = record[field.name]; + if (value === null || value === undefined || value === '') return []; + if (Array.isArray(value)) return value.map(item => text(item)).filter(Boolean); + return [text(value)].filter(Boolean); +} + +function relationTargetName(field: PocketBaseField, byId: Map) { + if (!field.collectionId) return ''; + return byId.get(field.collectionId)?.name || ''; +} + +function validateRelations( + issues: DryRunIssue[], + collection: PocketBaseCollection, + records: JsonRecord[], + idIndex: Map>, + byId: Map, +) { + const relationFields = fieldsOf(collection).filter(field => field.type === 'relation'); + for (const field of relationFields) { + const targetName = relationTargetName(field, byId); + if (!targetName) { + issue(issues, { + severity: 'warning', + code: 'relation_target_schema_missing', + collection: collection.name, + fieldPath: field.name, + message: `Relation field "${field.name}" points to an unknown collection id in pb_schema.json.`, + }); + continue; + } + + const targetIds = idIndex.get(targetName); + if (!targetIds) { + const hasValues = records.some(record => fieldValues(record, field).length > 0); + if (hasValues) { + issue(issues, { + severity: coreCollections.includes(collection.name) ? 'blocker' : 'warning', + code: 'relation_target_export_missing', + collection: collection.name, + fieldPath: field.name, + message: `Relation field "${field.name}" references "${targetName}", but ${targetName}.json is not present in export.`, + }); + } + continue; + } + + let unresolved = 0; + let sample = ''; + for (const record of records) { + for (const value of fieldValues(record, field)) { + if (!targetIds.has(value)) { + unresolved += 1; + if (!sample) sample = `${text(record.id) || '(missing id)'} -> ${value}`; + } + } + } + + if (unresolved > 0) { + issue(issues, { + severity: coreCollections.includes(collection.name) ? 'blocker' : 'warning', + code: 'relation_unresolved', + collection: collection.name, + fieldPath: field.name, + message: `Relation field "${field.name}" has ${unresolved} unresolved reference(s) to "${targetName}".`, + count: unresolved, + sample, + }); + } + } +} + +function loadSchema(issues: DryRunIssue[]) { + try { + return readPocketBaseSchema(); + } catch (error) { + issue(issues, { + severity: 'warning', + code: 'schema_unavailable', + message: `pb_schema.json could not be loaded; relation checks will be skipped. ${error instanceof Error ? error.message : ''}`.trim(), + }); + return null; + } +} + +function collectFieldNames(records: JsonRecord[]) { + const fields = new Set(); + for (const record of records.slice(0, 500)) { + Object.keys(record).forEach(key => fields.add(key)); + } + return fields; +} + +function buildReport(): DryRunReport { + const issues: DryRunIssue[] = []; + const schema = loadSchema(issues); + const collectionByName = schema?.byName || new Map(); + const files = fs.existsSync(exportDir) ? fs.readdirSync(exportDir).filter(file => file.toLowerCase().endsWith('.json')) : []; + const collections: CollectionReport[] = []; + const recordsByCollection = new Map(); + const idIndex = new Map>(); + + if (!fs.existsSync(exportDir)) { + issue(issues, { + severity: 'blocker', + code: 'export_dir_missing', + message: `Export dir not found: ${exportDir}`, + }); + } + + if (files.length === 0) { + issue(issues, { + severity: 'blocker', + code: 'export_files_missing', + message: `No PocketBase collection JSON files found in ${exportDir}`, + }); + } + + for (const file of files.sort()) { + const collection = collectionNameFromFile(file); + const filePath = path.join(exportDir, file); + let parsed: unknown; + try { + parsed = readJsonFile(filePath); + } catch (error) { + issue(issues, { + severity: 'blocker', + code: 'json_parse_failed', + collection, + file, + message: `Failed to parse ${file}: ${error instanceof Error ? error.message : String(error)}`, + }); + continue; + } + + const records = asRecords(parsed); + if (!records) { + issue(issues, { + severity: 'blocker', + code: 'collection_shape_invalid', + collection, + file, + message: `${file} must be an array or an object with items/records array.`, + }); + continue; + } + + const ids = new Set(); + const duplicateIds = new Set(); + let missingIdCount = 0; + for (const record of records) { + const id = text(record.id); + if (!id) { + missingIdCount += 1; + continue; + } + if (ids.has(id)) duplicateIds.add(id); + ids.add(id); + } + + recordsByCollection.set(collection, records); + idIndex.set(collection, ids); + const fields = collectFieldNames(records); + collections.push({ + collection, + file, + recordCount: records.length, + missingIdCount, + duplicateIdCount: duplicateIds.size, + fieldCount: fields.size, + sampleIds: [...ids].slice(0, 5), + supported: supportedCollections.has(collection), + }); + + if (!supportedCollections.has(collection) && !collection.startsWith('_')) { + issue(issues, { + severity: 'warning', + code: 'unsupported_collection', + collection, + file, + message: `${collection}.json is not currently normalized by import-json.ts. It will be preserved only if a future mapper is added.`, + count: records.length, + }); + } + if (missingIdCount > 0) { + issue(issues, { + severity: coreCollections.includes(collection) ? 'blocker' : 'warning', + code: 'record_id_missing', + collection, + file, + message: `${collection} has ${missingIdCount} record(s) without id.`, + count: missingIdCount, + }); + } + if (duplicateIds.size > 0) { + issue(issues, { + severity: 'blocker', + code: 'record_id_duplicate', + collection, + file, + message: `${collection} has duplicate legacy id(s).`, + count: duplicateIds.size, + sample: [...duplicateIds].slice(0, 5).join(', '), + }); + } + + records.slice(0, 200).forEach(record => walkSensitiveFields(issues, collection, record)); + } + + for (const collection of coreCollections) { + if (!recordsByCollection.has(collection)) { + issue(issues, { + severity: 'warning', + code: 'core_collection_missing', + collection, + message: `${collection}.json is not present. If old production has this collection, export it before migration.`, + }); + } + } + + if (schema) { + for (const [collectionName, records] of recordsByCollection.entries()) { + const collection = collectionByName.get(collectionName); + if (collection) validateRelations(issues, collection, records, idIndex, schema.byId); + } + } + + const businessCounts = Object.fromEntries( + businessCollections.map(collection => [collection, recordsByCollection.get(collection)?.length || 0]), + ); + + const supportedCount = collections.filter(item => item.supported).length; + const unsupportedCount = collections.length - supportedCount; + const recordCount = collections.reduce((sum, item) => sum + item.recordCount, 0); + const blockers = issues.filter(item => item.severity === 'blocker').length; + const warnings = issues.filter(item => item.severity === 'warning').length; + const infos = issues.filter(item => item.severity === 'info').length; + + return { + exportDir, + schemaPath: schema?.absPath, + generatedAt: new Date().toISOString(), + summary: { + files: files.length, + collections: collections.length, + records: recordCount, + supportedCollections: supportedCount, + unsupportedCollections: unsupportedCount, + blockers, + warnings, + infos, + }, + businessCounts, + collections: collections.sort((a, b) => a.collection.localeCompare(b.collection)), + issues: issues.sort((a, b) => { + const order: Record = { blocker: 0, warning: 1, info: 2 }; + return order[a.severity] - order[b.severity] || (a.collection || '').localeCompare(b.collection || '') || a.code.localeCompare(b.code); + }), + }; +} + +function printHuman(report: DryRunReport) { + console.log('PocketBase dry-run migration report'); + console.log(`Export dir: ${report.exportDir}`); + if (report.schemaPath) console.log(`Schema: ${report.schemaPath}`); + console.log( + `Summary: ${report.summary.records} records across ${report.summary.collections} collections; ` + + `${report.summary.blockers} blocker(s), ${report.summary.warnings} warning(s)`, + ); + console.log(''); + console.log('Business counts:'); + for (const [name, count] of Object.entries(report.businessCounts)) { + console.log(` ${name.padEnd(24)} ${count}`); + } + console.log(''); + console.log('Collections:'); + for (const item of report.collections) { + const marker = item.supported ? 'mapped' : 'unmapped'; + console.log( + ` ${item.collection.padEnd(24)} records=${String(item.recordCount).padStart(6)} ` + + `fields=${String(item.fieldCount).padStart(3)} ${marker}`, + ); + } + console.log(''); + console.log('Issues:'); + if (report.issues.length === 0) { + console.log(' none'); + } else { + for (const item of report.issues.slice(0, 200)) { + const location = [item.collection, item.fieldPath].filter(Boolean).join('.') || item.file || 'export'; + const count = item.count === undefined ? '' : ` (${item.count})`; + const sample = item.sample ? ` sample=${item.sample}` : ''; + console.log(` [${item.severity.toUpperCase()}] ${item.code} ${location}: ${item.message}${count}${sample}`); + } + if (report.issues.length > 200) console.log(` ... ${report.issues.length - 200} more issue(s) omitted`); + } +} + +function main() { + const report = buildReport(); + if (jsonOutput) console.log(JSON.stringify(report, null, 2)); + else printHuman(report); + + if (report.summary.blockers > 0 || (failOnWarnings && report.summary.warnings > 0)) { + process.exitCode = 1; + } +} + +main(); diff --git a/scripts/pb-dry-run-report-test.js b/scripts/pb-dry-run-report-test.js new file mode 100644 index 00000000..c6517b2d --- /dev/null +++ b/scripts/pb-dry-run-report-test.js @@ -0,0 +1,122 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { spawnSync } from 'node:child_process'; + +const repoRoot = process.cwd(); +const dryRunScript = path.join(repoRoot, 'scripts', 'import-pocketbase', 'src', 'dry-run-report.ts'); + +function writeJson(filePath, value) { + fs.writeFileSync(filePath, JSON.stringify(value, null, 2), 'utf8'); +} + +function runDryRun(exportDir) { + const result = spawnSync(process.execPath, ['--import', 'tsx', dryRunScript, '--json'], { + cwd: repoRoot, + encoding: 'utf8', + env: { + ...process.env, + PB_EXPORT_DIR: exportDir, + }, + }); + + if (result.error) throw result.error; + const stdout = result.stdout || ''; + const jsonStart = stdout.indexOf('{'); + const payload = JSON.parse(jsonStart >= 0 ? stdout.slice(jsonStart) : '{}'); + return { ...result, payload }; +} + +function createSafeExport() { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'tiku-pb-dry-safe-')); + writeJson(path.join(dir, 'users.json'), [ + { id: 'u1', phone: '13800000000', role: 'student', nickname: '学生A' }, + ]); + writeJson(path.join(dir, 'subjects.json'), [ + { id: 's1', name: '英语' }, + ]); + writeJson(path.join(dir, 'categories.json'), [ + { id: 'c1', name: '阅读理解', subjectId: 's1' }, + ]); + writeJson(path.join(dir, 'questions.json'), [ + { id: 'q1', subjectId: 's1', categoryId: 'c1', type: 'choice', content: '题干', answer: 'A' }, + ]); + writeJson(path.join(dir, 'orders.json'), [ + { id: 'o1', userId: 'u1', amount: 99, status: 'paid' }, + ]); + writeJson(path.join(dir, 'svip_plans.json'), [ + { id: 'p1', name: '题库会员', price: 99 }, + ]); + writeJson(path.join(dir, 'codes.json'), [ + { id: 'code1', code: 'SAFE-001', status: 'unused' }, + ]); + writeJson(path.join(dir, 'vocabulary_units.json'), [ + { id: 'vu1', name: 'Unit 1' }, + ]); + writeJson(path.join(dir, 'vocabulary.json'), [ + { id: 'vw1', unitId: 'vu1', word: 'abandon', meaning: '放弃' }, + ]); + writeJson(path.join(dir, 'handbook_subjects.json'), [ + { id: 'hs1', name: '政治' }, + ]); + writeJson(path.join(dir, 'handbook_chapters.json'), [ + { id: 'hc1', subjectId: 'hs1', title: '第一章' }, + ]); + writeJson(path.join(dir, 'handbook_entries.json'), [ + { id: 'he1', chapterId: 'hc1', title: '知识点', content: '内容' }, + ]); + return dir; +} + +function createUnsafeExport() { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'tiku-pb-dry-unsafe-')); + writeJson(path.join(dir, 'users.json'), [ + { id: 'u1', phone: '13800000000', password: 'plaintext-should-not-exist' }, + { phone: '13900000000' }, + ]); + writeJson(path.join(dir, 'questions.json'), [ + { id: 'q1', subjectId: 'missing-subject', content: '题干' }, + { id: 'q1', subjectId: 'missing-subject', content: '重复题' }, + ]); + writeJson(path.join(dir, 'settings.json'), [ + { id: 'settings1', appSecret: 'secret-in-public-export' }, + ]); + writeJson(path.join(dir, 'unknown_business.json'), [ + { id: 'x1', name: '尚未映射集合' }, + ]); + return dir; +} + +const safeDir = createSafeExport(); +const safe = runDryRun(safeDir); +assert.equal(safe.status, 0, `safe dry-run should pass: ${safe.stdout} ${safe.stderr}`); +assert.equal(safe.payload.summary?.blockers, 0, 'safe dry-run should have no blockers'); +assert.ok(safe.payload.summary?.records >= 12, 'safe dry-run should count fixture records'); +assert.equal(safe.payload.businessCounts?.questions, 1, 'safe dry-run should report question count'); + +const unsafeDir = createUnsafeExport(); +const unsafe = runDryRun(unsafeDir); +assert.notEqual(unsafe.status, 0, 'unsafe dry-run should fail'); +assert.ok(unsafe.payload.summary?.blockers > 0, 'unsafe dry-run should report blockers'); +assert.ok( + unsafe.payload.issues?.some(item => item.code === 'record_id_missing' && item.collection === 'users'), + 'unsafe dry-run should detect missing user id', +); +assert.ok( + unsafe.payload.issues?.some(item => item.code === 'record_id_duplicate' && item.collection === 'questions'), + 'unsafe dry-run should detect duplicate question id', +); +assert.ok( + unsafe.payload.issues?.some(item => item.code === 'sensitive_field_in_export'), + 'unsafe dry-run should detect sensitive fields', +); +assert.ok( + unsafe.payload.issues?.some(item => item.code === 'unsupported_collection'), + 'unsafe dry-run should warn about unsupported collections', +); + +fs.rmSync(safeDir, { recursive: true, force: true }); +fs.rmSync(unsafeDir, { recursive: true, force: true }); + +console.log('[PASS] PocketBase dry-run report'); diff --git a/scripts/smoke-seed.js b/scripts/smoke-seed.js index 0b440906..dd6caf91 100644 --- a/scripts/smoke-seed.js +++ b/scripts/smoke-seed.js @@ -87,6 +87,109 @@ async function main() { try { await client.query('begin'); + await client.query( + ` + create temporary table smoke_seed_transient_questions ( + id uuid primary key + ) on commit drop + `, + ); + + await client.query( + ` + insert into smoke_seed_transient_questions (id) + select distinct q.id + from public.questions q + left join public.question_collection_items ci + on ci.tenant_id = q.tenant_id + and ci.question_id = q.id + where q.tenant_id = $1 + and q.id <> all($3::uuid[]) + and ( + q.primary_collection_id = $2::uuid + or ci.collection_id = $2::uuid + or coalesce(q.legacy_id, '') like any($4::text[]) + ) + `, + [ + tenantId, + ids.questionCollection, + [ids.question, ids.questionTwo, ids.questionThree], + ['integration-import-%', 'worker-import-question-%', 'public-sync-source-%', 'content-import:%'], + ], + ); + + await client.query( + ` + delete from public.question_collection_items + where tenant_id = $1 + and ( + question_id in (select id from smoke_seed_transient_questions) + or ( + collection_id = $2::uuid + and question_id <> all($3::uuid[]) + ) + ) + `, + [tenantId, ids.questionCollection, [ids.question, ids.questionTwo, ids.questionThree]], + ); + + await client.query( + ` + delete from public.question_versions + where tenant_id = $1 + and question_id in (select id from smoke_seed_transient_questions) + `, + [tenantId], + ); + + await client.query( + ` + delete from public.questions + where tenant_id = $1 + and id in (select id from smoke_seed_transient_questions) + `, + [tenantId], + ); + + await client.query( + ` + delete from public.content_export_jobs + where tenant_id = $1 + and scope_id = $2::uuid + `, + [tenantId, ids.questionCollection], + ); + + await client.query( + ` + delete from public.content_import_jobs + where tenant_id = $1 + and ( + source_name = any($2::text[]) + or source_name like 'worker-import-%' + ) + `, + [ + tenantId, + [ + 'large-but-allowed-question-import.json', + 'invalid-question-import.json', + 'valid-question-import.json', + 'questions.csv', + 'async-question-import.json', + 'invalid-vocabulary-import.json', + 'legacy-vocabulary-import.json', + 'vocabulary.csv', + 'handbook-nested-import.json', + 'scoreline-batch-import.json', + 'scoreline-mixed-items.json', + 'scoreline.xlsx', + 'video-batch-import.json', + ], + ], + ); + await client.query( ` delete from public.report_status_events