forked from wangziqi/gongxue-base
test: harden pocketbase dry run readiness
This commit is contained in:
@@ -320,16 +320,18 @@ npm run pb:import:dry-run -- --json
|
|||||||
|
|
||||||
dry-run 会检查导出目录、JSON 形态、核心集合缺失、重复/缺失旧 ID、敏感字段、旧 schema 关系断裂和未映射集合。存在 blocker 时命令返回非 0;所有 blocker 处理完后,再执行 `npm run pb:import:json` 和 `npm run pb:import:validate`。
|
dry-run 会检查导出目录、JSON 形态、核心集合缺失、重复/缺失旧 ID、敏感字段、旧 schema 关系断裂和未映射集合。存在 blocker 时命令返回非 0;所有 blocker 处理完后,再执行 `npm run pb:import:json` 和 `npm run pb:import:validate`。
|
||||||
|
|
||||||
|
默认 dry-run 使用 `development` profile;正式迁移、预生产验收和 CI 应使用 `production` profile。生产 profile 会额外输出 `migrationReadiness`,检查用户、题目、科目、分类、订单、套餐、激活码、单词和知识手册等必需集合,以及用户手机号、题目归属、订单套餐、激活码、单词和手册归属等关键字段覆盖率。`--profile` 只接受 `development` 或 `production`,拼写错误会按 blocker 失败。
|
||||||
|
|
||||||
真实生产数据迁移不要只看命令是否能跑完,需要按迁移验收 runbook 执行 dry-run、正式导入演练、导入后校验、业务抽样、Taro 联调、冻结切换和回滚准备:
|
真实生产数据迁移不要只看命令是否能跑完,需要按迁移验收 runbook 执行 dry-run、正式导入演练、导入后校验、业务抽样、Taro 联调、冻结切换和回滚准备:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
docs/refactor/pocketbase-real-data-migration-runbook.md
|
docs/refactor/pocketbase-real-data-migration-runbook.md
|
||||||
```
|
```
|
||||||
|
|
||||||
正式切换前建议使用严格模式:
|
正式切换前建议使用 production 严格模式:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm run pb:import:dry-run -- --json --fail-on-warnings
|
npm run pb:import:dry-run -- --profile=production --json --fail-on-warnings
|
||||||
```
|
```
|
||||||
|
|
||||||
导入后校验建议在预生产/生产切换前把 warning 也作为阻断:
|
导入后校验建议在预生产/生产切换前把 warning 也作为阻断:
|
||||||
|
|||||||
@@ -167,7 +167,7 @@
|
|||||||
| 能力 | 状态 | 说明 |
|
| 能力 | 状态 | 说明 |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| PocketBase schema/导出分析 | 可联调 | `scripts/import-pocketbase` 支持 schema summary/risk、`npm run pb:import:dry-run` 导出目录静态迁移报告 |
|
| PocketBase schema/导出分析 | 可联调 | `scripts/import-pocketbase` 支持 schema summary/risk、`npm run pb:import:dry-run` 导出目录静态迁移报告 |
|
||||||
| PocketBase JSON dry-run | 可联调 | 不写数据库,检查导出目录、JSON 形态、核心集合、旧 ID、敏感字段、schema relation、未映射集合和关键业务计数 |
|
| PocketBase JSON dry-run | 可联调 | 不写数据库,检查导出目录、JSON 形态、核心集合、旧 ID、敏感字段、schema relation、未映射集合和关键业务计数;`--profile=production` 会额外检查生产迁移必需集合和关键字段覆盖率,正式切换建议配合 `--fail-on-warnings` |
|
||||||
| 题目 JSON preview/import | 可联调 | 后端负责规范化、issue、幂等、审计 |
|
| 题目 JSON preview/import | 可联调 | 后端负责规范化、issue、幂等、审计 |
|
||||||
| 公共题库采纳、手动同步和自动同步 | 可联调 | 平台授权后,租户可采纳公共题库并复制已发布题目快照;同步 API 和 `public-banks` worker 支持新增/更新题目、重新校验授权、跨租户拒绝、审计记录、租户内容通知和租户自改冲突保护;冲突处理 API 已支持单条/批量采纳平台版本和保留租户本地版本;已覆盖跨租户、重复采纳、采纳后组卷、同步新增题、通知隔离/已读/自动 resolved、冲突不覆盖、单条/批量冲突处理和 worker 自动同步测试 |
|
| 公共题库采纳、手动同步和自动同步 | 可联调 | 平台授权后,租户可采纳公共题库并复制已发布题目快照;同步 API 和 `public-banks` worker 支持新增/更新题目、重新校验授权、跨租户拒绝、审计记录、租户内容通知和租户自改冲突保护;冲突处理 API 已支持单条/批量采纳平台版本和保留租户本地版本;已覆盖跨租户、重复采纳、采纳后组卷、同步新增题、通知隔离/已读/自动 resolved、冲突不覆盖、单条/批量冲突处理和 worker 自动同步测试 |
|
||||||
| 单词 JSON preview/import | 可联调 | 兼容旧模板 |
|
| 单词 JSON preview/import | 可联调 | 兼容旧模板 |
|
||||||
|
|||||||
@@ -77,7 +77,7 @@
|
|||||||
|
|
||||||
- 生产鉴权:API 已支持 Supabase Auth JWT、HS256 本地测试和 JWKS/RS256 集成回归;`npm run test:rls` 已补本地运行时 RLS 深测;`npm run smoke:auth:remote` 已补真实云端 Auth/JWKS smoke 命令。继续在预生产/生产执行真实 token 回归和生产数据库 RLS 验收,并在生产关闭 `x-user-id` 与 `x-platform-admin-key` 兼容入口。生产配置 JWKS 时必须同时配置 `AUTH_JWT_ISSUER`。
|
- 生产鉴权:API 已支持 Supabase Auth JWT、HS256 本地测试和 JWKS/RS256 集成回归;`npm run test:rls` 已补本地运行时 RLS 深测;`npm run smoke:auth:remote` 已补真实云端 Auth/JWKS smoke 命令。继续在预生产/生产执行真实 token 回归和生产数据库 RLS 验收,并在生产关闭 `x-user-id` 与 `x-platform-admin-key` 兼容入口。生产配置 JWKS 时必须同时配置 `AUTH_JWT_ISSUER`。
|
||||||
- 对象存储:上传/下载签名已接入阿里云 OSS、腾讯云 COS、Supabase Storage;上传确认、PDF/图片预览签名、动态水印上下文、assets worker 复检、内置安全扫描、外部 HTTP scanner 接入层和题库导出 PDF/Word/每日一练 ZIP worker 已完成,继续补视频播放防盗链、真实 AV/内容安全服务联调和转码/CDN 级水印。
|
- 对象存储:上传/下载签名已接入阿里云 OSS、腾讯云 COS、Supabase Storage;上传确认、PDF/图片预览签名、动态水印上下文、assets worker 复检、内置安全扫描、外部 HTTP scanner 接入层和题库导出 PDF/Word/每日一练 ZIP worker 已完成,继续补视频播放防盗链、真实 AV/内容安全服务联调和转码/CDN 级水印。
|
||||||
- 真实数据 dry-run:导出 PocketBase 用户、题库、单词、知识手册、分数线、订单、权益,先跑 `npm run pb:import:dry-run`,再跑迁移和校验报告。
|
- 真实数据 dry-run:导出 PocketBase 用户、题库、单词、知识手册、分数线、订单、权益,先跑 `npm run pb:import:dry-run -- --profile=production --json --fail-on-warnings`,确认 `migrationReadiness` 的必需集合和关键字段覆盖率通过,再跑迁移和校验报告。
|
||||||
- 生产环境配置:`.env.example` 和 `npm run readiness:production` / `npm run readiness:production:db` 已补;继续补数据库迁移流程、备份恢复、日志、告警和 API 容器部署说明。
|
- 生产环境配置:`.env.example` 和 `npm run readiness:production` / `npm run readiness:production:db` 已补;继续补数据库迁移流程、备份恢复、日志、告警和 API 容器部署说明。
|
||||||
- Taro scaffold:`apps/taro` 地基已建立;学生端、租户后台、平台后台第一批 H5 页面已接真实 API,学生端已接地区选择、错题/收藏复习、阅读理解/案例分析多小题作答、题目反馈、视频解析、练习/模考报告、收银台、订单详情和售后入口第一版,平台后台关键写操作第一版已接入,租户工作台已接权限驱动模块入口,租户学生运营页已接学生创建/更新、禁用/恢复、批量导入、批量分班、备注和跟进任务第一版,租户内容页已接公共题库采纳/同步、冲突查看、单条/批量采纳平台或保留本地、导入问题、字段模板预览/下载、上传/粘贴预览、字段别名覆盖、同步/异步导入、异步轮询和复检详情第一版,租户设置页已接角色模板创建/编辑/停用、成员绑定模板和权限可见性配置第一版,租户营销中心已接 CRM 配置/队列和分佣结算操作台第一版;下一步补公式图片混排、更细数据范围 UI、平台后台审计增强和小程序兼容验证。
|
- Taro scaffold:`apps/taro` 地基已建立;学生端、租户后台、平台后台第一批 H5 页面已接真实 API,学生端已接地区选择、错题/收藏复习、阅读理解/案例分析多小题作答、题目反馈、视频解析、练习/模考报告、收银台、订单详情和售后入口第一版,平台后台关键写操作第一版已接入,租户工作台已接权限驱动模块入口,租户学生运营页已接学生创建/更新、禁用/恢复、批量导入、批量分班、备注和跟进任务第一版,租户内容页已接公共题库采纳/同步、冲突查看、单条/批量采纳平台或保留本地、导入问题、字段模板预览/下载、上传/粘贴预览、字段别名覆盖、同步/异步导入、异步轮询和复检详情第一版,租户设置页已接角色模板创建/编辑/停用、成员绑定模板和权限可见性配置第一版,租户营销中心已接 CRM 配置/队列和分佣结算操作台第一版;下一步补公式图片混排、更细数据范围 UI、平台后台审计增强和小程序兼容验证。
|
||||||
|
|
||||||
@@ -107,7 +107,7 @@
|
|||||||
- `npm run check:refactor` 在本地通过。
|
- `npm run check:refactor` 在本地通过。
|
||||||
- `npm run test:rls` 在本地通过,且新增租户表的 RLS 动态用例已补齐或有明确豁免说明。
|
- `npm run test:rls` 在本地通过,且新增租户表的 RLS 动态用例已补齐或有明确豁免说明。
|
||||||
- `npm run smoke:auth:remote` 在预生产/生产 API 上通过,真实 Supabase JWT 能映射学生、租户管理员和平台管理员,坏 token/错租户上下文被拒绝。
|
- `npm run smoke:auth:remote` 在预生产/生产 API 上通过,真实 Supabase JWT 能映射学生、租户管理员和平台管理员,坏 token/错租户上下文被拒绝。
|
||||||
- 使用真实 PocketBase 导出数据完成一次 dry-run,产出问题清单和修复后的二次导入报告。
|
- 使用真实 PocketBase 导出数据完成一次 production dry-run,产出问题清单和修复后的二次导入报告。
|
||||||
- 核心学生链路 API 可以在 Taro H5 和小程序双端跑通:登录、首页、题库、练习、错题、收藏、单词、手册、会员、排行榜、个人中心。
|
- 核心学生链路 API 可以在 Taro H5 和小程序双端跑通:登录、首页、题库、练习、错题、收藏、单词、手册、会员、排行榜、个人中心。
|
||||||
- 租户隔离、后台角色权限、资源访问权限、订单权益和内容导入至少有集成测试覆盖。
|
- 租户隔离、后台角色权限、资源访问权限、订单权益和内容导入至少有集成测试覆盖。
|
||||||
|
|
||||||
|
|||||||
@@ -67,7 +67,7 @@
|
|||||||
|
|
||||||
3. 真实导入 dry-run
|
3. 真实导入 dry-run
|
||||||
- 从 PocketBase 导出现有用户、题库、单词、知识手册、分数线、订单、权益数据。
|
- 从 PocketBase 导出现有用户、题库、单词、知识手册、分数线、订单、权益数据。
|
||||||
- 已补 `npm run pb:import:dry-run` 静态迁移报告工具、strict warning 门禁测试和真实数据迁移验收 runbook;拿到真实导出后先跑 strict dry-run,再跑 `pb:import:json`、`pb:import:validate` 和业务抽样。
|
- 已补 `npm run pb:import:dry-run` 静态迁移报告工具、`--profile=production` 生产迁移门禁、关键集合/关键字段覆盖率检查、strict warning 门禁测试和真实数据迁移验收 runbook;拿到真实导出后先跑 `npm run pb:import:dry-run -- --profile=production --json --fail-on-warnings`,再跑 `pb:import:json`、`pb:import:validate` 和业务抽样。
|
||||||
- 对题目 JSON、单词、知识手册、分数线、视频走后端 preview/import API 做二次验证。
|
- 对题目 JSON、单词、知识手册、分数线、视频走后端 preview/import API 做二次验证。
|
||||||
|
|
||||||
4. 部署配置
|
4. 部署配置
|
||||||
@@ -237,6 +237,6 @@
|
|||||||
2. 继续补 Taro 学生端旧体验:地区选择、刷题答题卡、后端权威断点续练、本地进度恢复、模拟倒计时、主观题后端自评、阅读理解/案例分析多小题、视频播放、反馈、模考报告、逐题复盘、错题/收藏专题、个人中心学习报告、收银台、订单详情、售后入口、题干/解析/知识手册 RichContent 安全渲染、H5 KaTeX 公式渲染、私有资源 ID 题图短签名、背单词卡片学习/发音/收藏练习、资料短签名水印预览/下载确认已接第一版;继续补小程序公式真机验收、题图资源字段化、个人中心任务系统、背单词更细统计、小程序支付容器、分享场景和状态管理。
|
2. 继续补 Taro 学生端旧体验:地区选择、刷题答题卡、后端权威断点续练、本地进度恢复、模拟倒计时、主观题后端自评、阅读理解/案例分析多小题、视频播放、反馈、模考报告、逐题复盘、错题/收藏专题、个人中心学习报告、收银台、订单详情、售后入口、题干/解析/知识手册 RichContent 安全渲染、H5 KaTeX 公式渲染、私有资源 ID 题图短签名、背单词卡片学习/发音/收藏练习、资料短签名水印预览/下载确认已接第一版;继续补小程序公式真机验收、题图资源字段化、个人中心任务系统、背单词更细统计、小程序支付容器、分享场景和状态管理。
|
||||||
3. 补平台后台增强:租户详情/编辑、平台审计报表、自动计费、账单批量操作和更细平台权限点。
|
3. 补平台后台增强:租户详情/编辑、平台审计报表、自动计费、账单批量操作和更细平台权限点。
|
||||||
4. 云服务器部署 Supabase/PostgreSQL 和 API,配置对象存储生产环境变量,跑 `check:refactor` 的远程等价测试。
|
4. 云服务器部署 Supabase/PostgreSQL 和 API,配置对象存储生产环境变量,跑 `check:refactor` 的远程等价测试。
|
||||||
5. 导出现有 PocketBase 数据,按 `docs/refactor/pocketbase-real-data-migration-runbook.md` 做完整 dry-run、导入演练、校验和抽样验收。
|
5. 导出现有 PocketBase 数据,按 `docs/refactor/pocketbase-real-data-migration-runbook.md` 做 production dry-run、导入演练、校验和抽样验收。
|
||||||
6. 并行补真实登录、真实生产账单格式验收、异常订单运营台、对象存储真实 AV/内容安全服务联调、转码/CDN 级水印/生命周期、题库导出模板精排/操作台、公共题库生产定时调度和失败告警。
|
6. 并行补真实登录、真实生产账单格式验收、异常订单运营台、对象存储真实 AV/内容安全服务联调、转码/CDN 级水印/生命周期、题库导出模板精排/操作台、公共题库生产定时调度和失败告警。
|
||||||
7. 前后端联调通过后,再做支付、权限、数据导入、资料下载、视频播放的商用验收。
|
7. 前后端联调通过后,再做支付、权限、数据导入、资料下载、视频播放的商用验收。
|
||||||
|
|||||||
@@ -84,7 +84,7 @@ npm run pb:import:dry-run
|
|||||||
|
|
||||||
## 阶段 1:静态 Dry-Run
|
## 阶段 1:静态 Dry-Run
|
||||||
|
|
||||||
先运行不写数据库的静态报告:
|
先运行不写数据库的静态报告。默认是 `development` profile,适合开发环境快速发现 JSON 形态、关系和敏感字段问题:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm run pb:import:dry-run
|
npm run pb:import:dry-run
|
||||||
@@ -96,12 +96,22 @@ npm run pb:import:dry-run
|
|||||||
npm run pb:import:dry-run -- --json > migration-dry-run-report.json
|
npm run pb:import:dry-run -- --json > migration-dry-run-report.json
|
||||||
```
|
```
|
||||||
|
|
||||||
严格模式会把 warning 也作为阻断条件,建议预生产验收和 CI 使用:
|
预生产验收、最终切换和 CI 必须使用 `production` profile,并建议同时打开 warning 阻断:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm run pb:import:dry-run -- --json --fail-on-warnings
|
npm run pb:import:dry-run -- --profile=production --json --fail-on-warnings
|
||||||
```
|
```
|
||||||
|
|
||||||
|
也可以用环境变量指定 profile:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
$env:PB_DRY_RUN_PROFILE="production"
|
||||||
|
npm run pb:import:dry-run -- --json --fail-on-warnings
|
||||||
|
Remove-Item Env:\PB_DRY_RUN_PROFILE
|
||||||
|
```
|
||||||
|
|
||||||
|
`--profile` 只接受 `development` 或 `production`,拼写错误会 fail closed 并返回 blocker,避免正式迁移时误用开发模式。
|
||||||
|
|
||||||
dry-run 会检查:
|
dry-run 会检查:
|
||||||
|
|
||||||
- 导出目录是否存在。
|
- 导出目录是否存在。
|
||||||
@@ -112,12 +122,15 @@ dry-run 会检查:
|
|||||||
- 敏感字段是否出现在旧导出中。
|
- 敏感字段是否出现在旧导出中。
|
||||||
- 未映射集合是否需要补 mapper。
|
- 未映射集合是否需要补 mapper。
|
||||||
- 用户、题目、订单、SVIP、激活码、单词、手册、分数线、视频等业务数量。
|
- 用户、题目、订单、SVIP、激活码、单词、手册、分数线、视频等业务数量。
|
||||||
|
- `production` profile 会额外检查生产迁移必需集合:`users`、`questions`、`subjects`、`categories`、`orders`、`svip_plans`、`codes`、`vocabulary_units`、`vocabulary`、`handbook_subjects`、`handbook_chapters`、`handbook_entries`。
|
||||||
|
- `migrationReadiness.criticalFieldCoverage` 会统计关键字段覆盖率,例如 `users.phone`、`questions.subjectId/categoryId/content`、`orders.userId/planId/status`、`codes.code`、单词和手册的归属字段;生产模式下低于阈值会变成 blocker。
|
||||||
|
|
||||||
准入标准:
|
准入标准:
|
||||||
|
|
||||||
- `blockers = 0`。
|
- `blockers = 0`。
|
||||||
- 正式切换前建议 `warnings = 0`;如确有历史脏数据,需要记录处理结论、影响范围和接受人。
|
- 正式切换前建议 `warnings = 0`;如确有历史脏数据,需要记录处理结论、影响范围和接受人。
|
||||||
- `businessCounts` 与旧后台统计口径差异必须能解释。
|
- `businessCounts` 与旧后台统计口径差异必须能解释。
|
||||||
|
- 生产切换前 `migrationProfile` 必须是 `production`,且 `migrationReadiness.requiredCollections` 不能有缺失或记录数不足。
|
||||||
|
|
||||||
## 阶段 2:正式导入演练
|
## 阶段 2:正式导入演练
|
||||||
|
|
||||||
@@ -217,7 +230,7 @@ npm run dev:api
|
|||||||
1. 公告维护窗口。
|
1. 公告维护窗口。
|
||||||
2. 旧 PocketBase 进入只读或暂停写入。
|
2. 旧 PocketBase 进入只读或暂停写入。
|
||||||
3. 导出最终 JSON。
|
3. 导出最终 JSON。
|
||||||
4. 执行 strict dry-run。
|
4. 执行 production strict dry-run。
|
||||||
5. 重置目标生产库或清理目标租户迁移数据。
|
5. 重置目标生产库或清理目标租户迁移数据。
|
||||||
6. 执行正式导入。
|
6. 执行正式导入。
|
||||||
7. 执行 `pb:import:validate` strict 模式。
|
7. 执行 `pb:import:validate` strict 模式。
|
||||||
@@ -240,11 +253,10 @@ npm run dev:api
|
|||||||
|
|
||||||
满足以下条件后,才建议进入生产切换:
|
满足以下条件后,才建议进入生产切换:
|
||||||
|
|
||||||
- `npm run pb:import:dry-run -- --json --fail-on-warnings` 通过,或全部 warning 有签字确认的处理结论。
|
- `npm run pb:import:dry-run -- --profile=production --json --fail-on-warnings` 通过,或全部 warning 有签字确认的处理结论。
|
||||||
- `npm run pb:import:validate` 无 FAIL;生产切换前 strict 模式无 WARN,或 WARN 已确认。
|
- `npm run pb:import:validate` 无 FAIL;生产切换前 strict 模式无 WARN,或 WARN 已确认。
|
||||||
- 核心业务抽样通过,尤其是题目答案解析、会员权益、订单支付、错题收藏、资料视频权限。
|
- 核心业务抽样通过,尤其是题目答案解析、会员权益、订单支付、错题收藏、资料视频权限。
|
||||||
- `npm run readiness:production` 和 `npm run readiness:production:db` 通过。
|
- `npm run readiness:production` 和 `npm run readiness:production:db` 通过。
|
||||||
- API/Taro 三端核心链路在迁移数据上跑通。
|
- API/Taro 三端核心链路在迁移数据上跑通。
|
||||||
- 对象存储、短信、OAuth、支付、CRM webhook 使用生产 provider 或明确标记为暂不开启。
|
- 对象存储、短信、OAuth、支付、CRM webhook 使用生产 provider 或明确标记为暂不开启。
|
||||||
- 已准备数据库备份、旧系统快照、回滚步骤和负责人。
|
- 已准备数据库备份、旧系统快照、回滚步骤和负责人。
|
||||||
|
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ loadEnv();
|
|||||||
|
|
||||||
type JsonRecord = Record<string, unknown> & { id?: unknown };
|
type JsonRecord = Record<string, unknown> & { id?: unknown };
|
||||||
type Severity = 'blocker' | 'warning' | 'info';
|
type Severity = 'blocker' | 'warning' | 'info';
|
||||||
|
type MigrationProfile = 'development' | 'production';
|
||||||
|
|
||||||
interface DryRunIssue {
|
interface DryRunIssue {
|
||||||
severity: Severity;
|
severity: Severity;
|
||||||
@@ -35,6 +36,7 @@ interface DryRunReport {
|
|||||||
exportDir: string;
|
exportDir: string;
|
||||||
schemaPath?: string;
|
schemaPath?: string;
|
||||||
generatedAt: string;
|
generatedAt: string;
|
||||||
|
migrationProfile: MigrationProfile;
|
||||||
summary: {
|
summary: {
|
||||||
files: number;
|
files: number;
|
||||||
collections: number;
|
collections: number;
|
||||||
@@ -46,13 +48,37 @@ interface DryRunReport {
|
|||||||
infos: number;
|
infos: number;
|
||||||
};
|
};
|
||||||
businessCounts: Record<string, number>;
|
businessCounts: Record<string, number>;
|
||||||
|
migrationReadiness: {
|
||||||
|
requiredCollections: Array<{
|
||||||
|
collection: string;
|
||||||
|
present: boolean;
|
||||||
|
recordCount: number;
|
||||||
|
minimumRecords: number;
|
||||||
|
}>;
|
||||||
|
criticalFieldCoverage: Array<{
|
||||||
|
collection: string;
|
||||||
|
field: string;
|
||||||
|
present: boolean;
|
||||||
|
missingCount: number;
|
||||||
|
presentCount: number;
|
||||||
|
total: number;
|
||||||
|
missingRatio: number;
|
||||||
|
requiredRatio: number;
|
||||||
|
}>;
|
||||||
|
};
|
||||||
collections: CollectionReport[];
|
collections: CollectionReport[];
|
||||||
issues: DryRunIssue[];
|
issues: DryRunIssue[];
|
||||||
}
|
}
|
||||||
|
|
||||||
const args = new Set(process.argv.slice(2));
|
const rawArgs = process.argv.slice(2);
|
||||||
|
const args = new Set(rawArgs);
|
||||||
const jsonOutput = args.has('--json');
|
const jsonOutput = args.has('--json');
|
||||||
const failOnWarnings = args.has('--fail-on-warnings') || process.env.PB_DRY_RUN_FAIL_ON_WARNINGS === 'true';
|
const failOnWarnings = args.has('--fail-on-warnings') || process.env.PB_DRY_RUN_FAIL_ON_WARNINGS === 'true';
|
||||||
|
const profileArg = rawArgs.find(arg => arg.startsWith('--profile='))?.split('=')[1]?.trim();
|
||||||
|
const migrationProfileInput = (profileArg || process.env.PB_DRY_RUN_PROFILE || 'development').trim().toLowerCase();
|
||||||
|
const validMigrationProfiles = new Set(['development', 'production']);
|
||||||
|
const migrationProfile: MigrationProfile = migrationProfileInput === 'production' ? 'production' : 'development';
|
||||||
|
const migrationProfileInvalid = !validMigrationProfiles.has(migrationProfileInput);
|
||||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../..');
|
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../..');
|
||||||
const exportDir = path.resolve(repoRoot, process.env.PB_EXPORT_DIR || 'pb_export');
|
const exportDir = path.resolve(repoRoot, process.env.PB_EXPORT_DIR || 'pb_export');
|
||||||
|
|
||||||
@@ -145,6 +171,43 @@ const businessCollections = [
|
|||||||
'badges',
|
'badges',
|
||||||
];
|
];
|
||||||
|
|
||||||
|
const productionRequiredCollections = [
|
||||||
|
{ collection: 'users', minimumRecords: 1 },
|
||||||
|
{ collection: 'questions', minimumRecords: 1 },
|
||||||
|
{ collection: 'subjects', minimumRecords: 1 },
|
||||||
|
{ collection: 'categories', minimumRecords: 1 },
|
||||||
|
{ collection: 'orders', minimumRecords: 1 },
|
||||||
|
{ collection: 'svip_plans', minimumRecords: 1 },
|
||||||
|
{ collection: 'codes', minimumRecords: 1 },
|
||||||
|
{ collection: 'vocabulary_units', minimumRecords: 1 },
|
||||||
|
{ collection: 'vocabulary', minimumRecords: 1 },
|
||||||
|
{ collection: 'handbook_subjects', minimumRecords: 1 },
|
||||||
|
{ collection: 'handbook_chapters', minimumRecords: 1 },
|
||||||
|
{ collection: 'handbook_entries', minimumRecords: 1 },
|
||||||
|
];
|
||||||
|
|
||||||
|
const criticalFieldRules = [
|
||||||
|
{ collection: 'users', field: 'id', requiredRatio: 1 },
|
||||||
|
{ collection: 'users', field: 'phone', requiredRatio: 0.8 },
|
||||||
|
{ collection: 'questions', field: 'id', requiredRatio: 1 },
|
||||||
|
{ collection: 'questions', field: 'subjectId', requiredRatio: 0.95 },
|
||||||
|
{ collection: 'questions', field: 'categoryId', requiredRatio: 0.8 },
|
||||||
|
{ collection: 'questions', field: 'content', requiredRatio: 0.95, alternatives: ['question', 'title', 'stem'] },
|
||||||
|
{ collection: 'orders', field: 'userId', requiredRatio: 0.98 },
|
||||||
|
{ collection: 'orders', field: 'planId', requiredRatio: 0.8 },
|
||||||
|
{ collection: 'orders', field: 'status', requiredRatio: 0.95 },
|
||||||
|
{ collection: 'codes', field: 'code', requiredRatio: 0.98 },
|
||||||
|
{ collection: 'vocabulary', field: 'unitId', requiredRatio: 0.95 },
|
||||||
|
{ collection: 'vocabulary', field: 'word', requiredRatio: 0.98 },
|
||||||
|
{ collection: 'handbook_chapters', field: 'subjectId', requiredRatio: 0.95 },
|
||||||
|
{ collection: 'handbook_entries', field: 'chapterId', requiredRatio: 0.95 },
|
||||||
|
{ collection: 'scoreline_records', field: 'schoolId', requiredRatio: 0.8 },
|
||||||
|
{ collection: 'question_videos', field: 'questionId', requiredRatio: 0.95 },
|
||||||
|
{ collection: 'question_videos', field: 'videoId', requiredRatio: 0.95 },
|
||||||
|
{ collection: 'user_word_progress', field: 'userId', requiredRatio: 0.95 },
|
||||||
|
{ collection: 'user_word_favorites', field: 'userId', requiredRatio: 0.95 },
|
||||||
|
];
|
||||||
|
|
||||||
const sensitiveKeyPattern =
|
const sensitiveKeyPattern =
|
||||||
/(password|token|secret|privatekey|sessionkey|accesskey|appkey|apikey|api_v3_key|notifytoken|aeskey|openid|unionid|wxaccesstoken|wechatsessionkey|smscode|verifycode|verificationcode|captcha)/i;
|
/(password|token|secret|privatekey|sessionkey|accesskey|appkey|apikey|api_v3_key|notifytoken|aeskey|openid|unionid|wxaccesstoken|wechatsessionkey|smscode|verifycode|verificationcode|captcha)/i;
|
||||||
const highRiskCollections = new Set(['settings', 'crm_config', 'smscodes']);
|
const highRiskCollections = new Set(['settings', 'crm_config', 'smscodes']);
|
||||||
@@ -172,6 +235,10 @@ function text(value: unknown) {
|
|||||||
return String(value).trim();
|
return String(value).trim();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function hasValue(record: JsonRecord, field: string, alternatives: string[] = []) {
|
||||||
|
return [field, ...alternatives].some(key => text(record[key]));
|
||||||
|
}
|
||||||
|
|
||||||
function readJsonFile(filePath: string) {
|
function readJsonFile(filePath: string) {
|
||||||
return JSON.parse(fs.readFileSync(filePath, 'utf8')) as unknown;
|
return JSON.parse(fs.readFileSync(filePath, 'utf8')) as unknown;
|
||||||
}
|
}
|
||||||
@@ -284,6 +351,68 @@ function validateRelations(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function buildReadiness(
|
||||||
|
issues: DryRunIssue[],
|
||||||
|
recordsByCollection: Map<string, JsonRecord[]>,
|
||||||
|
): DryRunReport['migrationReadiness'] {
|
||||||
|
const requiredCollections = productionRequiredCollections.map(rule => {
|
||||||
|
const count = recordsByCollection.get(rule.collection)?.length || 0;
|
||||||
|
const present = count >= rule.minimumRecords;
|
||||||
|
if (migrationProfile === 'production' && !present) {
|
||||||
|
issue(issues, {
|
||||||
|
severity: 'blocker',
|
||||||
|
code: 'production_required_collection_missing',
|
||||||
|
collection: rule.collection,
|
||||||
|
message: `${rule.collection}.json is required for production migration and must contain at least ${rule.minimumRecords} record(s).`,
|
||||||
|
count,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
collection: rule.collection,
|
||||||
|
present,
|
||||||
|
recordCount: count,
|
||||||
|
minimumRecords: rule.minimumRecords,
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
const criticalFieldCoverage = criticalFieldRules.flatMap(rule => {
|
||||||
|
const records = recordsByCollection.get(rule.collection);
|
||||||
|
if (!records || records.length === 0) return [];
|
||||||
|
let missingCount = 0;
|
||||||
|
for (const record of records) {
|
||||||
|
if (!hasValue(record, rule.field, rule.alternatives)) missingCount += 1;
|
||||||
|
}
|
||||||
|
const total = records.length;
|
||||||
|
const presentCount = total - missingCount;
|
||||||
|
const presentRatio = total === 0 ? 1 : presentCount / total;
|
||||||
|
const missingRatio = total === 0 ? 0 : missingCount / total;
|
||||||
|
if (presentRatio < rule.requiredRatio) {
|
||||||
|
issue(issues, {
|
||||||
|
severity: migrationProfile === 'production' ? 'blocker' : 'warning',
|
||||||
|
code: 'critical_field_coverage_low',
|
||||||
|
collection: rule.collection,
|
||||||
|
fieldPath: rule.field,
|
||||||
|
message:
|
||||||
|
`${rule.collection}.${rule.field} coverage is ${(presentRatio * 100).toFixed(1)}%, ` +
|
||||||
|
`below required ${(rule.requiredRatio * 100).toFixed(1)}% for reliable migration.`,
|
||||||
|
count: missingCount,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return [{
|
||||||
|
collection: rule.collection,
|
||||||
|
field: rule.field,
|
||||||
|
present: presentRatio >= rule.requiredRatio,
|
||||||
|
missingCount,
|
||||||
|
presentCount,
|
||||||
|
total,
|
||||||
|
missingRatio,
|
||||||
|
requiredRatio: rule.requiredRatio,
|
||||||
|
}];
|
||||||
|
});
|
||||||
|
|
||||||
|
return { requiredCollections, criticalFieldCoverage };
|
||||||
|
}
|
||||||
|
|
||||||
function loadSchema(issues: DryRunIssue[]) {
|
function loadSchema(issues: DryRunIssue[]) {
|
||||||
try {
|
try {
|
||||||
return readPocketBaseSchema();
|
return readPocketBaseSchema();
|
||||||
@@ -307,6 +436,13 @@ function collectFieldNames(records: JsonRecord[]) {
|
|||||||
|
|
||||||
function buildReport(): DryRunReport {
|
function buildReport(): DryRunReport {
|
||||||
const issues: DryRunIssue[] = [];
|
const issues: DryRunIssue[] = [];
|
||||||
|
if (migrationProfileInvalid) {
|
||||||
|
issue(issues, {
|
||||||
|
severity: 'blocker',
|
||||||
|
code: 'migration_profile_invalid',
|
||||||
|
message: `Invalid migration profile "${migrationProfileInput}". Use "development" or "production".`,
|
||||||
|
});
|
||||||
|
}
|
||||||
const schema = loadSchema(issues);
|
const schema = loadSchema(issues);
|
||||||
const collectionByName = schema?.byName || new Map<string, PocketBaseCollection>();
|
const collectionByName = schema?.byName || new Map<string, PocketBaseCollection>();
|
||||||
const files = fs.existsSync(exportDir) ? fs.readdirSync(exportDir).filter(file => file.toLowerCase().endsWith('.json')) : [];
|
const files = fs.existsSync(exportDir) ? fs.readdirSync(exportDir).filter(file => file.toLowerCase().endsWith('.json')) : [];
|
||||||
@@ -439,6 +575,8 @@ function buildReport(): DryRunReport {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const migrationReadiness = buildReadiness(issues, recordsByCollection);
|
||||||
|
|
||||||
const businessCounts = Object.fromEntries(
|
const businessCounts = Object.fromEntries(
|
||||||
businessCollections.map(collection => [collection, recordsByCollection.get(collection)?.length || 0]),
|
businessCollections.map(collection => [collection, recordsByCollection.get(collection)?.length || 0]),
|
||||||
);
|
);
|
||||||
@@ -454,6 +592,7 @@ function buildReport(): DryRunReport {
|
|||||||
exportDir,
|
exportDir,
|
||||||
schemaPath: schema?.absPath,
|
schemaPath: schema?.absPath,
|
||||||
generatedAt: new Date().toISOString(),
|
generatedAt: new Date().toISOString(),
|
||||||
|
migrationProfile,
|
||||||
summary: {
|
summary: {
|
||||||
files: files.length,
|
files: files.length,
|
||||||
collections: collections.length,
|
collections: collections.length,
|
||||||
@@ -465,6 +604,7 @@ function buildReport(): DryRunReport {
|
|||||||
infos,
|
infos,
|
||||||
},
|
},
|
||||||
businessCounts,
|
businessCounts,
|
||||||
|
migrationReadiness,
|
||||||
collections: collections.sort((a, b) => a.collection.localeCompare(b.collection)),
|
collections: collections.sort((a, b) => a.collection.localeCompare(b.collection)),
|
||||||
issues: issues.sort((a, b) => {
|
issues: issues.sort((a, b) => {
|
||||||
const order: Record<Severity, number> = { blocker: 0, warning: 1, info: 2 };
|
const order: Record<Severity, number> = { blocker: 0, warning: 1, info: 2 };
|
||||||
@@ -477,11 +617,27 @@ function printHuman(report: DryRunReport) {
|
|||||||
console.log('PocketBase dry-run migration report');
|
console.log('PocketBase dry-run migration report');
|
||||||
console.log(`Export dir: ${report.exportDir}`);
|
console.log(`Export dir: ${report.exportDir}`);
|
||||||
if (report.schemaPath) console.log(`Schema: ${report.schemaPath}`);
|
if (report.schemaPath) console.log(`Schema: ${report.schemaPath}`);
|
||||||
|
console.log(`Migration profile: ${report.migrationProfile}`);
|
||||||
console.log(
|
console.log(
|
||||||
`Summary: ${report.summary.records} records across ${report.summary.collections} collections; ` +
|
`Summary: ${report.summary.records} records across ${report.summary.collections} collections; ` +
|
||||||
`${report.summary.blockers} blocker(s), ${report.summary.warnings} warning(s)`,
|
`${report.summary.blockers} blocker(s), ${report.summary.warnings} warning(s)`,
|
||||||
);
|
);
|
||||||
console.log('');
|
console.log('');
|
||||||
|
console.log('Migration readiness:');
|
||||||
|
const missingRequired = report.migrationReadiness.requiredCollections.filter(item => !item.present);
|
||||||
|
const weakFields = report.migrationReadiness.criticalFieldCoverage.filter(item => !item.present);
|
||||||
|
console.log(` required collections missing/too small: ${missingRequired.length}`);
|
||||||
|
console.log(` critical field coverage warnings: ${weakFields.length}`);
|
||||||
|
for (const item of missingRequired.slice(0, 20)) {
|
||||||
|
console.log(` ${item.collection}: records=${item.recordCount}, minimum=${item.minimumRecords}`);
|
||||||
|
}
|
||||||
|
for (const item of weakFields.slice(0, 20)) {
|
||||||
|
console.log(
|
||||||
|
` ${item.collection}.${item.field}: present=${item.presentCount}/${item.total}, ` +
|
||||||
|
`required=${(item.requiredRatio * 100).toFixed(1)}%`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
console.log('');
|
||||||
console.log('Business counts:');
|
console.log('Business counts:');
|
||||||
for (const [name, count] of Object.entries(report.businessCounts)) {
|
for (const [name, count] of Object.entries(report.businessCounts)) {
|
||||||
console.log(` ${name.padEnd(24)} ${count}`);
|
console.log(` ${name.padEnd(24)} ${count}`);
|
||||||
|
|||||||
@@ -44,7 +44,7 @@ function createSafeExport() {
|
|||||||
{ id: 'q1', subjectId: 's1', categoryId: 'c1', type: 'choice', content: '题干', answer: 'A' },
|
{ id: 'q1', subjectId: 's1', categoryId: 'c1', type: 'choice', content: '题干', answer: 'A' },
|
||||||
]);
|
]);
|
||||||
writeJson(path.join(dir, 'orders.json'), [
|
writeJson(path.join(dir, 'orders.json'), [
|
||||||
{ id: 'o1', userId: 'u1', amount: 99, status: 'paid' },
|
{ id: 'o1', userId: 'u1', planId: 'p1', amount: 99, status: 'paid' },
|
||||||
]);
|
]);
|
||||||
writeJson(path.join(dir, 'svip_plans.json'), [
|
writeJson(path.join(dir, 'svip_plans.json'), [
|
||||||
{ id: 'p1', name: '题库会员', price: 99 },
|
{ id: 'p1', name: '题库会员', price: 99 },
|
||||||
@@ -105,12 +105,36 @@ function createBrokenRelationExport() {
|
|||||||
return dir;
|
return dir;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function createProductionIncompleteExport() {
|
||||||
|
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'tiku-pb-dry-prod-incomplete-'));
|
||||||
|
writeJson(path.join(dir, 'users.json'), [
|
||||||
|
{ id: 'u1', role: 'student', nickname: '缺手机号学生' },
|
||||||
|
]);
|
||||||
|
writeJson(path.join(dir, 'questions.json'), [
|
||||||
|
{ id: 'q1', categoryId: 'c1', type: 'choice', content: '缺科目题' },
|
||||||
|
]);
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
|
||||||
const safeDir = createSafeExport();
|
const safeDir = createSafeExport();
|
||||||
const safe = runDryRun(safeDir);
|
const safe = runDryRun(safeDir);
|
||||||
assert.equal(safe.status, 0, `safe dry-run should pass: ${safe.stdout} ${safe.stderr}`);
|
assert.equal(safe.status, 0, `safe dry-run should pass: ${safe.stdout} ${safe.stderr}`);
|
||||||
assert.equal(safe.payload.summary?.blockers, 0, 'safe dry-run should have no blockers');
|
assert.equal(safe.payload.summary?.blockers, 0, 'safe dry-run should have no blockers');
|
||||||
assert.ok(safe.payload.summary?.records >= 12, 'safe dry-run should count fixture records');
|
assert.ok(safe.payload.summary?.records >= 12, 'safe dry-run should count fixture records');
|
||||||
assert.equal(safe.payload.businessCounts?.questions, 1, 'safe dry-run should report question count');
|
assert.equal(safe.payload.businessCounts?.questions, 1, 'safe dry-run should report question count');
|
||||||
|
assert.equal(safe.payload.migrationProfile, 'development', 'default dry-run profile should be development');
|
||||||
|
assert.ok(Array.isArray(safe.payload.migrationReadiness?.requiredCollections), 'dry-run should report migration readiness');
|
||||||
|
|
||||||
|
const safeProduction = runDryRun(safeDir, { args: ['--profile=production'] });
|
||||||
|
assert.equal(safeProduction.status, 0, `complete production dry-run should pass: ${safeProduction.stdout} ${safeProduction.stderr}`);
|
||||||
|
assert.equal(safeProduction.payload.summary?.blockers, 0, 'complete production dry-run should have no blockers');
|
||||||
|
|
||||||
|
const invalidProfile = runDryRun(safeDir, { args: ['--profile=prodution'] });
|
||||||
|
assert.notEqual(invalidProfile.status, 0, 'invalid migration profile should fail closed');
|
||||||
|
assert.ok(
|
||||||
|
invalidProfile.payload.issues?.some(item => item.code === 'migration_profile_invalid'),
|
||||||
|
'invalid migration profile should be reported as a blocker',
|
||||||
|
);
|
||||||
|
|
||||||
const unsafeDir = createUnsafeExport();
|
const unsafeDir = createUnsafeExport();
|
||||||
const unsafe = runDryRun(unsafeDir);
|
const unsafe = runDryRun(unsafeDir);
|
||||||
@@ -152,9 +176,23 @@ assert.ok(
|
|||||||
'broken relation dry-run should detect unresolved question subject relation',
|
'broken relation dry-run should detect unresolved question subject relation',
|
||||||
);
|
);
|
||||||
|
|
||||||
|
const productionIncompleteDir = createProductionIncompleteExport();
|
||||||
|
const productionIncomplete = runDryRun(productionIncompleteDir, { args: ['--profile=production'] });
|
||||||
|
assert.notEqual(productionIncomplete.status, 0, 'production profile should fail incomplete export');
|
||||||
|
assert.ok(
|
||||||
|
productionIncomplete.payload.issues?.some(item => item.code === 'production_required_collection_missing'),
|
||||||
|
'production profile should block missing required collections',
|
||||||
|
);
|
||||||
|
assert.ok(
|
||||||
|
productionIncomplete.payload.issues?.some(item => item.code === 'critical_field_coverage_low' && item.collection === 'questions'),
|
||||||
|
'production profile should block critical field coverage gaps',
|
||||||
|
);
|
||||||
|
assert.equal(productionIncomplete.payload.migrationProfile, 'production', 'production profile should be reflected in report');
|
||||||
|
|
||||||
fs.rmSync(safeDir, { recursive: true, force: true });
|
fs.rmSync(safeDir, { recursive: true, force: true });
|
||||||
fs.rmSync(unsafeDir, { recursive: true, force: true });
|
fs.rmSync(unsafeDir, { recursive: true, force: true });
|
||||||
fs.rmSync(warningOnlyDir, { recursive: true, force: true });
|
fs.rmSync(warningOnlyDir, { recursive: true, force: true });
|
||||||
fs.rmSync(brokenRelationDir, { recursive: true, force: true });
|
fs.rmSync(brokenRelationDir, { recursive: true, force: true });
|
||||||
|
fs.rmSync(productionIncompleteDir, { recursive: true, force: true });
|
||||||
|
|
||||||
console.log('[PASS] PocketBase dry-run report');
|
console.log('[PASS] PocketBase dry-run report');
|
||||||
|
|||||||
Reference in New Issue
Block a user