feat: normalize legacy migration learning data

README.md

@@ -439,7 +439,7 @@ dry-run 会检查导出目录、JSON 形态、核心集合缺失、重复/缺失
 
 默认 dry-run 使用 `development` profile；正式迁移、预生产验收和 CI 应使用 `production` profile。生产 profile 会额外输出 `migrationReadiness`，检查用户、题目、科目、分类、订单、套餐、激活码、单词和知识手册等必需集合，以及用户手机号、题目归属、订单套餐、激活码、单词和手册归属等关键字段覆盖率。`--profile` 只接受 `development` 或 `production`，拼写错误会按 blocker 失败。
 
-当前真实 SQLite 基线已经跑通只读导出：58 个业务 collection、248555 条记录、9 个 storage 原始资源文件。production dry-run 目前仍有 2 个真实数据 blocker：30 个订单缺用户、22 个知识手册章节缺所属手册；还有 `user_answer_records`、`mock_exam_configs`、`referral_qrcodes`、`commission_settings` 等 mapper gap，详见：
+当前真实 SQLite 基线已经跑通只读导出：58 个业务 collection、248555 条记录、9 个 storage 原始资源文件。`user_answer_records`、`mock_exam_configs`、`referral_qrcodes`、`commission_settings` 已进入标准化导入。production dry-run 目前仍有 2 个真实数据 blocker：30 个订单缺用户、22 个知识手册章节缺所属手册；导入器会把它们隔离到财务复核/迁移待复核手册并写入 `pb_import_issues`，但正式切换前仍必须人工确认，详见：
 
 ```text
 docs/refactor/pocketbase-real-data-migration-runbook.md

docs/refactor/backend-capability-status.md

@@ -172,6 +172,7 @@
 | --- | --- | --- |
 | PocketBase schema/导出分析 | 可联调 | `scripts/import-pocketbase` 支持 schema summary/risk、`npm run pb:import:dry-run` 导出目录静态迁移报告 |
 | PocketBase JSON dry-run | 可联调 | 不写数据库，检查导出目录、JSON 形态、核心集合、旧 ID、敏感字段、schema relation、未映射集合和关键业务计数；`--profile=production` 会额外检查生产迁移必需集合和关键字段覆盖率，正式切换建议配合 `--fail-on-warnings` |
+| PocketBase SQLite 标准化导入 | 迁移期 | 已支持真实 SQLite 只读导出后的核心集合标准化，新增覆盖 `user_answer_records -> answer_records/wrong_questions`、`mock_exam_configs -> practice_blueprints`、`referral_qrcodes -> referral_qrcodes/referral_codes`、`commission_settings -> tenant_commission_settings`；缺用户订单会进入财务复核且不自动开权益，缺归属手册章节会进入“迁移待复核手册”并写 `pb_import_issues`；全量真实导入仍需继续做批量化性能优化 |
 | 题目 JSON preview/import | 可联调 | 后端负责规范化、issue、幂等、审计 |
 | 公共题库采纳、手动同步和自动同步 | 可联调 | 平台授权后，租户可采纳公共题库并复制已发布题目快照；同步 API 和 `public-banks` worker 支持新增/更新题目、重新校验授权、跨租户拒绝、审计记录、租户内容通知和租户自改冲突保护；冲突处理 API 已支持单条/批量采纳平台版本和保留租户本地版本；worker 失败会写 `public_question_bank_sync_failed` 租户通知并保留稳定错误码，恢复成功会自动关闭失败通知；平台可通过 `/api/platform-admin/question-bank-sync-status` 查看跨租户同步运营状态；已覆盖跨租户、重复采纳、采纳后组卷、同步新增题、通知隔离/已读/自动 resolved、冲突不覆盖、单条/批量冲突处理、worker 自动同步/失败通知/恢复关闭、starter 单地区不可见/不可采纳第二地区题库、pro 全国套餐可见第二地区题库等测试 |
 | 单词 JSON preview/import | 可联调 | 兼容旧模板 |

docs/refactor/next-development-todo.md

@@ -76,8 +76,9 @@
    - 从 PocketBase 导出现有用户、题库、单词、知识手册、分数线、订单、权益数据。
    - 已补 `npm run pb:export:sqlite` 只读 SQLite 导出工具，默认从 `F:\project\参考\旧题库数据库文件\data.db` 导出到 `.gitignore` 覆盖的 `pb_export/`，并生成 `sqlite-export-manifest.json`、`storage-manifest.json` 和脱敏统计。
    - 已补 `npm run pb:import:dry-run` 静态迁移报告工具、`--profile=production` 生产迁移门禁、关键集合/关键字段覆盖率检查、strict warning 门禁测试和真实数据迁移验收 runbook；真实 SQLite 已导出 58 个业务 collection、248555 条记录，并跑过 production dry-run。
-   - 当前真实 dry-run 剩余 blocker：30 个订单缺 `userId`，其中 7 个 paid；22 个知识手册章节缺 `subjectId`。正式导入前必须补自动修复/隔离 mapper 和人工复核报告。
-   - 当前真实 dry-run 剩余 mapper gap：`user_answer_records` 85442 条、`mock_exam_configs` 48 条、`referral_qrcodes` 79 条、`commission_settings` 1 条。它们分别对应学习历史/错题归因、全真模拟蓝图、推广码/小程序码、分佣设置，后续要进入标准化导入。
+   - 当前真实 dry-run 剩余 blocker：30 个订单缺 `userId`，其中 7 个 paid；22 个知识手册章节缺 `subjectId`。导入器已补隔离策略：缺用户订单进入财务复核且不自动开权益，缺归属手册章节进入“迁移待复核手册”。正式切换前仍必须人工找回/确认这些记录。
+   - 已补真实 mapper gap：`user_answer_records` 85442 条标准化到 `answer_records/wrong_questions`，`mock_exam_configs` 48 条标准化到 `practice_blueprints`，`referral_qrcodes` 79 条标准化到 `referral_qrcodes/referral_codes`，`commission_settings` 1 条标准化到 `tenant_commission_settings`。
+   - 真实 `pb:import:json` 试跑已验证 raw 导入和新 mapper 开始落库，但 248555 条真实记录的逐条标准化仍超过 10 分钟，需要下一阶段继续做批量化/事务分批优化，重点是 `questions`、`question_versions`、`user_answer_records`、`pb_raw_records`。
    - 处理完 blocker 后再跑 `npm run pb:import:dry-run -- --profile=production --json --fail-on-warnings`，再跑 `pb:import:json`、`pb:import:validate` 和业务抽样。
    - 对题目 JSON、单词、知识手册、分数线、视频走后端 preview/import API 做二次验证。
 

docs/refactor/pocketbase-real-data-migration-runbook.md

@@ -136,18 +136,29 @@ storage-manifest: 9 个原始资源文件
 auxiliary.db: _logs 121715 条，仅摘要
 ```
 
-最近一次 production dry-run 仍有 2 个 blocker，需要在正式导入前处理：
+最近一次 production dry-run 仍有 2 个真实数据 blocker，需要在正式切换前处理：
 
 - `orders.userId` 覆盖率 95.3%，30 个订单缺用户，其中 7 个为已支付订单。处理策略应优先从支付通知、手机号、订单号或人工售后记录找回用户；找不回的已支付订单必须进入人工异常台账，不允许静默开权益。
 - `handbook_chapters.subjectId` 覆盖率 94.4%，22 个章节缺所属手册。处理策略应按章节标题和条目归属归并到正确 `handbook_subjects`，无法归并的放入迁移隔离手册并标记待人工复核。
 
-production dry-run 还有这些 warning，属于后续 mapper 或运营处理项：
+当前导入器已经补齐下面 4 个旧集合的标准化 mapper：
+
+- `user_answer_records` 85442 条会导入到 `answer_records`，并按错误记录重建 `wrong_questions`。
+- `mock_exam_configs` 48 条会导入到 `practice_blueprints(mode='mock_exam', assembly_type='filters')`。
+- `referral_qrcodes` 79 条会导入到 `referral_qrcodes`，并同步补 `referral_codes`。
+- `commission_settings` 1 条会导入到 `tenant_commission_settings`。
+
+导入器对两个真实 blocker 采用“隔离 + 审计”策略，不会静默丢数据：
+
+- 缺用户订单仍导入 `orders/payments/order_items`，`raw_payload.migration.reviewRequired=true`、`entitlementBlocked=true`，并写入 `pb_import_issues`。缺用户订单不会自动开通权益。
+- 缺 `subjectId` 的手册章节会挂到 `handbook_subjects.legacy_id='__migration_orphan_handbook_subject__'` 的“迁移待复核手册”，章节 `metadata.reviewRequired=true`，并写入 `pb_import_issues`。
+
+production dry-run 还有这些 warning，属于运营处理项：
 
-- `user_answer_records` 85442 条尚未规范化到新练习历史/答题记录模型。
-- `mock_exam_configs` 48 条尚未规范化到 `practice_blueprints`。
-- `referral_qrcodes` 79 条尚未规范化到新推广码/小程序码模型。
-- `commission_settings` 1 条需要迁移到新分佣设置。
 - `dashboard_cache`、空 `customer_messages/smscodes/tenant_config` 可不作为正式数据源，必要时只保留审计摘要。
+- `recent_practices` 和少量 `user_answer_records` 有旧用户引用断裂，需要在导入后复核 `pb_import_issues`。
+
+性能注意：真实 `pb:import:json` 试跑已经证明 248555 条真实记录会超过 10 分钟。正式迁移前需要继续优化 importer 的批量写入和分批事务，重点是 `pb_raw_records`、`questions/question_versions`、`user_answer_records`。迁移演练必须记录总耗时、每阶段耗时和失败恢复策略。
 
 ## 阶段 1：静态 Dry-Run
 

scripts/import-pocketbase/src/dry-run-report.ts

@@ -96,6 +96,7 @@ const supportedCollections = new Set([
   'categories',
   'code_batches',
   'codes',
+  'commission_settings',
   'coupons',
   'coupon_redemptions',
   'crm_config',
@@ -109,6 +110,7 @@ const supportedCollections = new Set([
   'handbook_subjects',
   'images',
   'majors',
+  'mock_exam_configs',
   'module_nodes',
   'orders',
   'products',
@@ -117,6 +119,7 @@ const supportedCollections = new Set([
   'questions',
   'recent_practices',
   'referral_tracks',
+  'referral_qrcodes',
   'region_modules',
   'regions',
   'reports',
@@ -132,6 +135,7 @@ const supportedCollections = new Set([
   'svip_plans',
   'timelines',
   'user_badges',
+  'user_answer_records',
   'user_word_favorites',
   'user_word_progress',
   'users',

scripts/import-pocketbase/src/import-json.ts

@@ -39,6 +39,7 @@ const legacyLookupTables = new Set([
   'majors',
   'module_nodes',
   'orders',
+  'practice_blueprints',
   'products',
   'questions',
   'region_modules',
@@ -54,6 +55,12 @@ const legacyLookupTables = new Set([
   'vocabulary_words',
 ]);
 
+const nonCollectionJsonFiles = new Set([
+  'pb_schema.sqlite.json',
+  'sqlite-export-manifest.json',
+  'storage-manifest.json',
+]);
+
 function asArray(input: unknown): JsonRecord[] {
   if (Array.isArray(input)) return input as JsonRecord[];
   if (input && typeof input === 'object' && Array.isArray((input as { items?: unknown[] }).items)) {
@@ -179,6 +186,12 @@ function arrayValue(value: unknown): unknown[] {
   return [];
 }
 
+function objectValue(value: unknown): JsonRecord {
+  const parsed = parseJsonish(value, {});
+  if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) return parsed as JsonRecord;
+  return {};
+}
+
 function entriesValue(value: unknown): Array<[string, unknown]> {
   const parsed = parseJsonish(value, {});
   if (Array.isArray(parsed)) return parsed.map((item, index) => [String(index), item]);
@@ -190,6 +203,20 @@ function dateText(value: unknown): string {
   return text(value) || '';
 }
 
+function nullableDateText(...values: unknown[]): string {
+  for (const value of values) {
+    const result = dateText(value);
+    if (result) return result;
+  }
+  return '';
+}
+
+function clampRate(value: unknown, fallback = 0.2): number {
+  const parsed = numberValue(value, fallback);
+  if (parsed === null || !Number.isFinite(parsed)) return fallback;
+  return Math.min(1, Math.max(0, parsed));
+}
+
 function normalizeTenantRole(roleValue: unknown): string {
   const role = text(roleValue)?.toLowerCase();
   if (role === 'superadmin') return 'platform_admin';
@@ -395,22 +422,45 @@ async function markNormalized(runId: string, collection: string) {
   );
 }
 
+const legacyLookupCache = new Map<string, string | null>();
+const userLookupCache = new Map<string, string | null>();
+const questionVersionLookupCache = new Map<string, string | null>();
+
 async function legacyId(tableName: string, legacyValue: unknown): Promise<string | null> {
   const value = text(legacyValue);
   if (!value) return null;
   if (!legacyLookupTables.has(tableName)) throw new Error(`Unsafe legacy lookup table: ${tableName}`);
+  const cacheKey = `${tableName}:${value}`;
+  if (legacyLookupCache.has(cacheKey)) return legacyLookupCache.get(cacheKey) || null;
   const row = await queryOne<{ id: string }>(
     `select id from public.${tableName} where tenant_id = $1 and legacy_id = $2 limit 1`,
     [tenantId, value],
   );
-  return row?.id || null;
+  const result = row?.id || null;
+  legacyLookupCache.set(cacheKey, result);
+  return result;
 }
 
 async function userIdByLegacy(value: unknown): Promise<string | null> {
   const legacyValue = text(value);
   if (!legacyValue) return null;
+  if (userLookupCache.has(legacyValue)) return userLookupCache.get(legacyValue) || null;
   const row = await queryOne<{ id: string }>('select id from public.platform_users where legacy_id = $1 limit 1', [legacyValue]);
-  return row?.id || null;
+  const result = row?.id || null;
+  userLookupCache.set(legacyValue, result);
+  return result;
+}
+
+async function currentQuestionVersionId(questionId: string | null): Promise<string | null> {
+  if (!questionId) return null;
+  if (questionVersionLookupCache.has(questionId)) return questionVersionLookupCache.get(questionId) || null;
+  const row = await queryOne<{ current_version_id: string | null }>(
+    `select current_version_id from public.questions where tenant_id = $1 and id = $2 limit 1`,
+    [tenantId, questionId],
+  );
+  const result = row?.current_version_id || null;
+  questionVersionLookupCache.set(questionId, result);
+  return result;
 }
 
 async function upsertSecret(secretScope: string, secretKey: string, secretValue: unknown, provider: string | null = null) {
@@ -825,6 +875,109 @@ async function normalizeQuestions(records: JsonRecord[]) {
   }
 }
 
+async function normalizeUserAnswerRecords(runId: string, records: JsonRecord[]) {
+  const affectedWrongPairs = new Set<string>();
+
+  for (const r of records) {
+    const userId = await userIdByLegacy(r.userId);
+    if (!userId) {
+      await issue(
+        runId,
+        'user_answer_records',
+        r.id,
+        'answer_user_not_found',
+        `Answer record cannot be imported because userId was not resolved: ${text(r.userId) || '(empty)'}`,
+        'critical',
+      );
+      continue;
+    }
+
+    const questionId = await legacyId('questions', r.questionId);
+    if (!questionId) {
+      await issue(
+        runId,
+        'user_answer_records',
+        r.id,
+        'answer_question_not_found',
+        `Answer record kept with legacy_question_id only because questionId was not resolved: ${text(r.questionId) || '(empty)'}`,
+        'warning',
+      );
+    }
+
+    const isCorrect = r.isCorrect === null || r.isCorrect === undefined || r.isCorrect === ''
+      ? null
+      : boolValue(r.isCorrect);
+    const answeredAt = nullableDateText(r.answeredAt, r.updated, r.created);
+    const answerPayload = {
+      source: 'pocketbase.user_answer_records',
+      selectedOptions: arrayValue(r.selectedOptions),
+      legacyCategoryId: text(r.categoryId),
+      legacyQuestionId: text(r.questionId),
+      importedAt: new Date().toISOString(),
+    };
+
+    await pool.query(
+      `
+      insert into public.answer_records (
+        tenant_id, user_id, question_id, question_version_id, legacy_id,
+        legacy_question_id, legacy_category_id, selected_options,
+        answer_payload, is_correct, answered_at, created_at
+      )
+      values ($1,$2,$3,$4,$5,$6,$7,$8::jsonb,$9::jsonb,$10,
+        coalesce(nullif($11::text,'')::timestamptz, now()),
+        coalesce(nullif($12::text,'')::timestamptz, now())
+      )
+      on conflict (tenant_id, legacy_id) do update set
+        user_id = excluded.user_id,
+        question_id = excluded.question_id,
+        question_version_id = excluded.question_version_id,
+        legacy_question_id = excluded.legacy_question_id,
+        legacy_category_id = excluded.legacy_category_id,
+        selected_options = excluded.selected_options,
+        answer_payload = excluded.answer_payload,
+        is_correct = excluded.is_correct,
+        answered_at = excluded.answered_at
+      `,
+      [
+        tenantId,
+        userId,
+        questionId,
+        await currentQuestionVersionId(questionId),
+        r.id,
+        text(r.questionId),
+        text(r.categoryId),
+        JSON.stringify(arrayValue(r.selectedOptions)),
+        JSON.stringify(answerPayload),
+        isCorrect,
+        answeredAt,
+        nullableDateText(r.created, answeredAt),
+      ],
+    );
+
+    if (isCorrect === false && questionId) affectedWrongPairs.add(`${userId}:${questionId}`);
+  }
+
+  for (const pair of affectedWrongPairs) {
+    const [userId, questionId] = pair.split(':');
+    await pool.query(
+      `
+      insert into public.wrong_questions (tenant_id, user_id, question_id, wrong_count, last_wrong_at, resolved_at)
+      select $1, $2, $3, count(*)::integer, max(answered_at), null
+      from public.answer_records
+      where tenant_id = $1
+        and user_id = $2
+        and question_id = $3
+        and is_correct = false
+      on conflict (tenant_id, user_id, question_id)
+      do update set wrong_count = excluded.wrong_count,
+                    last_wrong_at = excluded.last_wrong_at,
+                    resolved_at = null
+      `,
+      [tenantId, userId, questionId],
+    );
+  }
+}
+
 async function normalizeUsers(records: JsonRecord[]) {
   for (const r of records) {
     const safeProfile = publicProfile(r);
@@ -1166,6 +1319,84 @@ async function normalizeSvipPlans(records: JsonRecord[]) {
   }
 }
 
+function normalizeMockExamSections(questionTypes: unknown) {
+  return arrayValue(questionTypes)
+    .map((item, index) => {
+      const object = objectValue(item);
+      const questionType = text(object.type) || text(object.questionType) || `section_${index + 1}`;
+      const count = intValue(object.count, 0);
+      return {
+        key: text(object.key) || questionType,
+        title: text(object.title) || text(object.name) || questionType,
+        questionType,
+        count,
+        sortOrder: intValue(object.order, index + 1) - 1,
+        scoreEach: numberValue(object.scoreEach ?? object.score, null),
+      };
+    })
+    .filter(section => section.count > 0)
+    .sort((a, b) => a.sortOrder - b.sortOrder);
+}
+
+async function normalizeMockExamConfigs(runId: string, records: JsonRecord[]) {
+  for (const r of records) {
+    const subjectId = await legacyId('subjects', r.subjectId);
+    const sections = normalizeMockExamSections(r.questionTypes);
+    const questionLimit = intValue(r.totalQuestions, sections.reduce((sum, section) => sum + section.count, 0));
+    if (!subjectId) {
+      await issue(
+        runId,
+        'mock_exam_configs',
+        r.id,
+        'mock_exam_subject_not_found',
+        `Mock exam blueprint imported without resolved subject because subjectId was not found: ${text(r.subjectId) || '(empty)'}`,
+        'warning',
+      );
+    }
+
+    await pool.query(
+      `
+      insert into public.practice_blueprints (
+        tenant_id, legacy_id, name, mode, assembly_type, question_limit,
+        duration_minutes, sections, rules, status, sort_order, created_at, updated_at
+      )
+      values ($1,$2,$3,'mock_exam','filters',$4,$5,$6::jsonb,$7::jsonb,$8,$9,
+        coalesce(nullif($10::text,'')::timestamptz, now()),
+        coalesce(nullif($11::text,'')::timestamptz, now())
+      )
+      on conflict (tenant_id, legacy_id) do update set
+        name = excluded.name,
+        question_limit = excluded.question_limit,
+        duration_minutes = excluded.duration_minutes,
+        sections = excluded.sections,
+        rules = excluded.rules,
+        status = excluded.status,
+        sort_order = excluded.sort_order,
+        updated_at = excluded.updated_at
+      `,
+      [
+        tenantId,
+        r.id,
+        text(r.name) || `全真模拟-${text(r.subjectId) || text(r.id) || '未命名'}`,
+        questionLimit > 0 ? questionLimit : null,
+        intValue(r.duration, 45),
+        JSON.stringify(sections),
+        JSON.stringify({
+          source: 'pocketbase.mock_exam_configs',
+          subjectId,
+          legacySubjectId: text(r.subjectId),
+          legacyQuestionTypes: arrayValue(r.questionTypes),
+          randomize: true,
+        }),
+        boolValue(r.isActive, true) ? 'active' : 'archived',
+        intValue(r.order),
+        dateText(r.created),
+        dateText(r.updated),
+      ],
+    );
+  }
+}
+
 async function normalizeCodeBatches(records: JsonRecord[]) {
   for (const r of records) {
     await pool.query(
@@ -1373,8 +1604,19 @@ async function normalizeCouponRedemptions(records: JsonRecord[]) {
   }
 }
 
-async function normalizeOrders(records: JsonRecord[]) {
+async function normalizeOrders(runId: string, records: JsonRecord[]) {
   for (const r of records) {
+    const resolvedUserId = await userIdByLegacy(r.userId);
+    const sanitized = sanitizeRecord(r);
+    const rawPayload = {
+      ...(sanitized as Record<string, unknown>),
+      migration: {
+        source: 'pocketbase.orders',
+        reviewRequired: !resolvedUserId,
+        reviewReason: !resolvedUserId ? 'legacy_user_not_resolved' : null,
+        entitlementBlocked: !resolvedUserId,
+      },
+    };
     const order = await queryOne<{ id: string }>(
       `
       insert into public.orders (
@@ -1410,7 +1652,7 @@ async function normalizeOrders(records: JsonRecord[]) {
       `,
       [
         tenantId,
-        await userIdByLegacy(r.userId),
+        resolvedUserId,
         r.id,
         text(r.userId),
         text(r.orderNo) || `legacy-${text(r.id)}`,
@@ -1426,13 +1668,23 @@ async function normalizeOrders(records: JsonRecord[]) {
         await legacyId('regions', r.regionId),
         text(r.regionId),
         dateText(r.paidAt),
-        JSON.stringify(sanitizeRecord(r)),
+        JSON.stringify(rawPayload),
         dateText(r.created),
         dateText(r.updated),
       ],
     );
 
     if (!order) continue;
+    if (!resolvedUserId) {
+      await issue(
+        runId,
+        'orders',
+        r.id,
+        'order_user_not_resolved',
+        `Order imported for finance review only because userId was not resolved: ${text(r.userId) || '(empty)'}`,
+        normalizeOrderStatus(r.status) === 'paid' ? 'critical' : 'warning',
+      );
+    }
 
     await pool.query(
       `
@@ -1625,17 +1877,60 @@ async function normalizeHandbookSubjects(records: JsonRecord[]) {
   }
 }
 
-async function normalizeHandbookChapters(records: JsonRecord[]) {
+async function migrationOrphanHandbookSubjectId(): Promise<string> {
+  const legacyIdValue = '__migration_orphan_handbook_subject__';
+  const row = await queryOne<{ id: string }>(
+    `
+    insert into public.handbook_subjects (
+      tenant_id, legacy_id, name, type, description, sort_order, is_active, metadata
+    )
+    values ($1,$2,'迁移待复核手册','migration_review','旧 PocketBase 章节缺少 subjectId 时自动挂载到这里，需人工归并。',999999,true,$3::jsonb)
+    on conflict (tenant_id, legacy_id) do update set
+      name = excluded.name,
+      description = excluded.description,
+      metadata = public.handbook_subjects.metadata || excluded.metadata,
+      updated_at = now()
+    returning id
+    `,
+    [
+      tenantId,
+      legacyIdValue,
+      JSON.stringify({
+        source: 'pocketbase.handbook_chapters',
+        reviewRequired: true,
+        reviewReason: 'legacy_chapter_subject_missing',
+      }),
+    ],
+  );
+  if (!row) throw new Error('Failed to create migration orphan handbook subject');
+  return row.id;
+}
+
+async function normalizeHandbookChapters(runId: string, records: JsonRecord[]) {
   for (const r of records) {
+    let subjectId = await legacyId('handbook_subjects', r.subjectId);
+    const missingSubject = !subjectId;
+    if (missingSubject) {
+      subjectId = await migrationOrphanHandbookSubjectId();
+      await issue(
+        runId,
+        'handbook_chapters',
+        r.id,
+        'handbook_chapter_subject_missing',
+        `Handbook chapter was attached to migration review subject because subjectId was missing or unresolved: ${text(r.subjectId) || '(empty)'}`,
+        'critical',
+      );
+    }
+
     await pool.query(
       `
       insert into public.handbook_chapters (
         tenant_id, subject_id, legacy_id, name, description, sort_order,
-        is_active, created_at, updated_at
+        is_active, metadata, created_at, updated_at
       )
-      values ($1,$2,$3,$4,$5,$6,$7,
-        coalesce(nullif($8::text,'')::timestamptz, now()),
-        coalesce(nullif($9::text,'')::timestamptz, now())
+      values ($1,$2,$3,$4,$5,$6,$7,$8::jsonb,
+        coalesce(nullif($9::text,'')::timestamptz, now()),
+        coalesce(nullif($10::text,'')::timestamptz, now())
       )
       on conflict (tenant_id, legacy_id) do update set
         subject_id = excluded.subject_id,
@@ -1643,16 +1938,23 @@ async function normalizeHandbookChapters(records: JsonRecord[]) {
         description = excluded.description,
         sort_order = excluded.sort_order,
         is_active = excluded.is_active,
+        metadata = excluded.metadata,
         updated_at = excluded.updated_at
       `,
       [
         tenantId,
-        await legacyId('handbook_subjects', r.subjectId),
+        subjectId,
         r.id,
         text(r.name) || '未命名手册章节',
         text(r.description),
         intValue(r.order),
         boolValue(r.isActive, true),
+        JSON.stringify({
+          source: 'pocketbase.handbook_chapters',
+          legacySubjectId: text(r.subjectId),
+          reviewRequired: missingSubject,
+          reviewReason: missingSubject ? 'legacy_subject_missing_or_unresolved' : null,
+        }),
         dateText(r.created),
         dateText(r.updated),
       ],
@@ -2273,6 +2575,130 @@ async function normalizeReferralTracks(records: JsonRecord[]) {
   }
 }
 
+function refCodeFromScene(sceneValue: unknown, fallback: unknown): string {
+  const scene = text(sceneValue);
+  if (scene) {
+    const match = /^ref[_=-]?(.+)$/i.exec(scene);
+    const code = (match?.[1] || scene).replace(/[^a-z0-9]/gi, '').toUpperCase();
+    if (code) return code;
+  }
+  const fallbackText = text(fallback)?.replace(/[^a-z0-9]/gi, '').toUpperCase();
+  return fallbackText || 'LEGACY';
+}
+
+async function normalizeReferralQrcodes(runId: string, records: JsonRecord[]) {
+  for (const r of records) {
+    const userId = await userIdByLegacy(r.userId);
+    const scene = text(r.scene) || `legacy_${text(r.id) || refCodeFromScene(null, r.userId)}`;
+    const page = text(r.page) || 'pages/index/index';
+    const refCode = refCodeFromScene(scene, r.id);
+    const qrcodeUrl = text(r.qrcodeUrl) || text(r.image);
+    const metadata = {
+      source: 'pocketbase.referral_qrcodes',
+      legacyId: text(r.id),
+      legacyUserId: text(r.userId),
+      image: text(r.image),
+      originalQrcodeUrl: text(r.qrcodeUrl),
+    };
+
+    if (!userId) {
+      await issue(
+        runId,
+        'referral_qrcodes',
+        r.id,
+        'referral_qrcode_user_not_found',
+        `Referral qrcode cannot be attached to a user because userId was not resolved: ${text(r.userId) || '(empty)'}`,
+        'critical',
+      );
+      continue;
+    }
+
+    await pool.query(
+      `
+      insert into public.referral_codes (tenant_id, user_id, code, status, channel, landing_path, metadata, created_at, updated_at)
+      values ($1,$2,$3::citext,'active','wechat-miniapp',$4,$5::jsonb,
+        coalesce(nullif($6::text,'')::timestamptz, now()),
+        coalesce(nullif($7::text,'')::timestamptz, now())
+      )
+      on conflict (tenant_id, user_id) do update set
+        code = excluded.code,
+        channel = excluded.channel,
+        landing_path = excluded.landing_path,
+        metadata = public.referral_codes.metadata || excluded.metadata,
+        updated_at = excluded.updated_at
+      `,
+      [tenantId, userId, refCode, page, JSON.stringify(metadata), dateText(r.created), dateText(r.updated)],
+    );
+
+    await pool.query(
+      `
+      insert into public.referral_qrcodes (
+        tenant_id, user_id, ref_code, scene, page, provider, qrcode_url, status,
+        metadata, created_at, updated_at
+      )
+      values ($1,$2,$3::citext,$4,$5,'wechat-miniapp',$6,$7,$8::jsonb,
+        coalesce(nullif($9::text,'')::timestamptz, now()),
+        coalesce(nullif($10::text,'')::timestamptz, now())
+      )
+      on conflict (tenant_id, provider, scene, page) do update set
+        user_id = excluded.user_id,
+        ref_code = excluded.ref_code,
+        qrcode_url = excluded.qrcode_url,
+        status = excluded.status,
+        metadata = public.referral_qrcodes.metadata || excluded.metadata,
+        updated_at = excluded.updated_at
+      `,
+      [
+        tenantId,
+        userId,
+        refCode,
+        scene,
+        page,
+        qrcodeUrl,
+        qrcodeUrl ? 'ready' : 'pending',
+        JSON.stringify(metadata),
+        dateText(r.created),
+        dateText(r.updated),
+      ],
+    );
+  }
+}
+
+async function normalizeCommissionSettings(records: JsonRecord[]) {
+  for (const r of records) {
+    const config = {
+      source: 'pocketbase.commission_settings',
+      legacyId: text(r.id),
+      remark: text(r.remark),
+      raw: sanitizeRecord(r),
+    };
+
+    await pool.query(
+      `
+      insert into public.tenant_commission_settings (
+        tenant_id, default_rate, min_settlement_cents, settlement_cycle, config,
+        created_at, updated_at
+      )
+      values ($1,$2,0,'monthly',$3::jsonb,
+        coalesce(nullif($4::text,'')::timestamptz, now()),
+        coalesce(nullif($5::text,'')::timestamptz, now())
+      )
+      on conflict (tenant_id) do update set
+        default_rate = excluded.default_rate,
+        config = public.tenant_commission_settings.config || excluded.config,
+        updated_at = excluded.updated_at
+      `,
+      [
+        tenantId,
+        clampRate(r.defaultRate, 0.2),
+        JSON.stringify(config),
+        dateText(r.created),
+        dateText(r.updated),
+      ],
+    );
+  }
+}
+
 async function normalizeBadges(records: JsonRecord[]) {
   for (const r of records) {
     await pool.query(
@@ -2820,6 +3246,7 @@ async function loadCollections(): Promise<CollectionMap> {
   const files = fs.readdirSync(exportDir).filter(file => file.toLowerCase().endsWith('.json'));
   const collections: CollectionMap = {};
   for (const file of files) {
+    if (nonCollectionJsonFiles.has(file)) continue;
     const collection = collectionNameFromFile(file);
     collections[collection] = asArray(JSON.parse(fs.readFileSync(path.join(exportDir, file), 'utf8')));
   }
@@ -2841,14 +3268,16 @@ async function normalizeAll(runId: string, collections: CollectionMap) {
   await runNormalizer(runId, collections, 'users', normalizeUsers);
   await normalizeUserEntitlementsAndStats(runId, collections.users || []);
   if ((collections.users || []).length > 0) await markNormalized(runId, 'users');
+  await runNormalizer(runId, collections, 'user_answer_records', records => normalizeUserAnswerRecords(runId, records));
 
   await runNormalizer(runId, collections, 'settings', normalizeSettings);
   await runNormalizer(runId, collections, 'svip_plans', normalizeSvipPlans);
+  await runNormalizer(runId, collections, 'mock_exam_configs', records => normalizeMockExamConfigs(runId, records));
   await runNormalizer(runId, collections, 'code_batches', normalizeCodeBatches);
   await runNormalizer(runId, collections, 'coupons', normalizeCoupons);
   await runNormalizer(runId, collections, 'coupon_redemptions', normalizeCouponRedemptions);
   await runNormalizer(runId, collections, 'codes', normalizeActivationCodes);
-  await runNormalizer(runId, collections, 'orders', normalizeOrders);
+  await runNormalizer(runId, collections, 'orders', records => normalizeOrders(runId, records));
 
   await runNormalizer(runId, collections, 'vocabulary_units', normalizeVocabularyUnits);
   await runNormalizer(runId, collections, 'vocabulary', normalizeVocabularyWords);
@@ -2863,7 +3292,7 @@ async function normalizeAll(runId: string, collections: CollectionMap) {
   }
 
   await runNormalizer(runId, collections, 'handbook_subjects', normalizeHandbookSubjects);
-  await runNormalizer(runId, collections, 'handbook_chapters', normalizeHandbookChapters);
+  await runNormalizer(runId, collections, 'handbook_chapters', records => normalizeHandbookChapters(runId, records));
   await runNormalizer(runId, collections, 'handbook_entries', normalizeHandbookEntries);
   await runNormalizer(runId, collections, 'banners', records => normalizeSimpleContent(records, 'banners'));
   await runNormalizer(runId, collections, 'faqs', records => normalizeSimpleContent(records, 'faqs'));
@@ -2880,6 +3309,8 @@ async function normalizeAll(runId: string, collections: CollectionMap) {
   await runNormalizer(runId, collections, 'scoreline_fields', normalizeScorelineFields);
   await runNormalizer(runId, collections, 'scoreline_records', normalizeScorelineRecords);
   await runNormalizer(runId, collections, 'referral_tracks', normalizeReferralTracks);
+  await runNormalizer(runId, collections, 'referral_qrcodes', records => normalizeReferralQrcodes(runId, records));
+  await runNormalizer(runId, collections, 'commission_settings', normalizeCommissionSettings);
   await runNormalizer(runId, collections, 'badges', normalizeBadges);
   await runNormalizer(runId, collections, 'user_badges', normalizeUserBadges);
   await runNormalizer(runId, collections, 'recent_practices', normalizeRecentPractices);

scripts/import-pocketbase/src/validate-import.ts

@@ -217,6 +217,120 @@ async function validationChecks(): Promise<CheckResult[]> {
     ),
   );
 
+  checks.push(
+    result(
+      'legacy_answers_missing_user',
+      await scalar(
+        `
+        select count(*)
+        from public.answer_records ar
+        left join public.platform_users u on u.id = ar.user_id
+        where ar.tenant_id = $1
+          and ar.legacy_id is not null
+          and u.id is null
+        `,
+        [tenantId],
+      ),
+      'Legacy answer_records exist without a valid user.',
+      'Legacy answer_records reference valid users.',
+    ),
+  );
+
+  checks.push(
+    result(
+      'legacy_answers_unresolved_questions',
+      await scalar(
+        `
+        select count(*)
+        from public.answer_records
+        where tenant_id = $1
+          and legacy_question_id is not null
+          and question_id is null
+        `,
+        [tenantId],
+      ),
+      'Some legacy answer_records kept only legacy_question_id because questions were not resolved.',
+      'Legacy answer_records question references are resolved.',
+      true,
+    ),
+  );
+
+  checks.push(
+    result(
+      'wrong_questions_invalid_references',
+      await scalar(
+        `
+        select count(*)
+        from public.wrong_questions wq
+        left join public.platform_users u on u.id = wq.user_id
+        left join public.questions q on q.id = wq.question_id and q.tenant_id = wq.tenant_id
+        where wq.tenant_id = $1
+          and (u.id is null or q.id is null)
+        `,
+        [tenantId],
+      ),
+      'Wrong-question rows reference missing users or questions.',
+      'Wrong-question rows reference valid users and questions.',
+    ),
+  );
+
+  checks.push(
+    result(
+      'legacy_mock_blueprints_incomplete',
+      await scalar(
+        `
+        select count(*)
+        from public.practice_blueprints
+        where tenant_id = $1
+          and mode = 'mock_exam'
+          and legacy_id is not null
+          and (
+            question_limit is null
+            or jsonb_array_length(sections) = 0
+            or rules->>'legacySubjectId' is null
+          )
+        `,
+        [tenantId],
+      ),
+      'Legacy mock exam blueprints are missing limits, sections, or legacy subject traceability.',
+      'Legacy mock exam blueprints have limits, sections, and legacy subject traceability.',
+    ),
+  );
+
+  checks.push(
+    result(
+      'referral_qrcodes_incomplete',
+      await scalar(
+        `
+        select count(*)
+        from public.referral_qrcodes
+        where tenant_id = $1
+          and (ref_code is null or scene = '' or page = '')
+        `,
+        [tenantId],
+      ),
+      'Referral qrcodes are missing ref_code, scene, or page.',
+      'Referral qrcodes have ref_code, scene, and page.',
+    ),
+  );
+
+  checks.push(
+    result(
+      'commission_settings_invalid_rate',
+      await scalar(
+        `
+        select count(*)
+        from public.tenant_commission_settings
+        where tenant_id = $1
+          and (default_rate < 0 or default_rate > 1)
+        `,
+        [tenantId],
+      ),
+      'Tenant commission settings contain invalid default_rate values.',
+      'Tenant commission settings default_rate values are valid.',
+    ),
+  );
+
   checks.push(
     result(
       'entitlements_unresolved_user',