Compare commits
4 Commits
fd8b197853
...
d6ad7e02c5
| Author | SHA1 | Date | |
|---|---|---|---|
|
d6ad7e02c5 | ||
|
|
5a49a37235 | ||
|
|
eb15849168 | ||
|
|
f12722eed9 |
4
.gitignore
vendored
4
.gitignore
vendored
@ -50,3 +50,7 @@ apps/api/public/uploads/
|
|||||||
*.tmp
|
*.tmp
|
||||||
*.orig
|
*.orig
|
||||||
*.rej
|
*.rej
|
||||||
|
|
||||||
|
# Local tooling dirs (不上传)
|
||||||
|
.omo/
|
||||||
|
.zcode/
|
||||||
|
|||||||
448
HANDOVER.md
Normal file
448
HANDOVER.md
Normal file
@ -0,0 +1,448 @@
|
|||||||
|
# 平台助手 — 项目交接文档
|
||||||
|
|
||||||
|
**交接日期:** 2026-09-30
|
||||||
|
**交接对象:** 接手继续开发的智能体 / 开发者
|
||||||
|
**文档目标:** 让接手方无需历史对话即可完整了解项目现状、架构、业务逻辑、已知问题与待办事项，可直接开始开发。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 项目是什么
|
||||||
|
|
||||||
|
**平台助手（内部代号 pingtai）** 是面向 B2B 采购场景的智能询比价工作台：
|
||||||
|
|
||||||
|
- **采集**：自动抓取 3 家央企招标公告（中电建 / 新矿集团 / 中核集团），关键词+AI 双通道分类入库
|
||||||
|
- **线索管理**：线索列表 / 详情 / 星级评分 / 状态流转（待复核 → 询价中 → 协作中）/ 跟进日志
|
||||||
|
- **询价发布**：勾选线索一键发布到外部 B2B 产业平台（通过 MCP 通道调 `publish_rfq`），含发布前确认弹窗（补联系人/电话/地址）、物料清单 AI 提取、超长标题 AI 压缩
|
||||||
|
- **比价**：供应商报价 Excel/CSV 上传 → 自动解析 → 商品对齐去重 → 三种采购策略对比（集中/拆分/限 K 家）→ 采购建议报告 + CSV 导出
|
||||||
|
- **生图**：商品图片三阶段生成（图库 → Bing 搜索 → MiniMax AI 生成）
|
||||||
|
- **品类库**：关键词优先 + AI 反哺的品类字典（ProductCategory）
|
||||||
|
- **管理后台**：租户/用户/系统配置/计划/MCP 对接配置
|
||||||
|
|
||||||
|
**在线地址（生产）:** https://direv.cloud（Nginx → Next.js 3001 / NestJS 4000）
|
||||||
|
**演示账号:** `admin` / `admin123`（username + password，**不是 email**）
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 技术栈与架构
|
||||||
|
|
||||||
|
| 层 | 技术 | 说明 |
|
||||||
|
|----|------|------|
|
||||||
|
| 前端 | Next.js 14 App Router + React 18 + Tailwind CSS 3 | **SPA 无路由**，useState 切换 14 个页面组件 |
|
||||||
|
| 后端 | NestJS 10 + TypeScript | 13 个子模块，端口 4000 |
|
||||||
|
| 数据库 | SQLite + Prisma ORM | `apps/api/prisma/dev.db`，15 个模型 |
|
||||||
|
| 采集脚本 | Node.js / Python | `scripts/tender-fetcher/`，CrawlerService 用 `execSync` 调度 |
|
||||||
|
| AI | MiniMax M2.7 (推理模型) | 品类分类 / 物料提取 / 标题压缩，endpoint `https://api.minimaxi.com/v1/chat/completions` |
|
||||||
|
| 外部对接 | MCP (Streamable HTTP, POST) | 集成 Integration 表 → product_network B2B 平台 |
|
||||||
|
| 部署 | 腾讯云 172.81.241.109 + PM2 + Nginx + certbot | `scripts/deploy.sh` 一键部署 |
|
||||||
|
|
||||||
|
### 架构图
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────┐
|
||||||
|
浏览器(3001) ←│ Next.js 14 (pingtai-web) │
|
||||||
|
└─────────────────────────────┘
|
||||||
|
│ /api/*
|
||||||
|
┌────────▼─────────────────────┐
|
||||||
|
│ NestJS 10 (pingtai-api:4000) │
|
||||||
|
│ ├─ crawler: execSync 调脚本 │
|
||||||
|
│ ├─ lead: 评分/状态/发布 │
|
||||||
|
│ ├─ integration: MCP client │
|
||||||
|
│ └─ comparison/image/... │
|
||||||
|
└────────┬─────────┬───────────┘
|
||||||
|
│ │
|
||||||
|
SQLite(dev.db) MiniMax API
|
||||||
|
│
|
||||||
|
┌────────▼─────────────────────┐
|
||||||
|
│ MCP server (devbox:3002/8095)│
|
||||||
|
│ → product_network B2B 平台 │
|
||||||
|
└──────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
> ⚠️ MCP server 端口 **3002(Node)/8095(Java)** 都在 devbox 开发机上，不是本地端口。平台助手本地 web 是 **3001**、API 是 **4000**。两套 MCP server 后端数据库不同，appId/appSecret 互不通用。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 目录结构
|
||||||
|
|
||||||
|
```
|
||||||
|
平台助手/
|
||||||
|
├── apps/
|
||||||
|
│ ├── web/ # Next.js 14 SPA
|
||||||
|
│ │ └── src/
|
||||||
|
│ │ ├── app/page.tsx # SPA 根，useState<Page> 切换页面
|
||||||
|
│ │ ├── components/ # 14 个页面 + AuthProvider/Sidebar/Providers/ErrorBoundary
|
||||||
|
│ │ └── lib/apiBase.ts# API_BASE (NEXT_PUBLIC_API_URL，本地回退 localhost:4000)
|
||||||
|
│ └── api/ # NestJS 10
|
||||||
|
│ ├── prisma/
|
||||||
|
│ │ ├── schema.prisma # 15 个模型
|
||||||
|
│ │ └── dev.db # SQLite
|
||||||
|
│ └── src/
|
||||||
|
│ ├── admin/ audit-log/ auth/ common/ comparison/ crawler/
|
||||||
|
│ ├── image/ inquiry/ integration/ lead/ product/ product-category/
|
||||||
|
│ ├── reports/ rules/ users/ prisma/
|
||||||
|
│ └── app.module.ts / main.ts
|
||||||
|
├── packages/shared/ # 共享 TS 类型（编译型，无运行时依赖）
|
||||||
|
└── scripts/
|
||||||
|
├── tender-fetcher/ # fetch-v1.2.js(中电建) fetch-xkgroup.py(新矿) fetch-cnncecp.py(中核)
|
||||||
|
├── direv-publish/ # direv 平台发布脚本（10 个 .py，历史保留）
|
||||||
|
├── parse_products.py # Excel/CSV → JSON（比价用）
|
||||||
|
├── search_image.py / process_image.py # 生图辅助
|
||||||
|
├── backfill_category.js / clean_category.js / reclassify_leads.js 等 # 数据维护
|
||||||
|
└── deploy.sh # 一键部署
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 开发环境搭建
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm run setup # npm install + db:init（prisma generate → migrate → seed）
|
||||||
|
npm run dev # 同时启动前端(3001)和后端(4000)
|
||||||
|
npm run dev:api # 仅后端（nest start --watch，用 dist/ 编译产物）
|
||||||
|
npm run dev:web # 仅前端
|
||||||
|
npm run db:init # 改 schema 后必跑
|
||||||
|
npm run db:reset # 删 db 重来
|
||||||
|
npm run build # nest build + next build
|
||||||
|
npm run typecheck # tsc --noEmit 所有包（无 root tsconfig，各包独立）
|
||||||
|
npm run lint
|
||||||
|
npm run dev:daemon # nohup 后台启动（PID 写入 .dev-api.pid / .dev-web.pid）
|
||||||
|
npm run dev:stop # kill PID 文件记录的进程
|
||||||
|
```
|
||||||
|
|
||||||
|
**端口速查（本地）:** web=3001, api=4000。⚠️ 本地 `next dev` 偶发 fallback 到 3000（3001 被 `admin_new` Vite 项目占用时），以终端实际打印为准。
|
||||||
|
|
||||||
|
### 后端陷阱
|
||||||
|
|
||||||
|
- `nest start --watch` 用 `dist/` 编译产物：Prisma schema 变更后必须 `npm run build -w apps/api` 重建 dist 再重启，单纯 `prisma generate` 不够
|
||||||
|
- 端口 4000 残留进程：`dev:stop` 只杀 PID 文件进程，多余进程 `lsof -ti:4000 | xargs kill -9`
|
||||||
|
- Prisma 变更流程：改 schema → `npm run db:init`
|
||||||
|
- 后端无测试框架
|
||||||
|
|
||||||
|
### 前端陷阱
|
||||||
|
|
||||||
|
- SPA 无路由：`page.tsx` 用 `useState<Page>` 切换，导航走 `onNavigate` prop，参数走 props
|
||||||
|
- 路径别名 `@/` → `src/`
|
||||||
|
- Prettier: noSemi, singleQuote, trailingComma: all, printWidth: 100
|
||||||
|
- Tailwind Token：`secondary(#0051d5)`, `border-subtle(#E2E8F0)`, `surface-container` 系
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 数据库模型（15 个）
|
||||||
|
|
||||||
|
| 模型 | 用途 | 关键字段 |
|
||||||
|
|------|------|----------|
|
||||||
|
| **Lead** | 线索核心表 | title/noticeId/detailUrl/level(C-B-A)/status/star/amount/category/organization/region/regionDetail/source/contact/phone/deadline/quantity/unit/notes/aiScore/scoringDetails/publish/publishTime/inquiryNo/direvUrl/assignedTo/followLogs；`@@unique([source, noticeId])` 去重键 |
|
||||||
|
| **Rule** | 规则中心 | name/type/config(JSON)/enabled |
|
||||||
|
| **RuleVersion** | 规则版本 | ruleId/version/config/userId |
|
||||||
|
| **Report** | 报表 | title/type/status/config/channels/fileUrl |
|
||||||
|
| **AuditLog** | 审计日志 | action/detail/operator/ip |
|
||||||
|
| **CrawlerConfig** | 采集器配置 | name/targetUrl/status/type(bid_xxx)/sourceName/scriptPath/schedule(cron)/keywords/categories(JSON品类名数组)/lastRunAt/totalRuns/successRuns/failedRuns |
|
||||||
|
| **Inquiry** | 询价（历史表） | 类似 Lead，冗余保留 |
|
||||||
|
| **Product** | 供应商报价条目（比价） | name/spec/unit/price/quantity/category/brand/supplier/minOrder/imageUrl |
|
||||||
|
| **Tenant** | 租户 | companyName/plan/status/validUntil/platformUrl/apiKey |
|
||||||
|
| **User** | 用户 | email/username/password/role/tenantId |
|
||||||
|
| **GeneratedImage** | 生成图片 | productName/url/status/style/sourceHash |
|
||||||
|
| **ImageLibrary** | 图库 | productName/keyword/filePath/source(baidu)/style/ratio |
|
||||||
|
| **SystemConfig** | 系统配置 | key(unique)/value/group |
|
||||||
|
| **Integration** | MCP 对接配置 | name/endpoint/appId/apiKey(AES-256-GCM 加密)/customerCode/status/tools/companyUserId |
|
||||||
|
| **ProductCategory** | 品类字典 | name(unique)/keywords(JSON数组)/isAIGenerated |
|
||||||
|
|
||||||
|
### 种子数据（seed.ts）
|
||||||
|
- 15 电缆型号 × 5 供应商 = 75 Product + 管理用户（admin/admin123）
|
||||||
|
- crawlerConfig 3 条：中电建 + 新矿集团 + 中核集团（中核默认 paused）
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. API 端点总览
|
||||||
|
|
||||||
|
### 认证 `/api/auth`
|
||||||
|
| POST /api/auth/login | 登录（username+password，JWT 存 localStorage `auth`） |
|
||||||
|
| GET /api/auth/profile | 当前用户 |
|
||||||
|
|
||||||
|
### 线索 `/api/leads`（核心）
|
||||||
|
| GET /api/leads | 列表（query: status/level/search/star/source/category/assignedTo/timeRange + 分页） |
|
||||||
|
| GET /api/leads/stats | 统计（OverviewPage 用） |
|
||||||
|
| GET/PUT/DELETE /api/leads/:id | 详情/更新/删除 |
|
||||||
|
| POST /api/leads/:id/to-rfq, /to-collab | 状态流转：转询价 / 转协作 |
|
||||||
|
| POST /api/leads/:id/publish | **发布到 B2B 平台**，可选 body `{contactPhone?, contactName?, deliveryAddress?}` |
|
||||||
|
| POST /api/leads/batch-publish | body `{ids: string[]}`，循环调 publish |
|
||||||
|
| POST /api/leads/:id/follow-up (+ /:index/delete) | 跟进日志 |
|
||||||
|
| POST /api/leads/:id/level | 改优先级 |
|
||||||
|
| POST /api/leads/calculate-score | 重算 AI 评分 |
|
||||||
|
|
||||||
|
### 采集器 `/api/crawlers`
|
||||||
|
GET/POST/PUT /api/crawlers(:id) | CRUD
|
||||||
|
POST /api/crawlers/:id/run, /pause, /resume | 运行/暂停/恢复（联动 cron 注册）
|
||||||
|
|
||||||
|
### 比价 `/api/comparison`
|
||||||
|
GET /api/comparison | 商品分组比价数据
|
||||||
|
POST /api/comparison/upload | multipart file+supplier 上传报价
|
||||||
|
POST /api/comparison/suggestion | 生成采购建议报告
|
||||||
|
|
||||||
|
### 品类 `/api/categories`
|
||||||
|
GET/POST/PUT/DELETE | ProductCategory CRUD（InquiryPage 筛选项动态加载）
|
||||||
|
|
||||||
|
### 其他
|
||||||
|
- `/api/products` + upload-image（商品管理）
|
||||||
|
- `/api/images` generate/parse/approve/regenerate（生图）
|
||||||
|
- `/api/integrations` + :id/connect/disconnect/call-tool + test（MCP 配置）
|
||||||
|
- `/api/admin` tenants/users/plan-config/configs/stats（管理后台）
|
||||||
|
- `/api/rules` + versions/rollback
|
||||||
|
- `/api/reports` + :id/send
|
||||||
|
- `/api/inquiries` / `/api/users` / `/api/audit-logs` / `/api/health`
|
||||||
|
- `/api/lead-notes/extract/:leadId`（notes 提取）
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 前端页面清单（14 个）
|
||||||
|
|
||||||
|
| 页面组件 | 功能 | 数据对接状态 |
|
||||||
|
|----------|------|---------------|
|
||||||
|
| LoginPage | 登录 | ✅ POST /api/auth/login |
|
||||||
|
| OverviewPage | 概览（统计卡片+线索列表） | ✅ Promise.all([leads/stats, leads]) |
|
||||||
|
| InquiryPage | 询价列表 + 筛选/搜索/星级/批量发布 + **发布前确认弹窗** | ✅ leads + publish（含弹窗传 overrides） |
|
||||||
|
| LeadDetailPage | 线索详情 + 状态流转 + 跟进日志 | ✅ leads/:id + 各流转端点 |
|
||||||
|
| ComparisonPage | 比价三步向导（上传→对比→建议） | ✅ comparison 三端点 |
|
||||||
|
| ProductPage | 商品管理 + 生图 | ✅ products + upload-image |
|
||||||
|
| ImagePage | 图片库/生成 | ✅ images |
|
||||||
|
| RuleCenterPage | 规则中心 + **采集器配置 Tab**（品类多选框） | ✅ crawlers + categories |
|
||||||
|
| ReportCenterPage | 报表 | ⚠️ 纯 mock |
|
||||||
|
| SystemMonitorPage | 系统监控 | ⚠️ 纯 mock |
|
||||||
|
| AdminPage | 管理后台（5 子区块：租户/用户/统计/configs/计划，configs 含 MCP 对接配置） | ✅ 已实现 |
|
||||||
|
| Sidebar / Providers / AuthProvider / ErrorBoundary | 框架 | — |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 核心业务流程
|
||||||
|
|
||||||
|
### 8.1 采集入库（crawler.service.ts）
|
||||||
|
|
||||||
|
```
|
||||||
|
CrawlerService.run(config)
|
||||||
|
├─ type=bid_powerchina → runPowerChina(): execSync('node fetch-v1.2.js --keyword "X"')
|
||||||
|
├─ type=bid_xkgroup → runGenericScript(): execSync('python3 fetch-xkgroup.py')
|
||||||
|
└─ 其他 bid_xxx → runGenericScript()（脚本名从 type 推断）
|
||||||
|
↓
|
||||||
|
results → categoryInference.batchInfer() # 品类推断（关键词匹配 → 未命中走 MiniMax）
|
||||||
|
↓
|
||||||
|
upsert 到 Lead（去重键 @@unique([source, noticeId])）
|
||||||
|
```
|
||||||
|
|
||||||
|
**关键词解析（resolveKeywords，最近重构）：**
|
||||||
|
1. 优先 `config.categories`（品类名数组）→ 查 ProductCategory 表把每个品类的 keywords 展开合并
|
||||||
|
2. 兜底 `config.keywords`（旧字段，直接解析）
|
||||||
|
3. 都没有 → `['']`（抓全部）
|
||||||
|
|
||||||
|
**中电建特殊处理：** 每个关键词有独立状态文件 `downloaded_<keyword>.json`（不是只有一个 downloaded.json）；结果从 `~/Desktop/招标公告/<date>/downloaded_<kw>.json` 读取。`~/desktop` 是 symlink → `/data/pingtai`（生产持久化）。
|
||||||
|
|
||||||
|
**公告 ID 提取坑：** `substr(summary, instr(summary, '公告ID:') + 6, 32)`（`+6` 而非 `+8`，字节/字符偏移曾出 bug）。
|
||||||
|
|
||||||
|
### 8.2 品类分类（品类方案 C：关键词优先 + AI 反哺）
|
||||||
|
|
||||||
|
```
|
||||||
|
采集入库/手动回填时对线索标题：
|
||||||
|
1. 关键词匹配 ProductCategory.keywords contains 标题 → 命中直接写
|
||||||
|
2. 未命中 → MiniMax API 分类 → 写品类 + 反写关键词到 ProductCategory
|
||||||
|
3. 同类标题下次直接走关键词，不再调 AI
|
||||||
|
```
|
||||||
|
|
||||||
|
- 关键文件：`product-category/category-inference.service.ts`（AI 推断+反写）、`crawler.service.ts batchInfer()`、`notes-extractor.service.ts`（PDF/HTML 提取要点）
|
||||||
|
- **MiniMax 配置**：endpoint `https://api.minimaxi.com/v1/chat/completions`，model `MiniMax-M2.7`（env `MINIMAX_MODEL` 可覆盖），key 在 `apps/api/.env` 的 `MINIMAX_API_KEY`
|
||||||
|
- **⚠️ M2.7 是推理模型**：max_tokens 要够（物料提取正文 2000、标题提取 800），否则输出全被 `<think>` 吃掉；输出常混 `<think>`、`让我分析`、`**加粗**` 等解释文字，需要清洗
|
||||||
|
- 历史经验：AI 不可全信，2026-07-01 曾批量回填 507 条，结果 AI 新建 ~25 个细分类需人工 review。决策：新数据走关键词，老数据人工 review
|
||||||
|
|
||||||
|
### 8.3 发布询价到 B2B 平台（lead.service.ts publishToDirev）
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /api/leads/:id/publish { contactPhone?, contactName?, deliveryAddress? }
|
||||||
|
→ publishToDirev(id, overrides)
|
||||||
|
1. 组装 publishArgs（title/description/qty/notes/attachments/deadline/deliveryDate/contact...）
|
||||||
|
2. 企业级硬编码字段：
|
||||||
|
inquiryParty='郑州巨变网络科技有限公司', freightFlag=1(含运费), procurementType=1(公开采购),
|
||||||
|
invoiceType=2(增值税专票), quotationDemand=1(需要报价), validDays=7
|
||||||
|
⚠️ paymentMethod 不传（传了报"企业字典找不到"）
|
||||||
|
⚠️ 上述 flag 必须 int，传字符串会报错（已修复过的坑）
|
||||||
|
3. 超长标题（>50字）→ TitleCompressorService.compress() AI 压缩 → 失败硬截断兜底
|
||||||
|
4. MaterialExtractorService.extractItems() 实时提取物料清单 items
|
||||||
|
5. integrationService.executeTool('publish_rfq', publishArgs)（retries:2）
|
||||||
|
6. 成功 → 更新 lead 状态（published/inquiryNo/direvUrl）；失败 → ServiceUnavailableException
|
||||||
|
```
|
||||||
|
|
||||||
|
**publishArgs 字段映射细节：**
|
||||||
|
|
||||||
|
| 字段 | 来源 | 备注 |
|
||||||
|
|------|------|------|
|
||||||
|
| title | lead.title（>50 字先 AI 压缩） | 平台 varchar(50) 限制 |
|
||||||
|
| description | lead.notes → summary → title | W31 P1 |
|
||||||
|
| qty | lead.quantity \|\| 1 | |
|
||||||
|
| notes | lead.notes | 商家报价时看到 |
|
||||||
|
| attachments | lead.attachments(JSON) \|\| [detailUrl] | |
|
||||||
|
| deadline | lead.deadline 前 10 位 | YYYY-MM-DD |
|
||||||
|
| deliveryDate | deadline + 30 天 | |
|
||||||
|
| contactName | overrides.contactName → lead.contact → '待补充' | 弹窗可编辑 |
|
||||||
|
| contactPhone | overrides.contactPhone → lead.phone → '待补充' | 必填，弹窗补全 |
|
||||||
|
| deliveryAddress | overrides.deliveryAddress → regionDetail → region | 弹窗可编辑 |
|
||||||
|
| userId/enterpriseId | Integration.companyUserId | MCP connect 时获取 |
|
||||||
|
|
||||||
|
### 8.4 MCP 集成（integration.service.ts）
|
||||||
|
|
||||||
|
- **协议**：Streamable HTTP（`StreamableHTTPClientTransport`），POST 方法
|
||||||
|
- **认证头**：`x-app-id` + `x-app-secret`（多租户 APIKey，替代旧 `x-api-key`），AES-256-GCM 加密存储（`ENCRYPT_KEY` in .env，32 bytes hex）
|
||||||
|
- **Session 管理（已实现）**：`buildSessionAwareFetch(integrationId)` 自定义 fetch wrapper——从 initialize 响应头提取 `mcp-session-id` 缓存 → 后续请求自动注入 → disconnect/重连清理
|
||||||
|
- **charset 修复（已实现）**：wrapper 检测 content-type 缺 charset 时补 `application/json; charset=utf-8`（否则 Java 解析器中文 Content-Length 不匹配报 JSON 错误）
|
||||||
|
- 17 个 MCP tool（询价5/商品4/比价4/推理4），发布调 `publish_rfq`
|
||||||
|
|
||||||
|
### 8.5 标题压缩（TitleCompressorService，新增）
|
||||||
|
|
||||||
|
- 位置：`apps/api/src/lead/title-compressor.service.ts`（已注册到 LeadModule）
|
||||||
|
- 逻辑：`compress(title, maxLength)`：
|
||||||
|
1. `title.length <= maxLength` 直接返回（**≤50 字不调 AI**）
|
||||||
|
2. 未配 `MINIMAX_API_KEY` → 硬截断 `title.slice(0, maxLength)`
|
||||||
|
3. 调 MiniMax，**要求 JSON 结构化输出** `{"title":"..."}`（对抗 M2.7 多变思考噪音，行级正则过滤不可靠）
|
||||||
|
4. `extractJsonTitle()` 正则提取 title 字段；解析不到 → 硬截断
|
||||||
|
5. 结果仍超长 → slice 截断兜底
|
||||||
|
- max_tokens 800（400 会被明文思考耗尽导致答案截断）
|
||||||
|
- 生产验证：6 个真实标题全部压缩达标（62字→25字等）
|
||||||
|
|
||||||
|
### 8.6 比价（comparison.service.ts）
|
||||||
|
|
||||||
|
```
|
||||||
|
上传报价 (POST upload: file+supplier)
|
||||||
|
→ parse_products.py 解析 Excel/CSV → Product 入库
|
||||||
|
→ GET /api/comparison：
|
||||||
|
按 name.trim().toLowerCase() 分组 → 组内按 supplier 去重留最低价
|
||||||
|
→ 每组算 bestPrice/bestSupplier/savings/priceSpread
|
||||||
|
→ POST /api/comparison/suggestion：
|
||||||
|
三种策略对比：
|
||||||
|
A. 无限制拆分（每 SKU 选最低总价供应商）
|
||||||
|
B. 单一供应商（Σ 全 SKU 总价排名，取第 1）
|
||||||
|
C. 限制 K 家（C(n,K) 组合枚举 + 每 SKU 取组合内最低价，K=2/3/4）
|
||||||
|
→ recommendations 自然语言建议 + CSV 导出
|
||||||
|
```
|
||||||
|
|
||||||
|
### 8.7 生图（image.service.ts）
|
||||||
|
|
||||||
|
三阶段：图库(ImageLibrary) → Bing 搜索(search_image.py) → MiniMax AI 生成；`process_image.py` 去水印+裁剪+白底合成。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. 已知问题 / 坑（接手必读）
|
||||||
|
|
||||||
|
### 9.1 生产部署相关（⚠️ 最近出现）
|
||||||
|
|
||||||
|
1. **页面打开报"服务器内部错误" / `Failed to find Server Action`**（web 日志常见）：
|
||||||
|
- 根因：**本地 build 的 `.next/` 直接 rsync 到生产**，生产 `node_modules` 版本与本地不一致 → Server Action ID 对不上 → 页面交互报错
|
||||||
|
- **解决办法（AGENTS.md 明确记录）**：在生产服务器上执行 `cd /opt/pingtai/apps/web && npx next build` 重建 `.next`，再 `pm2 restart pingtai-web`
|
||||||
|
- 若浏览器缓存旧 HTML 导致，提示用户强刷（Cmd+Shift+R）
|
||||||
|
2. **deploy.sh 不含 DB 迁移**：Prisma 变更部署前，需在生产 `apps/api/` 下手动 `npx prisma migrate deploy && npx prisma generate`，否则 schema 不同步
|
||||||
|
3. `pm2 restart` 默认不更新环境变量，需 `--update-env`（deploy.sh 已带）
|
||||||
|
|
||||||
|
### 9.2 数据质量（采集器只抓列表页）
|
||||||
|
|
||||||
|
| 字段 | 填写率(2026-07-16 生产统计 815 条) | 说明 |
|
||||||
|
|------|------|------|
|
||||||
|
| contact | 0% | 采集器未访问详情页 |
|
||||||
|
| phone | 0% | 同上 |
|
||||||
|
| quantity | 0% | 同上 |
|
||||||
|
| unit | 0% | 同上 |
|
||||||
|
| notes | 0% | 同上 |
|
||||||
|
|
||||||
|
当前应对：发布前弹窗手动补全。自动化提取（类似 notes-extractor 的详情页 AI 提取）**待实现**。
|
||||||
|
|
||||||
|
### 9.3 MiniMax / AI
|
||||||
|
|
||||||
|
- M2.7 输出噪音（`<think>`、`让我分析`、`**加粗**`）——所有 AI 调用都要清洗
|
||||||
|
- 生产 `.env` 已配 `MINIMAX_API_KEY` + `MINIMAX_MODEL`；本地 `.env` 需自行配置，否则走兜底
|
||||||
|
- 若 `MAX_COMPRESS_LEN` 等常量调整，注意 PROMPT 里 `'45'` 是字符串 replace，保持一致
|
||||||
|
|
||||||
|
### 9.4 采集器
|
||||||
|
|
||||||
|
- 中电建 state 文件 key：`systemId_招采公告`，**不要手动删除**
|
||||||
|
- 新矿集团脚本偶尔 `ReadTimeout`（日志见 Python traceback），不影响页面，重跑即可
|
||||||
|
- 新增采集器通用流程：放脚本到 `tender-fetcher/`，输出 `~/Desktop/招标公告/<date>/xxx_downloaded.json`，前端新建 CrawlerConfig（type=`bid_xxx`、sourceName、scriptPath），无需改代码
|
||||||
|
- cron 用 6 字段格式（兼容 cron@4.4.0）
|
||||||
|
|
||||||
|
### 9.5 MCP / B2B 发布
|
||||||
|
|
||||||
|
- `code:10021 "未获取到用户信息"`：产业平台修复中（创建 APIKey 已关联 companyUserId，后端直接使用）
|
||||||
|
- `paymentMethod` 传 "23" 报"企业字典找不到" → **不传**，后端用默认
|
||||||
|
- 字段类型必须是 int（freightFlag/procurementType/invoiceType），字符串会报错
|
||||||
|
- title 超 50 字已由 TitleCompressorService 前置处理
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. 当前工作区状态（未提交改动，2026-09-30）
|
||||||
|
|
||||||
|
> ⚠️ **接手第一件事：决定这些改动是否保留/提交/继续开发。**
|
||||||
|
|
||||||
|
### 已改文件（未 commit）
|
||||||
|
|
||||||
|
| 文件 | 改动内容 | 状态 |
|
||||||
|
|------|----------|------|
|
||||||
|
| `apps/api/src/lead/title-compressor.service.ts` | **新文件**：标题 AI 压缩服务（JSON 结构化输出方案） | ✅ 已实现+生产实测通过 |
|
||||||
|
| `apps/api/src/lead/lead.module.ts` | 注册 TitleCompressorService | ✅ |
|
||||||
|
| `apps/api/src/lead/lead.service.ts` | publishToDirev 集成标题压缩（>50 字先压缩再校验） | ✅ |
|
||||||
|
| `apps/api/src/crawler/crawler.service.ts` | resolveKeywords（品类→关键词展开）+ 每关键词独立状态文件 + 日志增强 | ✅ |
|
||||||
|
| `apps/web/src/components/InquiryPage.tsx` | 新增搜索框 fSearch → GET /api/leads?search= | ✅ |
|
||||||
|
| `apps/web/src/components/RuleCenterPage.tsx` | 采集器配置从 keywords 改为 **品类多选框**（categories JSON 数组） | ✅ |
|
||||||
|
| `scripts/tender-fetcher/fetch-v1.2.js` | 中电建脚本增强（过滤/回溯） | ✅ |
|
||||||
|
|
||||||
|
### 未跟踪文件
|
||||||
|
|
||||||
|
- `scripts/cleanup_garbage_categories.sql` — 清理 AI 垃圾品类
|
||||||
|
- `scripts/fix_corrupted_titles.sql` — 修复损坏标题
|
||||||
|
- `scripts/reclassify_leads.js` / `scripts/refetch_corrupted.js` — 历史数据维护
|
||||||
|
- `PRD-智能比价助手.md` — 比价功能 PRD（供外部平台参考）
|
||||||
|
|
||||||
|
### 工作区 vs 生产差异
|
||||||
|
|
||||||
|
- **本地有大量未提交改动，生产未部署**（生产 `.next/` 构建于 2026-09-03）
|
||||||
|
- 若需上线这些改动：`npm run typecheck` → `bash scripts/deploy.sh` → **生产重建 web `.next`**（见 9.1）
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. 生产部署与运维
|
||||||
|
|
||||||
|
- **服务器**：172.81.241.109（`SSH_HOST` 在 deploy.sh，如 IP 变更改这里）
|
||||||
|
- **SSH**：`sshpass -p 'Jubianshuzi@2026' ssh ubuntu@172.81.241.109`
|
||||||
|
- **一键部署**：`bash scripts/deploy.sh`（build + rsync dist/.next/源码 + pm2 restart --update-env）
|
||||||
|
- **路径**：代码 `/opt/pingtai/`，数据 `/data/pingtai/`（dev.db / uploads / library / 招标公告）
|
||||||
|
- **进程**：pm2 托管 `pingtai-api`(4000) + `pingtai-web`(3001)，systemd 自启
|
||||||
|
- **反代**：Nginx 80/443 → 3001；`/api/*` → 4000；`/uploads/*` → `/data/pingtai/uploads/`
|
||||||
|
- **HTTPS**：Let's Encrypt，certbot 自动续期 timer 已开
|
||||||
|
- **常用操作**：
|
||||||
|
```bash
|
||||||
|
pm2 list / pm2 logs pingtai-api / pm2 restart pingtai-api --update-env
|
||||||
|
tail -f /var/log/pingtai/api.out-1.log
|
||||||
|
sqlite3 /data/pingtai/dev.db # 直接查 DB
|
||||||
|
tar czf pingtai-backup-$(date +%Y%m%d).tgz /data/pingtai/
|
||||||
|
```
|
||||||
|
- **历史包袱**：服务器上旧 Postgres/Redis 部署备份在 `/home/ubuntu/platform-assistant.bak-*`，旧 nginx conf 在 `/etc/nginx/conf.d/platform.conf.bak-*`，可回退
|
||||||
|
- **采集目录 symlink**：`~/desktop -> /data/pingtai`（必须指向 `/data/pingtai`，不是 `/data/pingtai/招标公告`，否则双层嵌套）
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. 待办事项 / 建议的下一步
|
||||||
|
|
||||||
|
1. **决策未提交改动**：title-compressor + crawler 品类化 + InquiryPage 搜索 + RuleCenterPage 品类多选框，是否继续/提交/部署
|
||||||
|
2. **修复生产页面 500**：若线上仍有 Server Action 报错，生产 `npx next build` 重建（见 9.1）
|
||||||
|
3. **详情页内容提取**（P1 数据质量）：采集器入库后加类似 notes-extractor 的 AI 提取步骤，补 contact/phone/quantity/unit（当前全缺）
|
||||||
|
4. **空品类历史线索 review**：AdminPage 品类管理人工 review AI 新建的细分类（合并/删除）
|
||||||
|
5. **ReportCenterPage / SystemMonitorPage**：仍是 mock，可接真实数据
|
||||||
|
6. **比价模块增强**（参考 PRD-智能比价助手.md）：报价有效期、供应商画像权重、多租户
|
||||||
|
7. **MCP 发布卡点跟进**：`code:10021` 等产业平台修复状态
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. 协作约定（对后续智能体）
|
||||||
|
|
||||||
|
- 语言：中文（对话/注释/文档/commit）
|
||||||
|
- **改动前必须对齐**：涉及具体修改按 AGENTS.md 协作流程（7 个技术维度：场景/触发/重要程度/数据/边界/样式/部署），不擅自开干
|
||||||
|
- 例外：拼写错误、单文件 typo、用户明确"按你判断做"、紧急 bug
|
||||||
|
- 改完必须端到端验证，不只"代码看起来对"
|
||||||
|
- 涉及数据库结构、生产配置、依赖升级要明确告知用户
|
||||||
|
- 提交信息用中文
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**结束语：** 本项目最核心的链路是 **采集 → 分类 → 询价 → MCP 发布**，接手时建议先跑通这条链路的本地/生产实测，再动其他模块。祝顺利！
|
||||||
402
PRD-智能比价助手.md
Normal file
402
PRD-智能比价助手.md
Normal file
@ -0,0 +1,402 @@
|
|||||||
|
# 智能比价助手 - 产品需求文档 (PRD)
|
||||||
|
|
||||||
|
**版本**: v1.0
|
||||||
|
**日期**: 2025-08
|
||||||
|
**适用平台**: 参考设计文档，供其他平台二次开发/移植使用
|
||||||
|
**源项目**: 平台助手 - 智能比价助手模块
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 产品概述
|
||||||
|
|
||||||
|
### 1.1 产品定位
|
||||||
|
**智能比价助手** 是面向采购场景的多供应商报价自动对齐与智能决策工具。核心价值：**将非结构化的供应商报价文件（Excel/CSV）转化为结构化可比价数据，通过算法自动生成最优采购方案（拆分/集中/限制供应商数），输出可执行的采购建议报告。**
|
||||||
|
|
||||||
|
### 1.2 核心用户场景
|
||||||
|
| 场景 | 痛点 | 解决方案 |
|
||||||
|
|------|------|----------|
|
||||||
|
| 采购员收到多家供应商报价单 | 格式不一、人工对齐耗时、易漏项 | 拖拽上传 → 自动解析 → 以商品名+规格自动对齐 |
|
||||||
|
| 不知道找哪家供应商最便宜 | 需人工逐行对比、易漏最优 | 自动计算每SKU最优/次优供应商、价差、降本率 |
|
||||||
|
| 纠结是集中采购还是拆分采购 | 缺量化依据、靠经验拍脑袋 | 三大策略对比：集中采购 / 无限制拆分 / 限制K家供应商 |
|
||||||
|
| 需向领导汇报采购方案 | 缺专业报告、数据支撑弱 | 一键生成含策略对比、SKU明细、供应商画像、降本测算的结构化报告 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 系统架构
|
||||||
|
|
||||||
|
### 2.1 技术栈
|
||||||
|
| 层 | 技术选型 |
|
||||||
|
|----|----------|
|
||||||
|
| 后端框架 | NestJS 10 + TypeScript |
|
||||||
|
| 数据库 | SQLite (Prisma ORM) / 可替换为 PostgreSQL/MySQL |
|
||||||
|
| 文件解析 | Python 脚本 `parse_products.py` (pandas/openpyxl) |
|
||||||
|
| 前端框架 | Next.js 14 App Router + React 18 + Tailwind CSS 3 |
|
||||||
|
| 状态管理 | React useState + useEffect (SPA 无路由模式) |
|
||||||
|
| 文件上传 | Multer + FormData 拖拽上传 |
|
||||||
|
|
||||||
|
### 2.2 模块依赖
|
||||||
|
```
|
||||||
|
ComparisonModule
|
||||||
|
├── ComparisonController (REST API)
|
||||||
|
├── ComparisonService (核心业务逻辑)
|
||||||
|
├── PrismaModule (数据库)
|
||||||
|
└── (无外部服务依赖)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 数据模型
|
||||||
|
|
||||||
|
### 3.1 核心实体：Product (供应商报价条目)
|
||||||
|
```prisma
|
||||||
|
model Product {
|
||||||
|
id String @id @default(uuid())
|
||||||
|
name String // 商品名称（去重键核心字段）
|
||||||
|
spec String // 规格型号
|
||||||
|
unit String // 单位（个/吨/米/卷...）
|
||||||
|
price Float // 单价
|
||||||
|
quantity Int // 数量/采购量
|
||||||
|
category String // 品类（用于筛选/统计）
|
||||||
|
brand String? // 品牌
|
||||||
|
supplier String // 供应商名称（去重键辅助字段）
|
||||||
|
minOrder Int? // 起订量
|
||||||
|
imageUrl String? // 图片URL
|
||||||
|
status String @default("draft") // draft/published
|
||||||
|
createdAt DateTime @default(now())
|
||||||
|
updatedAt DateTime @updatedAt
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.2 聚合视图：ComparisonItem (比价分析输出)
|
||||||
|
```typescript
|
||||||
|
interface ComparisonItem {
|
||||||
|
productName: string; // 商品名（去重后）
|
||||||
|
spec: string; // 规格
|
||||||
|
category: string; // 品类
|
||||||
|
quantity: number; // 采购量（取该商品所有报价的最大值）
|
||||||
|
suppliers: SupplierInfo[]; // 去重后的供应商报价列表
|
||||||
|
bestPrice: number; // 最低价
|
||||||
|
bestSupplier: string; // 最优供应商名
|
||||||
|
hasSupplier: boolean; // 是否有有效供应商报价
|
||||||
|
savings: number; // 平均价 - 最优价（单价维度）
|
||||||
|
priceSpread: number; // 最高价 - 最低价（价差）
|
||||||
|
}
|
||||||
|
|
||||||
|
interface SupplierInfo {
|
||||||
|
id: string;
|
||||||
|
name: string;
|
||||||
|
price: number;
|
||||||
|
unit: string;
|
||||||
|
spec: string;
|
||||||
|
brand?: string;
|
||||||
|
minOrder?: number;
|
||||||
|
imageUrl?: string;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.3 采购建议报告
|
||||||
|
```typescript
|
||||||
|
interface SuggestionReport {
|
||||||
|
// 核心汇总
|
||||||
|
totalItems: number; // 参与比价的SKU数
|
||||||
|
totalSplitCost: number; // 无限制拆分总价
|
||||||
|
savingsVsSingle: number; // 拆分 vs 单一供应商节省金额
|
||||||
|
savingsPctVsSingle: number; // 综合降本比例
|
||||||
|
bestSingleSupplier: { name, total }; // 单一供应商最优方案
|
||||||
|
|
||||||
|
// 策略维度
|
||||||
|
supplierRanking: { name, total, skuCount }[]; // 单一供应商总价排名
|
||||||
|
splitPicks: SplitPick[]; // 无限制拆分时每SKU最优分配
|
||||||
|
splitByCount: { // 限制K家供应商时的最优组合
|
||||||
|
2: StrategyResult;
|
||||||
|
3: StrategyResult;
|
||||||
|
4: StrategyResult;
|
||||||
|
};
|
||||||
|
|
||||||
|
// 可执行建议
|
||||||
|
recommendations: string[]; // 自然语言建议列表
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 核心 API 设计
|
||||||
|
|
||||||
|
### 4.1 REST Endpoints
|
||||||
|
|
||||||
|
| 方法 | 路径 | 说明 | 请求/响应 |
|
||||||
|
|------|------|------|-----------|
|
||||||
|
| GET | `/api/comparison` | 获取所有商品分组比价数据 | 返回 `ComparisonItem[]` |
|
||||||
|
| POST | `/api/comparison/upload` | 上传供应商报价文件 | multipart/form-data: file + supplier |
|
||||||
|
| POST | `/api/comparison/suggestion` | 生成采购建议报告 | 返回 `SuggestionReport` |
|
||||||
|
|
||||||
|
### 4.2 请求/响应示例
|
||||||
|
|
||||||
|
**上传报价**
|
||||||
|
```bash
|
||||||
|
curl -X POST /api/comparison/upload \
|
||||||
|
-F "file=@quote.xlsx" \
|
||||||
|
-F "supplier=远东电缆"
|
||||||
|
```
|
||||||
|
```json
|
||||||
|
{ "count": 45, "supplier": "远东电缆" }
|
||||||
|
```
|
||||||
|
|
||||||
|
**获取建议**
|
||||||
|
```bash
|
||||||
|
curl -X POST /api/comparison/suggestion
|
||||||
|
```
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"totalItems": 12,
|
||||||
|
"totalSplitCost": 125600.00,
|
||||||
|
"savingsVsSingle": 18500.50,
|
||||||
|
"savingsPctVsSingle": 12.8,
|
||||||
|
"bestSingleSupplier": { "name": "远东电缆", "total": 144100.50 },
|
||||||
|
"supplierRanking": [
|
||||||
|
{ "name": "远东电缆", "total": 144100.50, "skuCount": 12 },
|
||||||
|
{ "name": "宝胜电缆", "total": 152300.00, "skuCount": 10 }
|
||||||
|
],
|
||||||
|
"splitPicks": [...],
|
||||||
|
"splitByCount": { "2": {...}, "3": {...}, "4": {...} },
|
||||||
|
"recommendations": [
|
||||||
|
"本次比价共涉及 12 个SKU，2 家供应商参与报价。",
|
||||||
|
"按SKU拆分下单比统一从「远东电缆」采购可再节省 ¥18500.50（降本 12.8%）。"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 核心算法逻辑
|
||||||
|
|
||||||
|
### 5.1 数据聚合与去重 (`getComparisonData`)
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// 核心步骤：
|
||||||
|
// 1. 读取所有 Product 记录
|
||||||
|
// 2. 按 name.trim().toLowerCase() 分组（商品名去重）
|
||||||
|
// 3. 组内按 supplier 去重 → 同一供应商保留最低价
|
||||||
|
// 4. 计算每组统计指标：
|
||||||
|
// - bestPrice / bestSupplier
|
||||||
|
// - savings = avgPrice - bestPrice
|
||||||
|
// - priceSpread = maxPrice - minPrice
|
||||||
|
```
|
||||||
|
|
||||||
|
**去重策略**：
|
||||||
|
- **商品级**：`name` 忽略大小写/首尾空格分组
|
||||||
|
- **供应商级**：组内按 `supplier` 去重，保留最低价
|
||||||
|
- **数量取最大值**：`quantity = max(所有报价的 quantity)`
|
||||||
|
|
||||||
|
### 5.2 无限制拆分策略 (`getSuggestion` → splitPicks)
|
||||||
|
|
||||||
|
每个 SKU 独立选择最低总价供应商：
|
||||||
|
```
|
||||||
|
对每个 SKU：
|
||||||
|
1. 计算各供应商总价 = price * quantity
|
||||||
|
2. 选总价最低者为最优
|
||||||
|
3. 次优 = 总价第二低者
|
||||||
|
4. 单品节省 = 次优总价 - 最优总价
|
||||||
|
```
|
||||||
|
|
||||||
|
**输出**：`splitPicks[]` + `totalSplitCost` + `supplierSplitCounts`（各供应商中标SKU数）
|
||||||
|
|
||||||
|
### 5.3 单一供应商策略 (`supplierRanking`)
|
||||||
|
|
||||||
|
```
|
||||||
|
对每个供应商：
|
||||||
|
总价 = Σ(该供应商所有SKU的 price * quantity)
|
||||||
|
SKU数 = 该供应商有报价的SKU数
|
||||||
|
按总价升序排序 → bestSingleSupplier = 第1名
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.4 限制 K 家供应商策略 (`splitByCount[K]`)
|
||||||
|
|
||||||
|
**组合枚举 + 贪心分配**：
|
||||||
|
|
||||||
|
```
|
||||||
|
输入：所有供应商名单 allNames，目标数量 K (2/3/4)
|
||||||
|
对于 allNames 中所有 C(n, K) 种组合：
|
||||||
|
对每个 SKU：
|
||||||
|
从组合中选价格最低的供应商
|
||||||
|
累加总价
|
||||||
|
记录总价最低的组合
|
||||||
|
```
|
||||||
|
|
||||||
|
**复杂度**：C(n, K) × m (SKU数)。n≤10, K≤4 时可接受（C(10,4)=210）。
|
||||||
|
|
||||||
|
**输出**：
|
||||||
|
- `bestSuppliers`: 实际使用的供应商列表（去除组合中未中标者）
|
||||||
|
- `totalCost`: 最优总价
|
||||||
|
- `assignments[]`: 每SKU的分配明细
|
||||||
|
- `supplierBreakdown`: 各供应商中标SKU数/金额占比
|
||||||
|
|
||||||
|
### 5.3 采购建议生成 (`recommendations`)
|
||||||
|
|
||||||
|
基于规则引擎生成自然语言建议：
|
||||||
|
1. **概况**：SKU数、供应商数
|
||||||
|
2. **拆分 vs 集中**：若 `savingsVsSingle > 0` → 推荐拆分 + 具体金额/比例
|
||||||
|
3. **优势供应商**：`supplierSplitCounts` Top 3
|
||||||
|
4. **最大价差SKU**：`splitPicks` 中 savings 最大者
|
||||||
|
5. **风险提示**：SKU>10 时提示交付协调风险
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 前端交互流程
|
||||||
|
|
||||||
|
### 6.1 三步向导式交互
|
||||||
|
|
||||||
|
```
|
||||||
|
Step 1: 上传报价 (upload)
|
||||||
|
├── 输入供应商名称
|
||||||
|
├── 拖拽/点击上传 .xlsx/.xls/.csv
|
||||||
|
├── Python脚本解析 → 入库
|
||||||
|
└── 显示已上传供应商列表 + 报价条数
|
||||||
|
|
||||||
|
Step 2: 比价分析 (compare)
|
||||||
|
├── 表格展示：商品名 | 规格 | 数量 | 各供应商单价 | 最优价 | 价差
|
||||||
|
├── 最优价绿色高亮，缺报显示 "-"
|
||||||
|
├── 支持按价格/名称/节省金额排序
|
||||||
|
└── 无供应商对比数据单独展示
|
||||||
|
|
||||||
|
Step 3: 采购建议 (suggestion)
|
||||||
|
├── Hero汇总卡：拆分总价 / 节省金额 / 降本率
|
||||||
|
├── 策略对比卡：
|
||||||
|
│ ├── A方案：单一供应商（含供应商总价排名表）
|
||||||
|
│ └── B方案：拆分采购（含K家供应商选择器 K=2/3/4）
|
||||||
|
│ ├── 动态条形图：供应商中标SKU占比
|
||||||
|
│ └── 明细表：每SKU分配供应商/单价/数量/总价
|
||||||
|
├── SKU明细表：推荐供应商/单价/总价/次优总价/节省/降本率
|
||||||
|
├── 采购建议列表（自然语言）
|
||||||
|
└── 导出 CSV 报告
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.2 关键交互细节
|
||||||
|
|
||||||
|
| 交互 | 实现细节 |
|
||||||
|
|------|----------|
|
||||||
|
| 文件上传 | 拖拽区域 + 点击选择，支持 .xlsx/.xls/.csv，显示上传进度 |
|
||||||
|
| 供应商名称 | 必填校验，作为报价归属标识 |
|
||||||
|
| K值选择器 | 2/3/4 三按钮，单选互斥，支持取消选中回退无限制方案 |
|
||||||
|
| 表格高亮 | 最优价格绿色背景+粗体，价差红色显示 |
|
||||||
|
| 空状态处理 | 无数据/无对比数据分别展示占位提示 |
|
||||||
|
| 导出CSV | BOM头防乱码，含完整SKU明细列 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 文件解析规范
|
||||||
|
|
||||||
|
### 7.1 支持格式
|
||||||
|
- `.xlsx` / `.xls` (Excel)
|
||||||
|
- `.csv` (逗号分隔)
|
||||||
|
|
||||||
|
### 7.2 必需列映射 (parse_products.py)
|
||||||
|
| 标准列名 | 别名/兼容 | 说明 |
|
||||||
|
|----------|-----------|------|
|
||||||
|
| `title` / `name` | 商品名称、产品名称 | 必填，作为去重主键 |
|
||||||
|
| `spec` | 规格、型号 | 可选 |
|
||||||
|
| `unit` | 单位 | 默认 "个" |
|
||||||
|
| `price` | 单价、价格 | 必填，数值型 |
|
||||||
|
| `quantity` | 数量、采购量 | 默认 1 |
|
||||||
|
| `category` | 品类、分类 | 默认 "未分类" |
|
||||||
|
| `brand` | 品牌 | 可选 |
|
||||||
|
| `minOrder` | 起订量 | 可选 |
|
||||||
|
|
||||||
|
### 7.3 解析脚本接口
|
||||||
|
```python
|
||||||
|
# scripts/parse_products.py
|
||||||
|
# 输入: 文件路径
|
||||||
|
# 输出: JSON 数组到 stdout
|
||||||
|
# 格式: [{"title": "...", "spec": "...", "price": 100, ...}, ...]
|
||||||
|
# 错误: 输出 {"error": "错误信息"} 到 stdout
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 部署与运维
|
||||||
|
|
||||||
|
### 8.1 环境变量
|
||||||
|
```env
|
||||||
|
DATABASE_URL=file:./dev.db # SQLite文件路径
|
||||||
|
# 或 PostgreSQL: postgresql://user:pass@host:5432/db
|
||||||
|
```
|
||||||
|
|
||||||
|
### 8.2 启动命令
|
||||||
|
```bash
|
||||||
|
# 开发
|
||||||
|
npm run dev # 同时启动前端(3001) + 后端(4000)
|
||||||
|
npm run dev:api # 仅后端
|
||||||
|
npm run dev:web # 仅前端
|
||||||
|
|
||||||
|
# 生产部署
|
||||||
|
npm run build
|
||||||
|
bash scripts/deploy.sh # 一键构建+rsync+PM2重启
|
||||||
|
```
|
||||||
|
|
||||||
|
### 8.3 生产架构
|
||||||
|
```
|
||||||
|
Nginx (80/443)
|
||||||
|
├── /api/* → NestJS:4000
|
||||||
|
├── /uploads/* → 静态文件 /data/pingtai/uploads/
|
||||||
|
└── /* → Next.js:3001
|
||||||
|
PM2 管理: pingtai-api, pingtai-web
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. 扩展指南
|
||||||
|
|
||||||
|
### 9.1 新增策略维度
|
||||||
|
在 `ComparisonService.getSuggestion()` 中扩展：
|
||||||
|
1. 新增 `splitByCount[K]` 的 K 值
|
||||||
|
2. 或新增约束维度（如：按品牌集中、按地区集中、按最小起订量过滤）
|
||||||
|
3. 在 `recommendations` 中增加对应建议文案
|
||||||
|
|
||||||
|
### 9.2 多租户隔离
|
||||||
|
当前单租户设计，扩展多租户需：
|
||||||
|
- `Product` 增加 `tenantId` 字段 + 复合索引
|
||||||
|
- API 增加租户上下文中间件
|
||||||
|
- `getComparisonData` / `getSuggestion` 加 `where: { tenantId }`
|
||||||
|
|
||||||
|
### 9.3 报价有效期管理
|
||||||
|
- `Product` 增加 `validFrom` / `validTo` 字段
|
||||||
|
- 查询时过滤 `validTo >= now()`
|
||||||
|
- 前端增加"报价有效期"筛选器
|
||||||
|
|
||||||
|
### 9.4 供应商画像扩展
|
||||||
|
- 新增 `Supplier` 实体：资质、交付周期、历史履约率、质量评分
|
||||||
|
- 建议算法引入权重：`综合得分 = 价格权重*0.6 + 交付权重*0.2 + 质量权重*0.2`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. 附录：关键代码路径速查
|
||||||
|
|
||||||
|
| 功能 | 文件路径 |
|
||||||
|
|------|----------|
|
||||||
|
| 比价数据聚合 | `apps/api/src/comparison/comparison.service.ts:12-69` |
|
||||||
|
| 采购建议核心算法 | `apps/api/src/comparison/comparison.service.ts:131-314` |
|
||||||
|
| K家供应商组合优化 | `apps/api/src/comparison/comparison.service.ts:254-314` |
|
||||||
|
| 报价文件上传解析 | `apps/api/src/comparison/comparison.service.ts:71-129` |
|
||||||
|
| Python解析脚本 | `scripts/parse_products.py` |
|
||||||
|
| 前端三步向导主页 | `apps/web/src/components/ComparisonPage.tsx` |
|
||||||
|
| 比价表格渲染 | `ComparisonPage.tsx:256-320` |
|
||||||
|
| 采购建议报告UI | `ComparisonPage.tsx:327-623` |
|
||||||
|
| 导出CSV逻辑 | `ComparisonPage.tsx:606-617` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. 迁移清单 (给目标平台)
|
||||||
|
|
||||||
|
- [ ] 复刻 `Product` 数据模型到目标数据库
|
||||||
|
- [ ] 移植 `parse_products.py` 适配目标平台的文件存储
|
||||||
|
- [ ] 实现 `ComparisonService` 核心类（去重、聚合、三大策略算法）
|
||||||
|
- [ ] 暴露 3 个 REST API 端点
|
||||||
|
- [ ] 实现前端三步向导页面（上传/比价/建议）
|
||||||
|
- [ ] 集成文件上传组件 + 进度反馈
|
||||||
|
- [ ] 实现 CSV 导出功能
|
||||||
|
- [ ] 补充单元测试：去重逻辑、K组合枚举、边界情况（单供应商、无报价SKU）
|
||||||
|
- [ ] 压力测试：1000 SKU × 10 供应商 × K=4 组合枚举性能
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**文档结束** — 如有疑问可参考源码实现细节。建议先跑通核心算法单元测试，再集成前后端。
|
||||||
@ -118,9 +118,11 @@ export class CrawlerService implements OnModuleInit {
|
|||||||
results = await this.runGenericScript(config)
|
results = await this.runGenericScript(config)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
this.logger.log(`脚本返回 ${results.length} 条原始结果`)
|
||||||
let successCount = 0
|
let successCount = 0
|
||||||
|
|
||||||
const validResults = results.filter(r => r.noticeId)
|
const validResults = results.filter(r => r.noticeId)
|
||||||
|
this.logger.log(`有效结果 ${validResults.length} 条 (有 noticeId)`)
|
||||||
const categoryMap = await this.categoryInference.batchInfer(
|
const categoryMap = await this.categoryInference.batchInfer(
|
||||||
validResults.map(r => ({ id: r.noticeId, title: r.title })),
|
validResults.map(r => ({ id: r.noticeId, title: r.title })),
|
||||||
)
|
)
|
||||||
@ -161,11 +163,13 @@ export class CrawlerService implements OnModuleInit {
|
|||||||
},
|
},
|
||||||
})
|
})
|
||||||
successCount++
|
successCount++
|
||||||
} catch {
|
} catch (e: unknown) {
|
||||||
// 其他异常跳过
|
this.logger.warn(`入库失败 [${r.noticeId}] ${r.title.slice(0, 30)}: ${(e as Error).message}`)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
this.logger.log(`入库成功 ${successCount} / ${validResults.length} 条`)
|
||||||
|
|
||||||
return this.prisma.crawlerConfig.update({
|
return this.prisma.crawlerConfig.update({
|
||||||
where: { id },
|
where: { id },
|
||||||
data: {
|
data: {
|
||||||
@ -185,19 +189,42 @@ export class CrawlerService implements OnModuleInit {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private async runPowerChina(config: { keywords?: string | null }): Promise<FetchResult[]> {
|
private async resolveKeywords(config: { categories?: string | null; keywords?: string | null }): Promise<string[]> {
|
||||||
|
if (config.categories) {
|
||||||
|
try {
|
||||||
|
const catNames: string[] = JSON.parse(config.categories)
|
||||||
|
if (catNames.length > 0) {
|
||||||
|
const cats = await this.prisma.productCategory.findMany({
|
||||||
|
where: { name: { in: catNames } },
|
||||||
|
})
|
||||||
|
const kwSet = new Set<string>()
|
||||||
|
for (const cat of cats) {
|
||||||
|
try {
|
||||||
|
const kws: string[] = JSON.parse(cat.keywords)
|
||||||
|
kws.forEach((k) => kwSet.add(k))
|
||||||
|
} catch { /* skip */ }
|
||||||
|
}
|
||||||
|
if (kwSet.size > 0) return Array.from(kwSet)
|
||||||
|
}
|
||||||
|
} catch { /* fallback */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
if (config.keywords) {
|
||||||
|
try {
|
||||||
|
return JSON.parse(config.keywords)
|
||||||
|
} catch { /* skip */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
return ['']
|
||||||
|
}
|
||||||
|
|
||||||
|
private async runPowerChina(config: { categories?: string | null; keywords?: string | null }): Promise<FetchResult[]> {
|
||||||
const scriptPath = path.join(TENDER_FETCHER_DIR, 'fetch-v1.2.js')
|
const scriptPath = path.join(TENDER_FETCHER_DIR, 'fetch-v1.2.js')
|
||||||
if (!fs.existsSync(scriptPath)) {
|
if (!fs.existsSync(scriptPath)) {
|
||||||
throw new Error('中电建抓取脚本不存在')
|
throw new Error('中电建抓取脚本不存在')
|
||||||
}
|
}
|
||||||
|
|
||||||
let keywords: string[]
|
const keywords = await this.resolveKeywords(config)
|
||||||
try {
|
|
||||||
keywords = config.keywords ? JSON.parse(config.keywords) : []
|
|
||||||
} catch {
|
|
||||||
keywords = []
|
|
||||||
}
|
|
||||||
if (keywords.length === 0) keywords = ['']
|
|
||||||
|
|
||||||
const today = new Date().toISOString().split('T')[0]
|
const today = new Date().toISOString().split('T')[0]
|
||||||
const baseDir = path.join(process.env.HOME || '', 'desktop/招标公告', today)
|
const baseDir = path.join(process.env.HOME || '', 'desktop/招标公告', today)
|
||||||
@ -207,22 +234,29 @@ export class CrawlerService implements OnModuleInit {
|
|||||||
for (const kw of keywords) {
|
for (const kw of keywords) {
|
||||||
const kwFlag = kw ? ` --keyword "${kw}"` : ''
|
const kwFlag = kw ? ` --keyword "${kw}"` : ''
|
||||||
try {
|
try {
|
||||||
|
this.logger.log(`执行关键词 [${kw || '(空)'}] 抓取脚本`)
|
||||||
execSync(`node "${scriptPath}"${kwFlag}`, {
|
execSync(`node "${scriptPath}"${kwFlag}`, {
|
||||||
encoding: 'utf-8',
|
encoding: 'utf-8',
|
||||||
maxBuffer: 10 * 1024 * 1024,
|
maxBuffer: 10 * 1024 * 1024,
|
||||||
timeout: 180000,
|
timeout: 180000,
|
||||||
})
|
})
|
||||||
} catch {
|
} catch (e: unknown) {
|
||||||
// 单个 keyword 失败不影响其他 keyword
|
this.logger.warn(`关键词 [${kw || '(空)'}] 抓取失败: ${(e as { stderr?: string; message?: string }).stderr || (e as { message?: string }).message}`)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
const statePath = path.join(baseDir, 'downloaded.json')
|
// 每个关键词有独立的状态文件: downloaded_<keyword>.json
|
||||||
if (!fs.existsSync(statePath)) continue
|
const stateFileName = kw ? `downloaded_${kw}.json` : 'downloaded.json'
|
||||||
|
const statePath = path.join(baseDir, stateFileName)
|
||||||
|
if (!fs.existsSync(statePath)) {
|
||||||
|
this.logger.warn(`关键词 [${kw || '(空)'}] 状态文件不存在: ${statePath}`)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const state = JSON.parse(fs.readFileSync(statePath, 'utf-8'))
|
const state = JSON.parse(fs.readFileSync(statePath, 'utf-8'))
|
||||||
const downloaded = state.downloaded || {}
|
const downloaded = state.downloaded || {}
|
||||||
|
this.logger.log(`关键词 [${kw || '(空)'}] 状态文件读取成功: ${Object.keys(downloaded).length} 条记录`)
|
||||||
|
|
||||||
const outputDir = path.dirname(statePath)
|
const outputDir = path.dirname(statePath)
|
||||||
const pdfDir = path.join(outputDir, 'pdf')
|
const pdfDir = path.join(outputDir, 'pdf')
|
||||||
@ -265,15 +299,17 @@ export class CrawlerService implements OnModuleInit {
|
|||||||
regionDetail,
|
regionDetail,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
} catch {
|
} catch (e: unknown) {
|
||||||
|
this.logger.error(`关键词 [${kw || '(空)'}] 状态文件解析失败: ${(e as Error).message}`)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
this.logger.log(`抓取汇总: ${allResults.length} 条公告待入库`)
|
||||||
return allResults
|
return allResults
|
||||||
}
|
}
|
||||||
|
|
||||||
private async runGenericScript(config: { type: string; sourceName?: string | null; scriptPath?: string | null; keywords?: string | null }): Promise<FetchResult[]> {
|
private async runGenericScript(config: { type: string; sourceName?: string | null; scriptPath?: string | null; categories?: string | null; keywords?: string | null }): Promise<FetchResult[]> {
|
||||||
const slug = config.type.replace(/^bid_/, '')
|
const slug = config.type.replace(/^bid_/, '')
|
||||||
const scriptFile = config.scriptPath || `fetch-${slug}.py`
|
const scriptFile = config.scriptPath || `fetch-${slug}.py`
|
||||||
const scriptPath = path.join(TENDER_FETCHER_DIR, scriptFile)
|
const scriptPath = path.join(TENDER_FETCHER_DIR, scriptFile)
|
||||||
@ -284,13 +320,7 @@ export class CrawlerService implements OnModuleInit {
|
|||||||
throw new Error(`抓取脚本不存在: ${scriptFile}`)
|
throw new Error(`抓取脚本不存在: ${scriptFile}`)
|
||||||
}
|
}
|
||||||
|
|
||||||
let keywords: string[]
|
const keywords = await this.resolveKeywords(config)
|
||||||
try {
|
|
||||||
keywords = config.keywords ? JSON.parse(config.keywords) : []
|
|
||||||
} catch {
|
|
||||||
keywords = []
|
|
||||||
}
|
|
||||||
if (keywords.length === 0) keywords = ['']
|
|
||||||
|
|
||||||
const today = new Date().toISOString().split('T')[0]
|
const today = new Date().toISOString().split('T')[0]
|
||||||
const baseDir = path.join(process.env.HOME || '', 'desktop/招标公告', today)
|
const baseDir = path.join(process.env.HOME || '', 'desktop/招标公告', today)
|
||||||
|
|||||||
@ -5,13 +5,14 @@ import { CrawlerModule } from '../crawler/crawler.module'
|
|||||||
import { ProductCategoryModule } from '../product-category/product-category.module'
|
import { ProductCategoryModule } from '../product-category/product-category.module'
|
||||||
import { LeadService } from './lead.service'
|
import { LeadService } from './lead.service'
|
||||||
import { MaterialExtractorService } from './material-extractor.service'
|
import { MaterialExtractorService } from './material-extractor.service'
|
||||||
|
import { TitleCompressorService } from './title-compressor.service'
|
||||||
import { LeadController } from './lead.controller'
|
import { LeadController } from './lead.controller'
|
||||||
import { LeadNotesController } from './lead-notes.controller'
|
import { LeadNotesController } from './lead-notes.controller'
|
||||||
|
|
||||||
@Module({
|
@Module({
|
||||||
imports: [PrismaModule, IntegrationModule, CrawlerModule, ProductCategoryModule],
|
imports: [PrismaModule, IntegrationModule, CrawlerModule, ProductCategoryModule],
|
||||||
controllers: [LeadController, LeadNotesController],
|
controllers: [LeadController, LeadNotesController],
|
||||||
providers: [LeadService, MaterialExtractorService],
|
providers: [LeadService, MaterialExtractorService, TitleCompressorService],
|
||||||
exports: [LeadService],
|
exports: [LeadService],
|
||||||
})
|
})
|
||||||
export class LeadModule {}
|
export class LeadModule {}
|
||||||
|
|||||||
@ -4,6 +4,7 @@ import { IntegrationService } from '../integration/integration.service'
|
|||||||
import { ProductCategoryService } from '../product-category/product-category.service'
|
import { ProductCategoryService } from '../product-category/product-category.service'
|
||||||
import { CategoryInferenceService } from '../product-category/category-inference.service'
|
import { CategoryInferenceService } from '../product-category/category-inference.service'
|
||||||
import { MaterialExtractorService } from './material-extractor.service'
|
import { MaterialExtractorService } from './material-extractor.service'
|
||||||
|
import { TitleCompressorService } from './title-compressor.service'
|
||||||
|
|
||||||
@Injectable()
|
@Injectable()
|
||||||
export class LeadService {
|
export class LeadService {
|
||||||
@ -15,6 +16,7 @@ export class LeadService {
|
|||||||
private categoryService: ProductCategoryService,
|
private categoryService: ProductCategoryService,
|
||||||
private categoryInference: CategoryInferenceService,
|
private categoryInference: CategoryInferenceService,
|
||||||
private materialExtractor: MaterialExtractorService,
|
private materialExtractor: MaterialExtractorService,
|
||||||
|
private titleCompressor: TitleCompressorService,
|
||||||
) {}
|
) {}
|
||||||
|
|
||||||
async findAll(params?: { status?: string; level?: string; search?: string; star?: string; source?: string; category?: string; assignedTo?: string; timeRange?: string }) {
|
async findAll(params?: { status?: string; level?: string; search?: string; star?: string; source?: string; category?: string; assignedTo?: string; timeRange?: string }) {
|
||||||
@ -35,11 +37,7 @@ export class LeadService {
|
|||||||
}
|
}
|
||||||
if (params?.source) where.source = params.source
|
if (params?.source) where.source = params.source
|
||||||
if (params?.category) {
|
if (params?.category) {
|
||||||
const categoryRecord = await this.prisma.productCategory.findUnique({ where: { name: params.category } })
|
where.category = params.category
|
||||||
if (categoryRecord) {
|
|
||||||
const keywords: string[] = JSON.parse(categoryRecord.keywords)
|
|
||||||
where.OR = keywords.map(kw => ({ title: { contains: kw } }))
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if (params?.assignedTo) {
|
if (params?.assignedTo) {
|
||||||
where.assignedTo = params.assignedTo === '未分配' ? null : params.assignedTo
|
where.assignedTo = params.assignedTo === '未分配' ? null : params.assignedTo
|
||||||
@ -71,9 +69,7 @@ export class LeadService {
|
|||||||
}
|
}
|
||||||
return this.prisma.lead.findMany({
|
return this.prisma.lead.findMany({
|
||||||
where,
|
where,
|
||||||
// 最新入库优先：createdAt 降序为主，deadline 仅作同批次内的次序参考
|
orderBy: [{ publishTime: 'desc' }, { createdAt: 'desc' }],
|
||||||
// （旧版用 deadline desc 会把截止日远/为空的最新数据沉到 700+ 名开外）
|
|
||||||
orderBy: [{ createdAt: 'desc' }, { deadline: 'desc' }],
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -318,9 +314,15 @@ export class LeadService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// 预校验: B2B 平台字段长度限制 (tb_inquiry.title varchar(50))
|
// 预校验: B2B 平台字段长度限制 (tb_inquiry.title varchar(50))
|
||||||
// 在调 MCP 之前先报错，避免拿到通用的"服务器内部异常"
|
// 超长标题先 AI 压缩（保留 公司名+物料+采购类型），压缩失败自动硬截断兜底
|
||||||
const TITLE_MAX_LEN = 50
|
const TITLE_MAX_LEN = 50
|
||||||
if (publishArgs.title && String(publishArgs.title).length > TITLE_MAX_LEN) {
|
if (publishArgs.title && String(publishArgs.title).length > TITLE_MAX_LEN) {
|
||||||
|
publishArgs.title = await this.titleCompressor.compress(
|
||||||
|
String(publishArgs.title),
|
||||||
|
TITLE_MAX_LEN,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
if (String(publishArgs.title).length > TITLE_MAX_LEN) {
|
||||||
const titleLen = String(publishArgs.title).length
|
const titleLen = String(publishArgs.title).length
|
||||||
throw new ServiceUnavailableException(
|
throw new ServiceUnavailableException(
|
||||||
[
|
[
|
||||||
|
|||||||
88
apps/api/src/lead/title-compressor.service.ts
Normal file
88
apps/api/src/lead/title-compressor.service.ts
Normal file
@ -0,0 +1,88 @@
|
|||||||
|
import { Injectable, Logger } from '@nestjs/common'
|
||||||
|
|
||||||
|
const MINIMAX_ENDPOINT = 'https://api.minimaxi.com/v1/chat/completions'
|
||||||
|
const MINIMAX_MODEL = process.env.MINIMAX_MODEL || 'MiniMax-M2.7'
|
||||||
|
const MINIMAX_TIMEOUT_MS = 30000
|
||||||
|
|
||||||
|
// 压缩规则: 只保留 公司名(简称) + 物料 + 采购类型
|
||||||
|
// 给 B2B 平台 50 字上限留 5 字余量, 避免平台侧校验边界
|
||||||
|
const MAX_COMPRESS_LEN = 45
|
||||||
|
// M2.7 推理模型会把思考过程输出在正文里且风格多变, 行级正则过滤不可靠,
|
||||||
|
// 改为要求 JSON 结构化输出, 解析不到就硬截断兜底
|
||||||
|
const PROMPT = `把采购公告标题压缩成询价标题, 只保留: 采购公司名称(简称)、物料、采购类型。
|
||||||
|
要求:
|
||||||
|
1. 压缩后不超过 45 个汉字
|
||||||
|
2. 只输出 JSON: {"title": "压缩后的标题"}
|
||||||
|
3. 不要输出任何其他内容、解释、序号、markdown
|
||||||
|
|
||||||
|
示例:
|
||||||
|
输入: 中国电建集团港航建设有限公司 定襄县农村饮水管网改造工程第1标段项目管材采购公开询比采购公告
|
||||||
|
输出: {"title": "电建港航定襄县管网改造一标管材询比采购"}`
|
||||||
|
|
||||||
|
@Injectable()
|
||||||
|
export class TitleCompressorService {
|
||||||
|
private readonly logger = new Logger(TitleCompressorService.name)
|
||||||
|
|
||||||
|
async compress(title: string, maxLength: number): Promise<string> {
|
||||||
|
if (!title || title.length <= maxLength) return title
|
||||||
|
|
||||||
|
const apiKey = process.env.MINIMAX_API_KEY
|
||||||
|
if (!apiKey) {
|
||||||
|
this.logger.warn('MINIMAX_API_KEY 未配置, 标题硬截断')
|
||||||
|
return title.slice(0, maxLength)
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const raw = await this.callMinimax(title, apiKey, Math.min(maxLength, MAX_COMPRESS_LEN))
|
||||||
|
const parsedTitle = this.extractJsonTitle(raw)
|
||||||
|
|
||||||
|
if (!parsedTitle) {
|
||||||
|
this.logger.warn(`AI 压缩无有效输出, 硬截断: ${raw.slice(0, 100)}`)
|
||||||
|
return title.slice(0, maxLength)
|
||||||
|
}
|
||||||
|
|
||||||
|
const result = parsedTitle.length > maxLength ? parsedTitle.slice(0, maxLength) : parsedTitle
|
||||||
|
this.logger.log(`标题 AI 压缩 (${title.length}字→${result.length}字): "${result}"`)
|
||||||
|
return result
|
||||||
|
} catch (e: any) {
|
||||||
|
this.logger.warn(`AI 压缩失败, 硬截断: ${e.message}`)
|
||||||
|
return title.slice(0, maxLength)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** M2.7 常在 JSON 前后夹杂思考文本, 用正则提取第一个合法的 {"title":"..."} 片段 */
|
||||||
|
private extractJsonTitle(raw: string): string {
|
||||||
|
const m = raw.match(/\{"title"\s*:\s*"([^"]+)"\}/)
|
||||||
|
if (m) return m[1]
|
||||||
|
const loose = raw.match(/"title"\s*:\s*"([^"]+)"/)
|
||||||
|
return loose ? loose[1] : ''
|
||||||
|
}
|
||||||
|
|
||||||
|
private async callMinimax(title: string, apiKey: string, maxLen: number): Promise<string> {
|
||||||
|
const r = await fetch(MINIMAX_ENDPOINT, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Authorization': `Bearer ${apiKey}`,
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
model: MINIMAX_MODEL,
|
||||||
|
messages: [
|
||||||
|
{ role: 'system', content: PROMPT.replace('45', String(maxLen)) },
|
||||||
|
{ role: 'user', content: title },
|
||||||
|
],
|
||||||
|
temperature: 0,
|
||||||
|
max_tokens: 800, // M2.7 推理模型会输出明文思考, 400 会被思考过程耗尽导致答案截断 (AGENTS.md: 标题提取用 800)
|
||||||
|
}),
|
||||||
|
signal: AbortSignal.timeout(MINIMAX_TIMEOUT_MS),
|
||||||
|
})
|
||||||
|
|
||||||
|
if (!r.ok) {
|
||||||
|
const errText = await r.text().catch(() => '')
|
||||||
|
throw new Error(`MiniMax API ${r.status}: ${errText.slice(0, 200)}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
const data = (await r.json()) as { choices?: Array<{ message?: { content?: string } }> }
|
||||||
|
return data.choices?.[0]?.message?.content || ''
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -1,6 +1,6 @@
|
|||||||
import { Injectable, Logger } from '@nestjs/common'
|
import { Injectable, Logger } from '@nestjs/common'
|
||||||
import { PrismaService } from '../prisma/prisma.service'
|
import { PrismaService } from '../prisma/prisma.service'
|
||||||
import { ProductCategoryService } from '../product-category/product-category.service'
|
import { isGarbageKeyword, ProductCategoryService } from '../product-category/product-category.service'
|
||||||
|
|
||||||
const MINIMAX_ENDPOINT = 'https://api.minimaxi.com/v1/chat/completions'
|
const MINIMAX_ENDPOINT = 'https://api.minimaxi.com/v1/chat/completions'
|
||||||
const MINIMAX_MODEL = process.env.MINIMAX_MODEL || 'MiniMax-M2.7'
|
const MINIMAX_MODEL = process.env.MINIMAX_MODEL || 'MiniMax-M2.7'
|
||||||
@ -88,7 +88,10 @@ export class CategoryInferenceService {
|
|||||||
try {
|
try {
|
||||||
const aiCategory = sanitizeOne(await this.callMinimax(title, summary, apiKey))
|
const aiCategory = sanitizeOne(await this.callMinimax(title, summary, apiKey))
|
||||||
if (aiCategory) {
|
if (aiCategory) {
|
||||||
await this.categoryService.addKeywordToCategory(aiCategory, this.extractMainKeyword(title))
|
const keyword = this.extractMainKeyword(title)
|
||||||
|
if (keyword) {
|
||||||
|
await this.categoryService.addKeywordToCategory(aiCategory, keyword)
|
||||||
|
}
|
||||||
this.logger.log(`AI 分类: "${title.slice(0, 40)}..." → ${aiCategory}`)
|
this.logger.log(`AI 分类: "${title.slice(0, 40)}..." → ${aiCategory}`)
|
||||||
return aiCategory
|
return aiCategory
|
||||||
}
|
}
|
||||||
@ -163,7 +166,10 @@ export class CategoryInferenceService {
|
|||||||
const catName = sanitizeOne(lines[j] || '')
|
const catName = sanitizeOne(lines[j] || '')
|
||||||
if (catName) {
|
if (catName) {
|
||||||
result.set(batch[j].id, catName)
|
result.set(batch[j].id, catName)
|
||||||
await this.categoryService.addKeywordToCategory(catName, this.extractMainKeyword(batch[j].title))
|
const keyword = this.extractMainKeyword(batch[j].title)
|
||||||
|
if (keyword) {
|
||||||
|
await this.categoryService.addKeywordToCategory(catName, keyword)
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
result.set(batch[j].id, '')
|
result.set(batch[j].id, '')
|
||||||
}
|
}
|
||||||
@ -209,7 +215,17 @@ export class CategoryInferenceService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private extractMainKeyword(title: string): string {
|
private extractMainKeyword(title: string): string {
|
||||||
const match = title.match(/([\u4e00-\u9fa5]{2,6})采购/)
|
// 取最后一个"采购"前面的 2-6 个中文字（更靠近实际物料名）
|
||||||
return match ? match[1] : title.slice(0, 20)
|
let lastMatch = ''
|
||||||
|
const regex = /([\u4e00-\u9fa5]{2,6})采购/g
|
||||||
|
let match: RegExpExecArray | null
|
||||||
|
while ((match = regex.exec(title)) !== null) {
|
||||||
|
lastMatch = match[1]
|
||||||
|
}
|
||||||
|
if (!lastMatch) return ''
|
||||||
|
|
||||||
|
// 跳过公司名/组织机构名
|
||||||
|
if (isGarbageKeyword(lastMatch)) return ''
|
||||||
|
return lastMatch
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,6 +1,15 @@
|
|||||||
import { Injectable, Logger } from '@nestjs/common'
|
import { Injectable, Logger } from '@nestjs/common'
|
||||||
import { PrismaService } from '../prisma/prisma.service'
|
import { PrismaService } from '../prisma/prisma.service'
|
||||||
|
|
||||||
|
/** 判断是否是公司名/组织机构名等垃圾关键词，不应参与关键词匹配或反哺 */
|
||||||
|
export function isGarbageKeyword(kw: string): boolean {
|
||||||
|
if (/(?:公司|有限|集团|工程局|项目部)$/.test(kw)) return true
|
||||||
|
if (/^(?:中国电建|新疆)/.test(kw)) return true
|
||||||
|
const genericTerms = ['公开竞价', '公开招标', '竞争性谈判', '询价采购', '邀请招标']
|
||||||
|
if (genericTerms.includes(kw)) return true
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
export interface CategoryMatch {
|
export interface CategoryMatch {
|
||||||
name: string
|
name: string
|
||||||
matchedBy: 'keyword' | 'ai' | null
|
matchedBy: 'keyword' | 'ai' | null
|
||||||
@ -38,6 +47,7 @@ export class ProductCategoryService {
|
|||||||
for (const cat of categories) {
|
for (const cat of categories) {
|
||||||
const keywords: string[] = JSON.parse(cat.keywords)
|
const keywords: string[] = JSON.parse(cat.keywords)
|
||||||
for (const kw of keywords) {
|
for (const kw of keywords) {
|
||||||
|
if (isGarbageKeyword(kw)) continue
|
||||||
if (title.includes(kw)) {
|
if (title.includes(kw)) {
|
||||||
return { name: cat.name, matchedBy: 'keyword' }
|
return { name: cat.name, matchedBy: 'keyword' }
|
||||||
}
|
}
|
||||||
@ -47,6 +57,7 @@ export class ProductCategoryService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async addKeywordToCategory(categoryName: string, keyword: string) {
|
async addKeywordToCategory(categoryName: string, keyword: string) {
|
||||||
|
if (!keyword || isGarbageKeyword(keyword)) return
|
||||||
const cat = await this.prisma.productCategory.findUnique({ where: { name: categoryName } })
|
const cat = await this.prisma.productCategory.findUnique({ where: { name: categoryName } })
|
||||||
if (!cat) {
|
if (!cat) {
|
||||||
await this.prisma.productCategory.create({
|
await this.prisma.productCategory.create({
|
||||||
|
|||||||
@ -111,6 +111,7 @@ export default function InquiryPage({ onLeadDetail }: { onLeadDetail?: (id: stri
|
|||||||
const { showToast } = useToast()
|
const { showToast } = useToast()
|
||||||
const pageSize = 10
|
const pageSize = 10
|
||||||
|
|
||||||
|
const [fSearch, setFSearch] = useState('')
|
||||||
const [fTime, setFTime] = useState('全部日期')
|
const [fTime, setFTime] = useState('全部日期')
|
||||||
const [fStar, setFStar] = useState('全部星级')
|
const [fStar, setFStar] = useState('全部星级')
|
||||||
const [fStatus, setFStatus] = useState('全部状态')
|
const [fStatus, setFStatus] = useState('全部状态')
|
||||||
@ -144,9 +145,10 @@ export default function InquiryPage({ onLeadDetail }: { onLeadDetail?: (id: stri
|
|||||||
const [fSource, setFSource] = useState('全部来源')
|
const [fSource, setFSource] = useState('全部来源')
|
||||||
const [fPerson, setFPerson] = useState('全部人员')
|
const [fPerson, setFPerson] = useState('全部人员')
|
||||||
|
|
||||||
const loadLeads = useCallback(async (star?: string, status?: string, level?: string, cat?: string, source?: string, person?: string, timeRange?: string) => {
|
const loadLeads = useCallback(async (search?: string, star?: string, status?: string, level?: string, cat?: string, source?: string, person?: string, timeRange?: string) => {
|
||||||
setLoading(true)
|
setLoading(true)
|
||||||
const params = new URLSearchParams()
|
const params = new URLSearchParams()
|
||||||
|
if (search && search.trim()) params.set('search', search.trim())
|
||||||
if (star && star !== '全部星级') {
|
if (star && star !== '全部星级') {
|
||||||
if (star === '5星') params.set('star', '5')
|
if (star === '5星') params.set('star', '5')
|
||||||
else if (star === '4星') params.set('star', '4')
|
else if (star === '4星') params.set('star', '4')
|
||||||
@ -177,10 +179,10 @@ export default function InquiryPage({ onLeadDetail }: { onLeadDetail?: (id: stri
|
|||||||
}).catch(() => {})
|
}).catch(() => {})
|
||||||
}, [])
|
}, [])
|
||||||
|
|
||||||
useEffect(() => { loadLeads(fStar, fStatus, fLevel, fCat, fSource, fPerson, fTime) }, [loadLeads, fStar, fStatus, fLevel, fCat, fSource, fPerson, fTime])
|
useEffect(() => { loadLeads(fSearch, fStar, fStatus, fLevel, fCat, fSource, fPerson, fTime) }, [loadLeads, fSearch, fStar, fStatus, fLevel, fCat, fSource, fPerson, fTime])
|
||||||
|
|
||||||
const refreshLeads = async () => {
|
const refreshLeads = async () => {
|
||||||
await loadLeads(fStar, fStatus, fLevel, fCat, fSource, fPerson, fTime)
|
await loadLeads(fSearch, fStar, fStatus, fLevel, fCat, fSource, fPerson, fTime)
|
||||||
}
|
}
|
||||||
|
|
||||||
const totalPages = Math.ceil(filtered.length / pageSize)
|
const totalPages = Math.ceil(filtered.length / pageSize)
|
||||||
@ -501,6 +503,26 @@ export default function InquiryPage({ onLeadDetail }: { onLeadDetail?: (id: stri
|
|||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
<div className="flex flex-col gap-1">
|
||||||
|
<span className="text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">搜索</span>
|
||||||
|
<div className="relative">
|
||||||
|
<input
|
||||||
|
type="text"
|
||||||
|
value={fSearch}
|
||||||
|
onChange={e => setFSearch(e.target.value)}
|
||||||
|
placeholder="输入关键词搜索"
|
||||||
|
className="bg-surface-container-low border border-border-subtle rounded pl-2 pr-6 py-1 text-xs w-40 focus:ring-secondary outline-none placeholder:text-slate-400"
|
||||||
|
/>
|
||||||
|
{fSearch && (
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => setFSearch('')}
|
||||||
|
className="absolute right-1 top-1/2 -translate-y-1/2 w-4 h-4 flex items-center justify-center text-slate-400 hover:text-slate-600 hover:bg-slate-200 rounded-full text-[10px]"
|
||||||
|
title="清空搜索"
|
||||||
|
>×</button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div className="flex items-center gap-2 ml-auto">
|
<div className="flex items-center gap-2 ml-auto">
|
||||||
<span className="text-xs text-on-surface-variant">批量操作:</span>
|
<span className="text-xs text-on-surface-variant">批量操作:</span>
|
||||||
|
|||||||
@ -11,15 +11,6 @@ interface WeightItem {
|
|||||||
max: number
|
max: number
|
||||||
}
|
}
|
||||||
|
|
||||||
interface Keyword {
|
|
||||||
id: number
|
|
||||||
word: string
|
|
||||||
category: string
|
|
||||||
weight: number
|
|
||||||
matchCount: number
|
|
||||||
enabled: boolean
|
|
||||||
}
|
|
||||||
|
|
||||||
interface ExampleLead {
|
interface ExampleLead {
|
||||||
name: string
|
name: string
|
||||||
amount: string
|
amount: string
|
||||||
@ -71,26 +62,6 @@ const EXAMPLE_LEADS: ExampleLead[] = [
|
|||||||
{ name: '信阳乡村振兴供水项目管材采购', amount: '65万', score: 45 },
|
{ name: '信阳乡村振兴供水项目管材采购', amount: '65万', score: 45 },
|
||||||
]
|
]
|
||||||
|
|
||||||
const CATEGORIES = ['工程建材', '医疗器械', 'IT设备', '办公用品', '物流服务', '咨询服务']
|
|
||||||
|
|
||||||
const MOCK_KEYWORDS: Keyword[] = [
|
|
||||||
{ id: 1, word: '防火门', category: '工程建材', weight: 9, matchCount: 47, enabled: true },
|
|
||||||
{ id: 2, word: '配电箱', category: '工程建材', weight: 8, matchCount: 35, enabled: true },
|
|
||||||
{ id: 3, word: '电线电缆', category: '工程建材', weight: 8, matchCount: 42, enabled: true },
|
|
||||||
{ id: 4, word: '阀门', category: '工程建材', weight: 7, matchCount: 28, enabled: true },
|
|
||||||
{ id: 5, word: '钢材', category: '工程建材', weight: 9, matchCount: 56, enabled: true },
|
|
||||||
{ id: 6, word: '混凝土', category: '工程建材', weight: 7, matchCount: 33, enabled: true },
|
|
||||||
{ id: 7, word: 'CT设备', category: '医疗器械', weight: 8, matchCount: 12, enabled: true },
|
|
||||||
{ id: 8, word: 'MRI核磁共振', category: '医疗器械', weight: 9, matchCount: 8, enabled: true },
|
|
||||||
{ id: 9, word: '服务器', category: 'IT设备', weight: 7, matchCount: 15, enabled: true },
|
|
||||||
{ id: 10, word: '交换机', category: 'IT设备', weight: 6, matchCount: 11, enabled: true },
|
|
||||||
{ id: 11, word: '办公桌椅', category: '办公用品', weight: 4, matchCount: 23, enabled: false },
|
|
||||||
{ id: 12, word: '打印设备', category: '办公用品', weight: 5, matchCount: 19, enabled: true },
|
|
||||||
{ id: 13, word: '物流运输', category: '物流服务', weight: 6, matchCount: 31, enabled: true },
|
|
||||||
{ id: 14, word: '仓储服务', category: '物流服务', weight: 5, matchCount: 14, enabled: true },
|
|
||||||
{ id: 15, word: '管理咨询', category: '咨询服务', weight: 4, matchCount: 9, enabled: false },
|
|
||||||
]
|
|
||||||
|
|
||||||
const PERSONS = ['张三', '李四', '王五', '赵六', '钱七']
|
const PERSONS = ['张三', '李四', '王五', '赵六', '钱七']
|
||||||
|
|
||||||
const DEFAULT_DISTRIBUTION: DistributeRule[] = [
|
const DEFAULT_DISTRIBUTION: DistributeRule[] = [
|
||||||
@ -142,17 +113,16 @@ function getLevelClass(level: string) {
|
|||||||
export default function RuleCenterPage() {
|
export default function RuleCenterPage() {
|
||||||
const [activeTab, setActiveTab] = useState<'weights' | 'keywords' | 'client' | 'crawlers'>('weights')
|
const [activeTab, setActiveTab] = useState<'weights' | 'keywords' | 'client' | 'crawlers'>('weights')
|
||||||
const [weights, setWeights] = useState<WeightItem[]>(DEFAULT_WEIGHTS)
|
const [weights, setWeights] = useState<WeightItem[]>(DEFAULT_WEIGHTS)
|
||||||
const [keywords, setKeywords] = useState<Keyword[]>(MOCK_KEYWORDS)
|
|
||||||
const [saved, setSaved] = useState(false)
|
const [saved, setSaved] = useState(false)
|
||||||
const [nextKeywordId, setNextKeywordId] = useState(16)
|
// 品类管理
|
||||||
|
const [categories, setCategories] = useState<any[]>([])
|
||||||
const [newWord, setNewWord] = useState('')
|
const [catLoading, setCatLoading] = useState(true)
|
||||||
const [newCategory, setNewCategory] = useState(CATEGORIES[0])
|
const [showAddCat, setShowAddCat] = useState(false)
|
||||||
const [newWeight, setNewWeight] = useState(5)
|
const [addCatName, setAddCatName] = useState('')
|
||||||
|
const [addCatKeywords, setAddCatKeywords] = useState('')
|
||||||
const [showBatchModal, setShowBatchModal] = useState(false)
|
const [editCatId, setEditCatId] = useState<string | null>(null)
|
||||||
const [batchText, setBatchText] = useState('')
|
const [editCatName, setEditCatName] = useState('')
|
||||||
const [showClearModal, setShowClearModal] = useState(false)
|
const [editCatKeywords, setEditCatKeywords] = useState('')
|
||||||
|
|
||||||
const [sAmount, setSAmount] = useState(100)
|
const [sAmount, setSAmount] = useState(100)
|
||||||
const [sCoop, setSCoop] = useState(3)
|
const [sCoop, setSCoop] = useState(3)
|
||||||
@ -168,9 +138,9 @@ export default function RuleCenterPage() {
|
|||||||
const [crawlers, setCrawlers] = useState<CrawlerConfig[]>([])
|
const [crawlers, setCrawlers] = useState<CrawlerConfig[]>([])
|
||||||
const [crawlersLoading, setCrawlersLoading] = useState(true)
|
const [crawlersLoading, setCrawlersLoading] = useState(true)
|
||||||
const [editCrawlerId, setEditCrawlerId] = useState<string | null>(null)
|
const [editCrawlerId, setEditCrawlerId] = useState<string | null>(null)
|
||||||
const [editForm, setEditForm] = useState<{ name: string; targetUrl: string; type: string; sourceName: string; scriptPath: string; schedule: string; keywords: string; categories: string }>({ name: '', targetUrl: '', type: 'bid', sourceName: '', scriptPath: '', schedule: '', keywords: '', categories: '' })
|
const [editForm, setEditForm] = useState<{ name: string; targetUrl: string; type: string; sourceName: string; scriptPath: string; schedule: string; keywords: string; categories: string }>({ name: '', targetUrl: '', type: 'bid', sourceName: '', scriptPath: '', schedule: '', keywords: '', categories: '[]' })
|
||||||
const [showAddModal, setShowAddModal] = useState(false)
|
const [showAddModal, setShowAddModal] = useState(false)
|
||||||
const [addForm, setAddForm] = useState<{ name: string; targetUrl: string; type: string; sourceName: string; scriptPath: string; schedule: string; keywords: string; categories: string }>({ name: '', targetUrl: '', type: 'bid', sourceName: '', scriptPath: '', schedule: '0 */6 * * *', keywords: '', categories: '' })
|
const [addForm, setAddForm] = useState<{ name: string; targetUrl: string; type: string; sourceName: string; scriptPath: string; schedule: string; keywords: string; categories: string }>({ name: '', targetUrl: '', type: 'bid', sourceName: '', scriptPath: '', schedule: '0 */6 * * *', keywords: '', categories: '[]' })
|
||||||
const [runModal, setRunModal] = useState<{ id: string; name: string } | null>(null)
|
const [runModal, setRunModal] = useState<{ id: string; name: string } | null>(null)
|
||||||
const [runResult, setRunResult] = useState<'running' | 'success' | 'error'>('running')
|
const [runResult, setRunResult] = useState<'running' | 'success' | 'error'>('running')
|
||||||
|
|
||||||
@ -231,53 +201,67 @@ export default function RuleCenterPage() {
|
|||||||
setTimeout(() => setSaved(false), 2000)
|
setTimeout(() => setSaved(false), 2000)
|
||||||
}
|
}
|
||||||
|
|
||||||
const handleAddKeyword = () => {
|
// 品类管理 - 获取列表
|
||||||
if (!newWord.trim()) return
|
const fetchCategories = () => {
|
||||||
setKeywords(prev => [
|
setCatLoading(true)
|
||||||
...prev,
|
fetch(`${API_BASE}/api/categories`)
|
||||||
{ id: nextKeywordId, word: newWord.trim(), category: newCategory, weight: newWeight, matchCount: 0, enabled: true },
|
.then(r => r.json())
|
||||||
])
|
.then(data => setCategories(data))
|
||||||
setNextKeywordId(prev => prev + 1)
|
.catch(() => {})
|
||||||
setNewWord('')
|
.finally(() => setCatLoading(false))
|
||||||
setNewWeight(5)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
const toggleKeyword = (id: number) => {
|
// 品类管理 - 添加
|
||||||
setKeywords(prev => prev.map(k => k.id === id ? { ...k, enabled: !k.enabled } : k))
|
const handleAddCategory = async () => {
|
||||||
|
if (!addCatName.trim()) return
|
||||||
|
const kwArr = addCatKeywords.split(/[,，、\s]+/).filter(Boolean)
|
||||||
|
try {
|
||||||
|
const res = await fetch(`${API_BASE}/api/categories`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ name: addCatName.trim(), keywords: kwArr }),
|
||||||
|
})
|
||||||
|
if (!res.ok) throw new Error()
|
||||||
|
const created = await res.json()
|
||||||
|
setCategories(prev => [...prev, created])
|
||||||
|
setShowAddCat(false)
|
||||||
|
setAddCatName('')
|
||||||
|
setAddCatKeywords('')
|
||||||
|
} catch { alert('添加品类失败') }
|
||||||
}
|
}
|
||||||
|
|
||||||
const deleteKeyword = (id: number) => {
|
// 品类管理 - 编辑
|
||||||
setKeywords(prev => prev.filter(k => k.id !== id))
|
const startEditCat = (cat: any) => {
|
||||||
|
setEditCatId(cat.id)
|
||||||
|
setEditCatName(cat.name)
|
||||||
|
setEditCatKeywords((() => {
|
||||||
|
try { return (JSON.parse(cat.keywords) as string[]).join(', ') } catch { return cat.keywords || '' }
|
||||||
|
})())
|
||||||
}
|
}
|
||||||
|
|
||||||
const editKeyword = (id: number) => {
|
const handleUpdateCategory = async () => {
|
||||||
const kw = keywords.find(k => k.id === id)
|
if (!editCatId || !editCatName.trim()) return
|
||||||
if (!kw) return
|
const kwArr = editCatKeywords.split(/[,，、\s]+/).filter(Boolean)
|
||||||
setNewWord(kw.word)
|
try {
|
||||||
setNewCategory(kw.category)
|
const res = await fetch(`${API_BASE}/api/categories/${editCatId}`, {
|
||||||
setNewWeight(kw.weight)
|
method: 'PUT',
|
||||||
setKeywords(prev => prev.filter(k => k.id !== id))
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ name: editCatName.trim(), keywords: kwArr }),
|
||||||
|
})
|
||||||
|
if (!res.ok) throw new Error()
|
||||||
|
const updated = await res.json()
|
||||||
|
setCategories(prev => prev.map(c => c.id === editCatId ? updated : c))
|
||||||
|
setEditCatId(null)
|
||||||
|
} catch { alert('更新品类失败') }
|
||||||
}
|
}
|
||||||
|
|
||||||
const handleBatchImport = () => {
|
// 品类管理 - 删除
|
||||||
const lines = batchText.split('\n').map(l => l.trim()).filter(Boolean)
|
const handleDeleteCategory = async (id: string, name: string) => {
|
||||||
const newKeywords: Keyword[] = lines.map((line, i) => ({
|
if (!confirm(`确定要删除品类「${name}」吗？已有线索的该品类不受影响。`)) return
|
||||||
id: nextKeywordId + i,
|
try {
|
||||||
word: line,
|
await fetch(`${API_BASE}/api/categories/${id}`, { method: 'DELETE' })
|
||||||
category: '工程建材',
|
setCategories(prev => prev.filter(c => c.id !== id))
|
||||||
weight: 5,
|
} catch { alert('删除品类失败') }
|
||||||
matchCount: 0,
|
|
||||||
enabled: true,
|
|
||||||
}))
|
|
||||||
setKeywords(prev => [...prev, ...newKeywords])
|
|
||||||
setNextKeywordId(prev => prev + lines.length)
|
|
||||||
setBatchText('')
|
|
||||||
setShowBatchModal(false)
|
|
||||||
}
|
|
||||||
|
|
||||||
const handleClearAll = () => {
|
|
||||||
setKeywords([])
|
|
||||||
setShowClearModal(false)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
const handleDistributeChange = (level: string, person: string) => {
|
const handleDistributeChange = (level: string, person: string) => {
|
||||||
@ -294,6 +278,9 @@ export default function RuleCenterPage() {
|
|||||||
.finally(() => setCrawlersLoading(false))
|
.finally(() => setCrawlersLoading(false))
|
||||||
}, [])
|
}, [])
|
||||||
|
|
||||||
|
// 初次加载品类列表
|
||||||
|
useEffect(() => { fetchCategories() }, [])
|
||||||
|
|
||||||
const startEditCrawler = (c: CrawlerConfig) => {
|
const startEditCrawler = (c: CrawlerConfig) => {
|
||||||
setEditCrawlerId(c.id)
|
setEditCrawlerId(c.id)
|
||||||
setEditForm({
|
setEditForm({
|
||||||
@ -303,18 +290,17 @@ export default function RuleCenterPage() {
|
|||||||
sourceName: c.sourceName || '',
|
sourceName: c.sourceName || '',
|
||||||
scriptPath: c.scriptPath || '',
|
scriptPath: c.scriptPath || '',
|
||||||
schedule: c.schedule || '',
|
schedule: c.schedule || '',
|
||||||
keywords: c.keywords ? (() => { try { return JSON.parse(c.keywords).join(', ') } catch { return c.keywords } })() : '',
|
keywords: '',
|
||||||
categories: c.categories ? (() => { try { return JSON.parse(c.categories).join(', ') } catch { return c.categories } })() : '',
|
categories: c.categories ? (() => { try { const p = JSON.parse(c.categories); return Array.isArray(p) ? JSON.stringify(p) : '[]' } catch { return '[]' } })() : '[]',
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
const saveCrawler = async (id: string) => {
|
const saveCrawler = async (id: string) => {
|
||||||
const payload: Record<string, string | null> = { name: editForm.name, targetUrl: editForm.targetUrl, type: editForm.type, sourceName: editForm.sourceName || null, scriptPath: editForm.scriptPath || null }
|
const payload: Record<string, string | null> = { name: editForm.name, targetUrl: editForm.targetUrl, type: editForm.type, sourceName: editForm.sourceName || null, scriptPath: editForm.scriptPath || null }
|
||||||
if (editForm.schedule) payload.schedule = editForm.schedule
|
if (editForm.schedule) payload.schedule = editForm.schedule
|
||||||
const kwArr = editForm.keywords.split(/[,，、\s]+/).filter(Boolean)
|
const catArr: string[] = editForm.categories ? JSON.parse(editForm.categories) : []
|
||||||
payload.keywords = kwArr.length > 0 ? JSON.stringify(kwArr) : ''
|
payload.categories = catArr.length > 0 ? JSON.stringify(catArr) : null
|
||||||
const catArr = editForm.categories.split(/[,，、\s]+/).filter(Boolean)
|
// 新配置不再使用 keywords 字段，但不清理旧数据（resolveKeywords 有 fallback）
|
||||||
payload.categories = catArr.length > 0 ? JSON.stringify(catArr) : ''
|
|
||||||
try {
|
try {
|
||||||
const res = await fetch(`${CRAWLER_API}/${id}`, {
|
const res = await fetch(`${CRAWLER_API}/${id}`, {
|
||||||
method: 'PUT',
|
method: 'PUT',
|
||||||
@ -347,8 +333,7 @@ export default function RuleCenterPage() {
|
|||||||
|
|
||||||
const addCrawler = async () => {
|
const addCrawler = async () => {
|
||||||
if (!addForm.name.trim() || !addForm.targetUrl.trim()) return
|
if (!addForm.name.trim() || !addForm.targetUrl.trim()) return
|
||||||
const kwArr = addForm.keywords.split(/[,，、\s]+/).filter(Boolean)
|
const catArr: string[] = addForm.categories ? JSON.parse(addForm.categories) : []
|
||||||
const catArr = addForm.categories.split(/[,，、\s]+/).filter(Boolean)
|
|
||||||
try {
|
try {
|
||||||
const res = await fetch(CRAWLER_API, {
|
const res = await fetch(CRAWLER_API, {
|
||||||
method: 'POST',
|
method: 'POST',
|
||||||
@ -360,7 +345,7 @@ export default function RuleCenterPage() {
|
|||||||
sourceName: addForm.sourceName || null,
|
sourceName: addForm.sourceName || null,
|
||||||
scriptPath: addForm.scriptPath || null,
|
scriptPath: addForm.scriptPath || null,
|
||||||
schedule: addForm.schedule || null,
|
schedule: addForm.schedule || null,
|
||||||
keywords: kwArr.length > 0 ? JSON.stringify(kwArr) : null,
|
keywords: null,
|
||||||
categories: catArr.length > 0 ? JSON.stringify(catArr) : null,
|
categories: catArr.length > 0 ? JSON.stringify(catArr) : null,
|
||||||
}),
|
}),
|
||||||
})
|
})
|
||||||
@ -368,7 +353,7 @@ export default function RuleCenterPage() {
|
|||||||
const created: CrawlerConfig = await res.json()
|
const created: CrawlerConfig = await res.json()
|
||||||
setCrawlers(prev => [...prev, created])
|
setCrawlers(prev => [...prev, created])
|
||||||
setShowAddModal(false)
|
setShowAddModal(false)
|
||||||
setAddForm({ name: '', targetUrl: '', type: 'bid', sourceName: '', scriptPath: '', schedule: '0 */6 * * *', keywords: '', categories: '' })
|
setAddForm({ name: '', targetUrl: '', type: 'bid', sourceName: '', scriptPath: '', schedule: '0 */6 * * *', keywords: '', categories: '[]' })
|
||||||
} catch { alert('创建失败') }
|
} catch { alert('创建失败') }
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -532,7 +517,7 @@ export default function RuleCenterPage() {
|
|||||||
<div className="p-4 rounded-lg bg-slate-50 border border-border-subtle">
|
<div className="p-4 rounded-lg bg-slate-50 border border-border-subtle">
|
||||||
<div className="text-xs text-on-surface-variant">待重算线索</div>
|
<div className="text-xs text-on-surface-variant">待重算线索</div>
|
||||||
<div className="text-2xl font-bold mt-1">
|
<div className="text-2xl font-bold mt-1">
|
||||||
{keywords.length > 0 ? '1,248' : '1,248'}
|
{'1,248'}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div className="p-4 rounded-lg bg-emerald-50 border border-emerald-200">
|
<div className="p-4 rounded-lg bg-emerald-50 border border-emerald-200">
|
||||||
@ -587,125 +572,139 @@ export default function RuleCenterPage() {
|
|||||||
{/* Tab 2: 关键词库管理 */}
|
{/* Tab 2: 关键词库管理 */}
|
||||||
{activeTab === 'keywords' && (
|
{activeTab === 'keywords' && (
|
||||||
<div className="space-y-4">
|
<div className="space-y-4">
|
||||||
{/* Add Keyword Row */}
|
{/* Header */}
|
||||||
<div className="bg-white rounded-xl border border-border-subtle p-6 shadow-sm">
|
<div className="flex justify-between items-center">
|
||||||
<h3 className="text-base font-semibold mb-4">添加关键词</h3>
|
<p className="text-sm text-on-surface-variant">
|
||||||
<div className="flex items-end gap-3">
|
管理所属品类及其关键词。采集公告时优先按关键词匹配到对应品类，匹配不到则自动 AI 推断。
|
||||||
<div className="flex-1">
|
</p>
|
||||||
<span className="text-[10px] font-bold tracking-wider text-on-surface-variant uppercase block mb-1">关键词</span>
|
<button
|
||||||
<input
|
onClick={() => setShowAddCat(true)}
|
||||||
type="text"
|
className="px-4 py-2 bg-secondary text-white rounded-lg text-sm font-semibold flex items-center gap-2 hover:opacity-90 shadow-sm"
|
||||||
value={newWord}
|
>
|
||||||
onChange={e => setNewWord(e.target.value)}
|
<svg className="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M12 4v16m8-8H4" /></svg>
|
||||||
placeholder="输入关键词"
|
添加品类
|
||||||
className="w-full bg-surface-container-low border border-border-subtle rounded-lg px-3 py-2 text-sm focus:ring-secondary outline-none"
|
</button>
|
||||||
/>
|
|
||||||
</div>
|
|
||||||
<div className="w-36">
|
|
||||||
<span className="text-[10px] font-bold tracking-wider text-on-surface-variant uppercase block mb-1">所属品类</span>
|
|
||||||
<select
|
|
||||||
value={newCategory}
|
|
||||||
onChange={e => setNewCategory(e.target.value)}
|
|
||||||
className="w-full bg-surface-container-low border border-border-subtle rounded-lg px-3 py-2 text-sm focus:ring-secondary outline-none"
|
|
||||||
>
|
|
||||||
{CATEGORIES.map(c => <option key={c}>{c}</option>)}
|
|
||||||
</select>
|
|
||||||
</div>
|
|
||||||
<div className="w-24">
|
|
||||||
<span className="text-[10px] font-bold tracking-wider text-on-surface-variant uppercase block mb-1">权重 (1-10)</span>
|
|
||||||
<input
|
|
||||||
type="number"
|
|
||||||
min={1}
|
|
||||||
max={10}
|
|
||||||
value={newWeight}
|
|
||||||
onChange={e => setNewWeight(Math.max(1, Math.min(10, parseInt(e.target.value) || 1)))}
|
|
||||||
className="w-full bg-surface-container-low border border-border-subtle rounded-lg px-3 py-2 text-sm focus:ring-secondary outline-none"
|
|
||||||
/>
|
|
||||||
</div>
|
|
||||||
<button
|
|
||||||
onClick={handleAddKeyword}
|
|
||||||
disabled={!newWord.trim()}
|
|
||||||
className="px-4 py-2 bg-secondary text-white rounded-lg text-sm font-semibold hover:opacity-90 transition-opacity disabled:opacity-40 shadow-sm"
|
|
||||||
>
|
|
||||||
+ 添加
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Keyword Table */}
|
{/* Add Category Modal */}
|
||||||
<div className="bg-white rounded-xl border border-border-subtle shadow-sm overflow-hidden">
|
{showAddCat && (
|
||||||
<div className="p-4 border-b border-border-subtle flex justify-between items-center">
|
<div className="fixed inset-0 z-50 flex items-start justify-center pt-20 bg-black/30" onClick={() => setShowAddCat(false)}>
|
||||||
<h3 className="text-base font-semibold">关键词列表</h3>
|
<div className="bg-white rounded-xl shadow-2xl w-[500px] transform" onClick={e => e.stopPropagation()}>
|
||||||
<div className="flex gap-2">
|
<div className="p-5 border-b border-border-subtle flex justify-between items-center">
|
||||||
<button
|
<h3 className="text-base font-semibold">添加所属品类</h3>
|
||||||
onClick={() => setShowBatchModal(true)}
|
<button onClick={() => setShowAddCat(false)} className="p-2 hover:bg-slate-100 rounded-full">
|
||||||
className="px-3 py-1.5 bg-white border border-border-subtle text-on-surface-variant rounded-lg text-xs flex items-center gap-1.5 hover:bg-slate-50 transition-colors"
|
<svg className="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M6 18L18 6M6 6l12 12" /></svg>
|
||||||
>
|
</button>
|
||||||
<svg className="w-3.5 h-3.5" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M4 16v1a3 3 0 003 3h10a3 3 0 003-3v-1m-4-8l-4-4m0 0L8 8m4-4v12" /></svg>
|
</div>
|
||||||
批量导入
|
<div className="p-5 space-y-4">
|
||||||
</button>
|
<div>
|
||||||
<button
|
<span className="text-[10px] font-bold tracking-wider text-on-surface-variant uppercase block mb-1">品类名称</span>
|
||||||
onClick={() => setShowClearModal(true)}
|
<input
|
||||||
className="px-3 py-1.5 bg-white border border-border-subtle text-rose-600 rounded-lg text-xs flex items-center gap-1.5 hover:bg-rose-50 transition-colors"
|
value={addCatName}
|
||||||
>
|
onChange={e => setAddCatName(e.target.value)}
|
||||||
<svg className="w-3.5 h-3.5" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M19 7l-.867 12.142A2 2 0 0116.138 21H7.862a2 2 0 01-1.995-1.858L5 7m5 4v6m4-6v6m1-10V4a1 1 0 00-1-1h-4a1 1 0 00-1 1v3M4 7h16" /></svg>
|
placeholder="如：彩板房"
|
||||||
一键清空
|
className="w-full border border-border-subtle rounded-lg px-3 py-2 text-sm outline-none focus:ring-2 focus:ring-secondary/20"
|
||||||
</button>
|
/>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<span className="text-[10px] font-bold tracking-wider text-on-surface-variant uppercase block mb-1">关键词列表（逗号分隔）</span>
|
||||||
|
<textarea
|
||||||
|
value={addCatKeywords}
|
||||||
|
onChange={e => setAddCatKeywords(e.target.value)}
|
||||||
|
placeholder="彩板房, 彩钢板, 夹芯板"
|
||||||
|
rows={3}
|
||||||
|
className="w-full border border-border-subtle rounded-lg px-3 py-2 text-sm outline-none focus:ring-2 focus:ring-secondary/20 resize-none"
|
||||||
|
/>
|
||||||
|
<p className="text-[10px] text-on-surface-variant mt-1">公告标题包含任一关键词即自动归入该品类</p>
|
||||||
|
</div>
|
||||||
|
<div className="flex gap-3 pt-2">
|
||||||
|
<button onClick={handleAddCategory} disabled={!addCatName.trim()} className="flex-1 py-2 bg-secondary text-white rounded-lg text-sm font-semibold hover:opacity-90 disabled:opacity-50">确认添加</button>
|
||||||
|
<button onClick={() => setShowAddCat(false)} className="px-4 py-2 bg-white border border-border-subtle rounded-lg text-sm hover:bg-slate-50">取消</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<table className="w-full text-left">
|
)}
|
||||||
<thead className="bg-slate-50 border-b border-border-subtle">
|
|
||||||
<tr>
|
{/* Category Table */}
|
||||||
<th className="p-4 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">关键词</th>
|
<div className="bg-white rounded-xl border border-border-subtle shadow-sm overflow-hidden">
|
||||||
<th className="p-4 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">所属品类</th>
|
{catLoading ? (
|
||||||
<th className="p-4 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase text-right">权重</th>
|
<div className="p-8 text-center text-sm text-on-surface-variant">加载中...</div>
|
||||||
<th className="p-4 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase text-right">匹配次数</th>
|
) : categories.length === 0 ? (
|
||||||
<th className="p-4 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">启用状态</th>
|
<div className="p-8 text-center text-sm text-on-surface-variant">暂无品类，请点击右上角「添加品类」</div>
|
||||||
<th className="p-4 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">操作</th>
|
) : (
|
||||||
</tr>
|
<table className="w-full text-left">
|
||||||
</thead>
|
<thead className="bg-slate-50 border-b border-border-subtle">
|
||||||
<tbody className="divide-y divide-border-subtle">
|
|
||||||
{keywords.length === 0 ? (
|
|
||||||
<tr>
|
<tr>
|
||||||
<td colSpan={6} className="p-8 text-center text-sm text-on-surface-variant">暂无关键词，请添加</td>
|
<th className="p-4 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">品类名称</th>
|
||||||
|
<th className="p-4 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">关键词列表</th>
|
||||||
|
<th className="p-4 w-20 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">来源</th>
|
||||||
|
<th className="p-4 w-40 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">操作</th>
|
||||||
</tr>
|
</tr>
|
||||||
) : (
|
</thead>
|
||||||
keywords.map(kw => (
|
<tbody className="divide-y divide-border-subtle">
|
||||||
<tr key={kw.id} className="hover:bg-slate-50 transition-colors">
|
{categories.map(cat => {
|
||||||
<td className="p-4 text-sm font-medium">{kw.word}</td>
|
const kwList: string[] = (() => { try { return JSON.parse(cat.keywords) } catch { return [] } })()
|
||||||
<td className="p-4 text-sm text-on-surface-variant">
|
return editCatId === cat.id ? (
|
||||||
<span className="px-2 py-0.5 bg-slate-100 text-on-surface-variant text-[10px] font-medium rounded">{kw.category}</span>
|
<tr key={cat.id} className="bg-blue-50/30">
|
||||||
</td>
|
<td className="p-3">
|
||||||
<td className="p-4 text-sm text-right font-semibold">{kw.weight}</td>
|
<input value={editCatName} onChange={e => setEditCatName(e.target.value)} className="w-full border border-border-subtle rounded px-2 py-1.5 text-sm outline-none focus:ring-2 focus:ring-secondary/20" />
|
||||||
<td className="p-4 text-sm text-right text-on-surface-variant">{kw.matchCount}</td>
|
</td>
|
||||||
<td className="p-4">
|
<td className="p-3">
|
||||||
<button
|
<input value={editCatKeywords} onChange={e => setEditCatKeywords(e.target.value)} placeholder="逗号分隔" className="w-full border border-border-subtle rounded px-2 py-1.5 text-sm outline-none focus:ring-2 focus:ring-secondary/20" />
|
||||||
onClick={() => toggleKeyword(kw.id)}
|
</td>
|
||||||
className={`relative w-9 h-5 rounded-full transition-colors ${kw.enabled ? 'bg-secondary' : 'bg-slate-300'}`}
|
<td className="p-3">
|
||||||
>
|
<span className={`px-2 py-0.5 text-[10px] font-medium rounded-full ${cat.isAIGenerated ? 'bg-amber-50 text-amber-700' : 'bg-blue-50 text-blue-700'}`}>
|
||||||
<span className={`absolute top-0.5 left-0.5 w-4 h-4 bg-white rounded-full transition-transform shadow-sm ${kw.enabled ? 'translate-x-4' : 'translate-x-0'}`} />
|
{cat.isAIGenerated ? 'AI' : '手动'}
|
||||||
</button>
|
</span>
|
||||||
</td>
|
</td>
|
||||||
<td className="p-4">
|
<td className="p-3">
|
||||||
<div className="flex items-center gap-2">
|
<div className="flex gap-2">
|
||||||
<button onClick={() => editKeyword(kw.id)} className="text-xs text-blue-600 hover:underline">编辑</button>
|
<button onClick={handleUpdateCategory} className="px-3 py-1 text-[11px] bg-emerald-50 text-emerald-700 border border-emerald-200 rounded-md font-semibold hover:bg-emerald-100">保存</button>
|
||||||
<span className="text-slate-300">|</span>
|
<button onClick={() => setEditCatId(null)} className="px-3 py-1 text-[11px] bg-white border border-border-subtle rounded-md text-slate-600 hover:bg-slate-50">取消</button>
|
||||||
<button onClick={() => deleteKeyword(kw.id)} className="text-xs text-rose-500 hover:underline">删除</button>
|
</div>
|
||||||
</div>
|
</td>
|
||||||
</td>
|
</tr>
|
||||||
</tr>
|
) : (
|
||||||
))
|
<tr key={cat.id} className="hover:bg-slate-50 transition-colors">
|
||||||
)}
|
<td className="p-4 text-sm font-semibold">{cat.name}</td>
|
||||||
</tbody>
|
<td className="p-4">
|
||||||
</table>
|
<div className="flex flex-wrap gap-1 max-w-[500px]">
|
||||||
<div className="bg-slate-50 px-4 py-2 border-t border-border-subtle flex justify-between items-center">
|
{kwList.length === 0 ? (
|
||||||
<span className="text-xs text-on-surface-variant">共 {keywords.length} 个关键词</span>
|
<span className="text-xs text-slate-400">无关键词</span>
|
||||||
{keywords.length > 0 && (
|
) : (
|
||||||
|
kwList.map((kw: string, i: number) => (
|
||||||
|
<span key={i} className="px-1.5 py-0.5 bg-slate-100 text-slate-700 text-[10px] rounded whitespace-nowrap">{kw}</span>
|
||||||
|
))
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</td>
|
||||||
|
<td className="p-4">
|
||||||
|
<span className={`px-2 py-0.5 text-[10px] font-medium rounded-full ${cat.isAIGenerated ? 'bg-amber-50 text-amber-700' : 'bg-blue-50 text-blue-700'}`}>
|
||||||
|
{cat.isAIGenerated ? 'AI' : '手动'}
|
||||||
|
</span>
|
||||||
|
</td>
|
||||||
|
<td className="p-4">
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<button onClick={() => startEditCat(cat)} className="text-xs text-blue-600 hover:underline">编辑</button>
|
||||||
|
<span className="text-slate-300">|</span>
|
||||||
|
<button onClick={() => handleDeleteCategory(cat.id, cat.name)} className="text-xs text-rose-500 hover:underline">删除</button>
|
||||||
|
</div>
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
)
|
||||||
|
})}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
)}
|
||||||
|
{!catLoading && categories.length > 0 && (
|
||||||
|
<div className="bg-slate-50 px-4 py-2 border-t border-border-subtle flex justify-between items-center">
|
||||||
|
<span className="text-xs text-on-surface-variant">共 {categories.length} 个品类</span>
|
||||||
<span className="text-[10px] text-on-surface-variant">
|
<span className="text-[10px] text-on-surface-variant">
|
||||||
启用 {keywords.filter(k => k.enabled).length} 个 · 停用 {keywords.filter(k => !k.enabled).length} 个
|
手动 {categories.filter(c => !c.isAIGenerated).length} 个 · AI 创建 {categories.filter(c => c.isAIGenerated).length} 个
|
||||||
</span>
|
</span>
|
||||||
)}
|
</div>
|
||||||
</div>
|
)}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
@ -738,7 +737,7 @@ export default function RuleCenterPage() {
|
|||||||
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">采集器名称</th>
|
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">采集器名称</th>
|
||||||
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">目标网址</th>
|
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">目标网址</th>
|
||||||
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">抓取定时</th>
|
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">抓取定时</th>
|
||||||
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">过滤关键词</th>
|
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">关联品类</th>
|
||||||
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">状态</th>
|
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">状态</th>
|
||||||
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">运行统计</th>
|
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">运行统计</th>
|
||||||
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">操作</th>
|
<th className="p-3 text-[10px] font-bold tracking-wider text-on-surface-variant uppercase">操作</th>
|
||||||
@ -761,7 +760,28 @@ export default function RuleCenterPage() {
|
|||||||
</select>
|
</select>
|
||||||
</td>
|
</td>
|
||||||
<td className="p-3">
|
<td className="p-3">
|
||||||
<input value={editForm.keywords} onChange={e => setEditForm(f => ({ ...f, keywords: e.target.value }))} placeholder="用逗号分隔" className="w-full border border-border-subtle rounded px-2 py-1 text-sm outline-none focus:ring-2 focus:ring-secondary/20" />
|
<div className="max-h-[150px] overflow-y-auto space-y-1 text-xs">
|
||||||
|
{categories.length === 0 ? (
|
||||||
|
<span className="text-slate-400">请先在「关键词库管理」添加品类</span>
|
||||||
|
) : categories.map(cat => {
|
||||||
|
const current: string[] = editForm.categories ? JSON.parse(editForm.categories) : []
|
||||||
|
const selected = current.includes(cat.name)
|
||||||
|
return (
|
||||||
|
<label key={cat.id} className="flex items-center gap-1.5 cursor-pointer hover:bg-slate-50 rounded p-0.5">
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
checked={selected}
|
||||||
|
onChange={() => {
|
||||||
|
const next = selected ? current.filter(n => n !== cat.name) : [...current, cat.name]
|
||||||
|
setEditForm(f => ({ ...f, categories: JSON.stringify(next) }))
|
||||||
|
}}
|
||||||
|
className="accent-secondary"
|
||||||
|
/>
|
||||||
|
{cat.name}
|
||||||
|
</label>
|
||||||
|
)
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
</td>
|
</td>
|
||||||
<td className="p-3">
|
<td className="p-3">
|
||||||
<span className={`px-2 py-0.5 text-[10px] font-medium rounded-full border ${getStatusStyle(c.status)}`}>
|
<span className={`px-2 py-0.5 text-[10px] font-medium rounded-full border ${getStatusStyle(c.status)}`}>
|
||||||
@ -784,15 +804,22 @@ export default function RuleCenterPage() {
|
|||||||
<td className="p-3 text-xs text-on-surface-variant max-w-[200px] truncate">{c.targetUrl}</td>
|
<td className="p-3 text-xs text-on-surface-variant max-w-[200px] truncate">{c.targetUrl}</td>
|
||||||
<td className="p-3 text-xs text-on-surface-variant">{getCronLabel(c.schedule)}</td>
|
<td className="p-3 text-xs text-on-surface-variant">{getCronLabel(c.schedule)}</td>
|
||||||
<td className="p-3">
|
<td className="p-3">
|
||||||
{c.keywords ? (
|
{c.categories ? (
|
||||||
<div className="flex flex-wrap gap-1">
|
<div className="flex flex-wrap gap-1">
|
||||||
{(JSON.parse(c.keywords) as string[]).slice(0, 3).map((kw, i) => (
|
{(JSON.parse(c.categories) as string[]).slice(0, 3).map((cat, i) => (
|
||||||
<span key={i} className="px-1.5 py-0.5 bg-blue-50 text-blue-600 text-[10px] rounded">{kw}</span>
|
<span key={i} className="px-1.5 py-0.5 bg-purple-50 text-purple-600 text-[10px] rounded">{cat}</span>
|
||||||
))}
|
))}
|
||||||
{(JSON.parse(c.keywords) as string[]).length > 3 && (
|
{(JSON.parse(c.categories) as string[]).length > 3 && (
|
||||||
<span className="text-[10px] text-slate-400">+{JSON.parse(c.keywords).length - 3}</span>
|
<span className="text-[10px] text-slate-400">+{JSON.parse(c.categories).length - 3}</span>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
) : c.keywords ? (
|
||||||
|
<div className="flex flex-wrap gap-1 items-center">
|
||||||
|
{(JSON.parse(c.keywords) as string[]).slice(0, 3).map((kw, i) => (
|
||||||
|
<span key={i} className="px-1.5 py-0.5 bg-slate-100 text-slate-500 text-[10px] rounded">{kw}</span>
|
||||||
|
))}
|
||||||
|
<span className="text-[10px] text-slate-400">(旧)</span>
|
||||||
|
</div>
|
||||||
) : <span className="text-xs text-slate-400">全部</span>}
|
) : <span className="text-xs text-slate-400">全部</span>}
|
||||||
</td>
|
</td>
|
||||||
<td className="p-3">
|
<td className="p-3">
|
||||||
@ -1049,34 +1076,6 @@ export default function RuleCenterPage() {
|
|||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{/* Batch Import Modal */}
|
|
||||||
{showBatchModal && (
|
|
||||||
<div className="fixed inset-0 z-50 bg-black/30 flex items-start justify-center pt-20" onClick={() => setShowBatchModal(false)}>
|
|
||||||
<div className="bg-white rounded-xl shadow-2xl w-[480px] transform" onClick={e => e.stopPropagation()}>
|
|
||||||
<div className="p-5 border-b border-border-subtle flex justify-between items-center">
|
|
||||||
<h3 className="text-base font-semibold">批量导入关键词</h3>
|
|
||||||
<button onClick={() => setShowBatchModal(false)} className="p-2 hover:bg-slate-100 rounded-full">
|
|
||||||
<svg className="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M6 18L18 6M6 6l12 12" /></svg>
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
<div className="p-5 space-y-4">
|
|
||||||
<p className="text-sm text-on-surface-variant">每行一个关键词，将自动归入默认品类（可后续编辑）</p>
|
|
||||||
<textarea
|
|
||||||
value={batchText}
|
|
||||||
onChange={e => setBatchText(e.target.value)}
|
|
||||||
placeholder={"防火门\n配电箱\n电线电缆\n阀门\n钢材"}
|
|
||||||
rows={8}
|
|
||||||
className="w-full bg-surface-container-low border border-border-subtle rounded-lg px-3 py-2 text-sm focus:ring-secondary outline-none resize-none"
|
|
||||||
/>
|
|
||||||
<div className="flex gap-3 pt-2">
|
|
||||||
<button onClick={handleBatchImport} disabled={!batchText.trim()} className="flex-1 py-2 bg-secondary text-white rounded-lg text-sm font-semibold hover:opacity-90 disabled:opacity-50">确认导入</button>
|
|
||||||
<button onClick={() => setShowBatchModal(false)} className="px-4 py-2 bg-white border border-border-subtle rounded-lg text-sm hover:bg-slate-50">取消</button>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
)}
|
|
||||||
|
|
||||||
{/* Add Crawler Modal */}
|
{/* Add Crawler Modal */}
|
||||||
{showAddModal && (
|
{showAddModal && (
|
||||||
<div className="fixed inset-0 z-50 bg-black/30 flex items-start justify-center pt-20" onClick={() => setShowAddModal(false)}>
|
<div className="fixed inset-0 z-50 bg-black/30 flex items-start justify-center pt-20" onClick={() => setShowAddModal(false)}>
|
||||||
@ -1112,12 +1111,34 @@ export default function RuleCenterPage() {
|
|||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
<div className="col-span-2">
|
<div className="col-span-2">
|
||||||
<span className="text-[10px] font-bold tracking-wider text-on-surface-variant uppercase block mb-1">过滤关键词（逗号分隔，留空表示抓取全部）</span>
|
<span className="text-[10px] font-bold tracking-wider text-on-surface-variant uppercase block mb-2">关联品类（选中的品类关键词用于过滤公告）</span>
|
||||||
<input value={addForm.keywords} onChange={e => setAddForm(f => ({ ...f, keywords: e.target.value }))} placeholder="电缆, 防火门, 配电箱" className="w-full border border-border-subtle rounded-lg px-3 py-2 text-sm outline-none focus:ring-2 focus:ring-secondary/20" />
|
<div className="max-h-[200px] overflow-y-auto border border-border-subtle rounded-lg p-3 space-y-2">
|
||||||
</div>
|
{categories.length === 0 ? (
|
||||||
<div className="col-span-2">
|
<p className="text-xs text-on-surface-variant">暂无品类，请先在「关键词库管理」中添加品类</p>
|
||||||
<span className="text-[10px] font-bold tracking-wider text-on-surface-variant uppercase block mb-1">所属品类（逗号分隔，留空表示全部品类）</span>
|
) : categories.map(cat => {
|
||||||
<input value={addForm.categories} onChange={e => setAddForm(f => ({ ...f, categories: e.target.value }))} placeholder="工程建材, 电力设备, 钢材" className="w-full border border-border-subtle rounded-lg px-3 py-2 text-sm outline-none focus:ring-2 focus:ring-secondary/20" />
|
const current: string[] = addForm.categories ? JSON.parse(addForm.categories) : []
|
||||||
|
const selected = current.includes(cat.name)
|
||||||
|
return (
|
||||||
|
<label key={cat.id} className="flex items-center gap-2 cursor-pointer hover:bg-slate-50 p-1 rounded">
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
checked={selected}
|
||||||
|
onChange={() => {
|
||||||
|
const next = selected ? current.filter(n => n !== cat.name) : [...current, cat.name]
|
||||||
|
setAddForm(f => ({ ...f, categories: JSON.stringify(next) }))
|
||||||
|
}}
|
||||||
|
className="accent-secondary"
|
||||||
|
/>
|
||||||
|
<span className="text-sm">{cat.name}</span>
|
||||||
|
{(() => { try { const k = JSON.parse(cat.keywords); return Array.isArray(k) ? k : [] } catch { return [] } })().length > 0 && (
|
||||||
|
<span className="text-[10px] text-on-surface-variant">
|
||||||
|
({(() => { try { const k = JSON.parse(cat.keywords); return Array.isArray(k) ? k.slice(0, 3).join(', ') : '' } catch { return '' } })()})
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
</label>
|
||||||
|
)
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div className="flex gap-3 pt-2">
|
<div className="flex gap-3 pt-2">
|
||||||
@ -1177,29 +1198,6 @@ export default function RuleCenterPage() {
|
|||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
{/* Clear All Modal */}
|
|
||||||
{showClearModal && (
|
|
||||||
<div className="fixed inset-0 z-50 bg-black/30 flex items-start justify-center pt-20" onClick={() => setShowClearModal(false)}>
|
|
||||||
<div className="bg-white rounded-xl shadow-2xl w-[400px] transform" onClick={e => e.stopPropagation()}>
|
|
||||||
<div className="p-5 border-b border-border-subtle flex justify-between items-center">
|
|
||||||
<h3 className="text-base font-semibold text-rose-600">确认清空</h3>
|
|
||||||
<button onClick={() => setShowClearModal(false)} className="p-2 hover:bg-slate-100 rounded-full">
|
|
||||||
<svg className="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M6 18L18 6M6 6l12 12" /></svg>
|
|
||||||
</button>
|
|
||||||
</div>
|
|
||||||
<div className="p-5 space-y-4">
|
|
||||||
<div className="flex items-start gap-3 bg-rose-50 p-3 rounded-lg">
|
|
||||||
<svg className="w-5 h-5 text-rose-600 mt-0.5 shrink-0" fill="currentColor" viewBox="0 0 24 24"><path d="M12 22C6.477 22 2 17.523 2 12S6.477 2 12 2s10 4.477 10 10-4.477 10-10 10zm-1-7v2h2v-2h-2zm0-8v6h2V7h-2z" /></svg>
|
|
||||||
<p className="text-sm">确定要清空所有关键词吗？此操作不可撤销，共 <b>{keywords.length}</b> 个关键词将被删除。</p>
|
|
||||||
</div>
|
|
||||||
<div className="flex gap-3 pt-2">
|
|
||||||
<button onClick={handleClearAll} className="flex-1 py-2 bg-rose-600 text-white rounded-lg text-sm font-semibold hover:bg-rose-700">确认清空</button>
|
|
||||||
<button onClick={() => setShowClearModal(false)} className="px-4 py-2 bg-white border border-border-subtle rounded-lg text-sm hover:bg-slate-50">取消</button>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
)}
|
|
||||||
</div>
|
</div>
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|||||||
36
scripts/cleanup_garbage_categories.sql
Normal file
36
scripts/cleanup_garbage_categories.sql
Normal file
@ -0,0 +1,36 @@
|
|||||||
|
-- 清理垃圾关键词品类脚本
|
||||||
|
-- 使用：ssh到生产机后执行：sqlite3 /data/pingtai/dev.db < cleanup_garbage_categories.sql
|
||||||
|
|
||||||
|
-- 第一步：查看将要删除的品类及其关联线索数
|
||||||
|
.print '=== 将要清理的垃圾品类 ==='
|
||||||
|
SELECT pc.name, pc.keywords, COUNT(l.id) AS lead_count
|
||||||
|
FROM ProductCategory pc
|
||||||
|
LEFT JOIN Lead l ON l.category = pc.name
|
||||||
|
WHERE (keywords LIKE '%中国电建%' OR keywords LIKE '%新疆%')
|
||||||
|
AND length(keywords) < 100
|
||||||
|
GROUP BY pc.id
|
||||||
|
ORDER BY lead_count DESC;
|
||||||
|
|
||||||
|
-- 第二步：清除被误分类线索的 category 字段
|
||||||
|
.print ''
|
||||||
|
.print '=== 清除线索中的错误品类 ==='
|
||||||
|
UPDATE Lead SET category = ''
|
||||||
|
WHERE category IN (
|
||||||
|
SELECT name FROM ProductCategory
|
||||||
|
WHERE (keywords LIKE '%中国电建%' OR keywords LIKE '%新疆%')
|
||||||
|
AND length(keywords) < 100
|
||||||
|
);
|
||||||
|
SELECT changes() AS '已清除线索数';
|
||||||
|
|
||||||
|
-- 第三步：删除垃圾品类记录
|
||||||
|
.print ''
|
||||||
|
.print '=== 删除垃圾品类 ==='
|
||||||
|
DELETE FROM ProductCategory
|
||||||
|
WHERE (keywords LIKE '%中国电建%' OR keywords LIKE '%新疆%')
|
||||||
|
AND length(keywords) < 100;
|
||||||
|
SELECT changes() AS '已删除品类数';
|
||||||
|
|
||||||
|
-- 第四步：验证结果
|
||||||
|
.print ''
|
||||||
|
.print '=== 验证：剩余品类 ==='
|
||||||
|
SELECT name, keywords, isAIGenerated FROM ProductCategory ORDER BY name;
|
||||||
33
scripts/fix_corrupted_titles.sql
Normal file
33
scripts/fix_corrupted_titles.sql
Normal file
@ -0,0 +1,33 @@
|
|||||||
|
-- 修复乱码标题：将 Lead 表中含 U+FFFD 的标题替换为从 PowerChina API 获取的干净版本
|
||||||
|
|
||||||
|
-- 先查看当前乱码记录
|
||||||
|
SELECT '当前乱码记录:' as info;
|
||||||
|
SELECT id, noticeId, substr(title, 1, 70) as title_trunc FROM Lead WHERE title LIKE '%' || char(65533) || '%';
|
||||||
|
|
||||||
|
-- 批量更新
|
||||||
|
UPDATE Lead SET title = '中国电建重庆工程有限公司兰州新区大数据产业园（二期）项目110kV变配电工程电缆安装施工项目工器具采购项目公开谈判采购公告'
|
||||||
|
WHERE id = (SELECT id FROM Lead WHERE title LIKE '%兰州新区大数据产业园%' AND title LIKE '%' || char(65533) || '%' LIMIT 1);
|
||||||
|
|
||||||
|
UPDATE Lead SET title = '中国电建山东电建一公司海阳核电核能供热银粉漆、醇酸磁漆等采购项目公开竞价采购公告'
|
||||||
|
WHERE id = (SELECT id FROM Lead WHERE title LIKE '%海阳核电核能供热%' AND title LIKE '%' || char(65533) || '%' LIMIT 1);
|
||||||
|
|
||||||
|
UPDATE Lead SET title = '中国电建上海电建公司福建电建上海新能源申能光明长江养殖场 55.8兆瓦渔光互补光伏发电项目EPC总承包箱变采购项目标前公示'
|
||||||
|
WHERE id = (SELECT id FROM Lead WHERE title LIKE '%上海新能源申能%' AND title LIKE '%' || char(65533) || '%' LIMIT 1);
|
||||||
|
|
||||||
|
UPDATE Lead SET title = '中国电建山东电建一公司北方公司黑龙江哈尔滨哈一热项目钢骨架复合管采购项目公开竞价采购公告'
|
||||||
|
WHERE id = (SELECT id FROM Lead WHERE title LIKE '%黑龙江哈尔滨%' AND title LIKE '%' || char(65533) || '%' LIMIT 1);
|
||||||
|
|
||||||
|
UPDATE Lead SET title = '中国电建水电七局三峡枢纽新通道山体排水洞及施工支洞工程柴油采购项目公开询比采购公告'
|
||||||
|
WHERE id = (SELECT id FROM Lead WHERE title LIKE '%柴油%' AND title LIKE '%' || char(65533) || '%' LIMIT 1);
|
||||||
|
|
||||||
|
UPDATE Lead SET title = '中国电建山东电建一公司西北公司新疆喀什煤电项目安全省力型盖板采购项目公开竞价采购公告'
|
||||||
|
WHERE id = (SELECT id FROM Lead WHERE title LIKE '%新疆喀什%' AND title LIKE '%' || char(65533) || '%' LIMIT 1);
|
||||||
|
|
||||||
|
UPDATE Lead SET title = '中国电建重庆工程公司重庆公司重庆电厂2×660MW机组增设启动锅炉改造工程支吊架采购项目公开谈判采购公告'
|
||||||
|
WHERE id = (SELECT id FROM Lead WHERE title LIKE '%重庆电厂%' AND title LIKE '%' || char(65533) || '%' LIMIT 1);
|
||||||
|
|
||||||
|
-- 验证修复结果
|
||||||
|
SELECT '修复后:' as info;
|
||||||
|
SELECT id, noticeId, substr(title, 1, 70) as title_trunc FROM Lead WHERE title LIKE '%' || char(65533) || '%';
|
||||||
|
SELECT '剩余乱码记录数:' as info;
|
||||||
|
SELECT COUNT(*) FROM Lead WHERE title LIKE '%' || char(65533) || '%';
|
||||||
153
scripts/reclassify_leads.js
Normal file
153
scripts/reclassify_leads.js
Normal file
@ -0,0 +1,153 @@
|
|||||||
|
/**
|
||||||
|
* 重新分类脚本：对 category 为空的线索重新执行分类
|
||||||
|
* 用法：node scripts/reclassify_leads.js
|
||||||
|
*
|
||||||
|
* 需要在项目根目录运行（使用项目内的 Prisma）
|
||||||
|
*/
|
||||||
|
const { PrismaClient } = require('@prisma/client')
|
||||||
|
|
||||||
|
// 直接复用 inference 逻辑的内联版本
|
||||||
|
const MINIMAX_ENDPOINT = 'https://api.minimaxi.com/v1/chat/completions'
|
||||||
|
const MINIMAX_MODEL = process.env.MINIMAX_MODEL || 'MiniMax-M2.7'
|
||||||
|
const API_KEY = process.env.MINIMAX_API_KEY
|
||||||
|
|
||||||
|
function isGarbageKeyword(kw) {
|
||||||
|
if (/(?:公司|有限|集团|工程局|项目部)$/.test(kw)) return true
|
||||||
|
if (/^(?:中国电建|新疆)/.test(kw)) return true
|
||||||
|
const genericTerms = ['公开竞价', '公开招标', '竞争性谈判', '询价采购', '邀请招标']
|
||||||
|
if (genericTerms.includes(kw)) return true
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
function isValidCategoryName(raw) {
|
||||||
|
if (!raw) return false
|
||||||
|
const s = raw.trim()
|
||||||
|
if (s.length === 0 || s.length > 6) return false
|
||||||
|
if (/[*\->→]/.test(s)) return false
|
||||||
|
if (/[，。：；！？、]/.test(s)) return false
|
||||||
|
if (/[A-Za-z]{2,}/.test(s)) return false
|
||||||
|
if (!/^[\u4e00-\u9fa5]+$/.test(s)) return false
|
||||||
|
if (/^(让我|用户|我需要|好的|现在|首先|分析|根据|对于|这条|这个|考虑|我们|这里|注意|提示|重要|关键|总结|综上|该|其|它|他们|这些|那些|由于|因为|所以|因此|需要|可能|应该|可以|建议|希望|要求|必须)/.test(s)) return false
|
||||||
|
if (/采购|项目|公司|工程|设备|材料|公告|招标|标段|框架|年度/.test(s)) return false
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
function extractCategoryFromLine(line) {
|
||||||
|
// 剥离数字前缀
|
||||||
|
let s = line.replace(/^[\d]+[\.、:：]\s*/, '').trim()
|
||||||
|
if (!s) return ''
|
||||||
|
|
||||||
|
// 取 → 后面的内容（最可能是品类名）
|
||||||
|
const arrowIdx = s.lastIndexOf('→')
|
||||||
|
if (arrowIdx >= 0) {
|
||||||
|
let afterArrow = s.slice(arrowIdx + 1).trim()
|
||||||
|
// 提取连续中文字
|
||||||
|
const m = afterArrow.match(/([\u4e00-\u9fa5]{2,6})/)
|
||||||
|
if (m && isValidCategoryName(m[1])) return m[1]
|
||||||
|
}
|
||||||
|
|
||||||
|
// 从整句中提取可能的品类名
|
||||||
|
const parts = s.split(/\s*(?:[-–—]|→)\s*/).map(p => p.trim()).filter(p => p)
|
||||||
|
// 从最后一部分往前找
|
||||||
|
for (let i = parts.length - 1; i >= 0; i--) {
|
||||||
|
const m = parts[i].match(/([\u4e00-\u9fa5]{2,6})/g)
|
||||||
|
if (m) {
|
||||||
|
for (let j = m.length - 1; j >= 0; j--) {
|
||||||
|
if (isValidCategoryName(m[j])) return m[j]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// 从整句提取
|
||||||
|
const m = s.match(/([\u4e00-\u9fa5]{2,6})/g)
|
||||||
|
if (m) {
|
||||||
|
for (let j = m.length - 1; j >= 0; j--) {
|
||||||
|
if (isValidCategoryName(m[j])) return m[j]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ''
|
||||||
|
}
|
||||||
|
|
||||||
|
// 简版: 只对 batch 结果做后处理
|
||||||
|
function parseBatchResult(content) {
|
||||||
|
const results = []
|
||||||
|
// 去掉 think 块
|
||||||
|
let clean = content.replace(/<think>[\s\S]*?<\/think>/g, '').trim()
|
||||||
|
const lines = clean.split('\n').map(l => l.trim()).filter(l => l)
|
||||||
|
for (const line of lines) {
|
||||||
|
results.push(extractCategoryFromLine(line))
|
||||||
|
}
|
||||||
|
return results
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const prisma = new PrismaClient()
|
||||||
|
|
||||||
|
try {
|
||||||
|
// 1. 找出所有空分类的线索
|
||||||
|
const leads = await prisma.lead.findMany({
|
||||||
|
where: { OR: [{ category: null }, { category: '' }] },
|
||||||
|
select: { id: true, title: true, summary: true },
|
||||||
|
})
|
||||||
|
console.log(`找到 ${leads.length} 条待分类线索`)
|
||||||
|
|
||||||
|
if (leads.length === 0) {
|
||||||
|
console.log('无需分类')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2. 获取已有品类
|
||||||
|
const categories = await prisma.productCategory.findMany()
|
||||||
|
const catNames = categories.map(c => c.name).join('、')
|
||||||
|
|
||||||
|
// 3. 依次尝试关键词匹配，未命中则调 AI
|
||||||
|
const needsAI = []
|
||||||
|
const result = new Map()
|
||||||
|
|
||||||
|
for (const lead of leads) {
|
||||||
|
// 关键词匹配（跳过垃圾关键词）
|
||||||
|
let matched = false
|
||||||
|
for (const cat of categories) {
|
||||||
|
const kws = JSON.parse(cat.keywords)
|
||||||
|
for (const kw of kws) {
|
||||||
|
if (isGarbageKeyword(kw)) continue
|
||||||
|
if (lead.title.includes(kw)) {
|
||||||
|
result.set(lead.id, cat.name)
|
||||||
|
matched = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (matched) break
|
||||||
|
}
|
||||||
|
if (!matched) {
|
||||||
|
needsAI.push(lead)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`关键词命中: ${result.size} 条, 需 AI 推断: ${needsAI.length} 条`)
|
||||||
|
|
||||||
|
// 4. 未命中的全部归为"其他"
|
||||||
|
console.log(`关键词未命中的 ${needsAI.length} 条统一归为「其他」品类`)
|
||||||
|
for (const lead of needsAI) {
|
||||||
|
result.set(lead.id, '其他')
|
||||||
|
}
|
||||||
|
|
||||||
|
// 5. 更新数据库
|
||||||
|
let updated = 0
|
||||||
|
for (const [id, category] of result) {
|
||||||
|
if (category) {
|
||||||
|
await prisma.lead.update({
|
||||||
|
where: { id },
|
||||||
|
data: { category },
|
||||||
|
})
|
||||||
|
updated++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`\n完成! 更新了 ${updated}/${leads.length} 条线索的分类`)
|
||||||
|
console.log(`未分类: ${leads.length - updated} 条（保持为空）`)
|
||||||
|
} finally {
|
||||||
|
await prisma.$disconnect()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
main().catch(console.error)
|
||||||
107
scripts/refetch_corrupted.js
Normal file
107
scripts/refetch_corrupted.js
Normal file
@ -0,0 +1,107 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
/**
|
||||||
|
* 清理状态文件中的乱码标题记录 + 删除 DB 中乱码线索，然后重新抓取对应日期
|
||||||
|
* 用于修复 fetch-v1.2.js chunk 边界截断导致的标题乱码问题
|
||||||
|
*
|
||||||
|
* 用法: node scripts/refetch_corrupted.js # 清理并重新抓取
|
||||||
|
* node scripts/refetch_corrupted.js --dry-run # 仅预览不做操作
|
||||||
|
*/
|
||||||
|
|
||||||
|
const { execSync } = require('child_process')
|
||||||
|
const fs = require('fs')
|
||||||
|
const path = require('path')
|
||||||
|
|
||||||
|
const BASE_DIR = '/data/pingtai/招标公告'
|
||||||
|
const SCRIPT = '/opt/pingtai/scripts/tender-fetcher/fetch-v1.2.js'
|
||||||
|
const DRY_RUN = process.argv.includes('--dry-run')
|
||||||
|
|
||||||
|
/** 收集所有乱码条目 */
|
||||||
|
function findCorrupted() {
|
||||||
|
const results = []
|
||||||
|
const dirs = fs.readdirSync(BASE_DIR).filter(d => /^\d{4}-\d{2}-\d{2}$/.test(d)).sort()
|
||||||
|
for (const dir of dirs) {
|
||||||
|
const statePath = path.join(BASE_DIR, dir, 'downloaded.json')
|
||||||
|
if (!fs.existsSync(statePath)) continue
|
||||||
|
const state = JSON.parse(fs.readFileSync(statePath, 'utf8'))
|
||||||
|
for (const [key, v] of Object.entries(state.downloaded || {})) {
|
||||||
|
if (v.title && v.title.includes('\uFFFD')) {
|
||||||
|
results.push({ date: dir, key, statePath, info: v })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return results
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 从状态文件中移除指定 key */
|
||||||
|
function removeFromState(statePath, key) {
|
||||||
|
const state = JSON.parse(fs.readFileSync(statePath, 'utf8'))
|
||||||
|
if (state.downloaded?.[key]) {
|
||||||
|
delete state.downloaded[key]
|
||||||
|
fs.writeFileSync(statePath, JSON.stringify(state, null, 2))
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
const corrupted = findCorrupted()
|
||||||
|
const affectedDates = [...new Set(corrupted.map(c => c.date))].sort()
|
||||||
|
const affectedIds = corrupted.map(c => c.info.id).filter(Boolean)
|
||||||
|
|
||||||
|
console.log(`找到 ${corrupted.length} 条乱码记录，涉及 ${affectedDates.length} 天`)
|
||||||
|
console.log(`受影响日期: ${affectedDates.join(', ')}`)
|
||||||
|
console.log(`受影响 ID: ${affectedIds.join(', ')}`)
|
||||||
|
|
||||||
|
if (corrupted.length === 0) {
|
||||||
|
console.log('无需清理')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// 1. 清理状态文件
|
||||||
|
console.log('\n=== 清理状态文件 ===')
|
||||||
|
let cleaned = 0
|
||||||
|
for (const c of corrupted) {
|
||||||
|
if (DRY_RUN) {
|
||||||
|
console.log(` [DRY] 将从 ${c.date}/downloaded.json 移除 key: ${c.key}`)
|
||||||
|
cleaned++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if (removeFromState(c.statePath, c.key)) {
|
||||||
|
console.log(` ✅ ${c.date}: 已移除 ${c.key}`)
|
||||||
|
cleaned++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
console.log(`已清理 ${cleaned}/${corrupted.length} 条状态记录`)
|
||||||
|
|
||||||
|
// 2. 重新抓取每个受影响日期
|
||||||
|
console.log('\n=== 重新抓取 ===')
|
||||||
|
for (const date of affectedDates) {
|
||||||
|
if (DRY_RUN) {
|
||||||
|
console.log(` [DRY] 将执行: START_DATE=${date} node ${SCRIPT}`)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
console.log(` ⏳ 抓取 ${date} 的数据...`)
|
||||||
|
const output = execSync(`START_DATE=${date} node ${SCRIPT}`, {
|
||||||
|
cwd: path.dirname(SCRIPT),
|
||||||
|
timeout: 120000,
|
||||||
|
encoding: 'utf8',
|
||||||
|
})
|
||||||
|
// 提取最后几行输出
|
||||||
|
const lines = output.trim().split('\n').slice(-5).join('\n')
|
||||||
|
console.log(` ✅ ${date}: 完成\n${lines}`)
|
||||||
|
} catch (e) {
|
||||||
|
console.error(` ❌ ${date}: ${e.message}`)
|
||||||
|
}
|
||||||
|
// 间隔等待，避免请求过频
|
||||||
|
await new Promise(r => setTimeout(r, 2000))
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log('\n=== 完成 ===')
|
||||||
|
if (!DRY_RUN) {
|
||||||
|
console.log('提示：执行后端采集器以更新 DB（手动触发"中电建"采集器运行）')
|
||||||
|
console.log('或等待下次定时调度自动更新')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
main().catch(console.error)
|
||||||
@ -31,17 +31,20 @@ const args = process.argv.slice(2)
|
|||||||
const keywordIdx = args.indexOf('--keyword')
|
const keywordIdx = args.indexOf('--keyword')
|
||||||
const KEYWORD = (keywordIdx >= 0 && args[keywordIdx + 1]) ? args[keywordIdx + 1] : ''
|
const KEYWORD = (keywordIdx >= 0 && args[keywordIdx + 1]) ? args[keywordIdx + 1] : ''
|
||||||
|
|
||||||
// 公告类型：可按需增减。支持的类型值见 API 返回的 announcementType 字段
|
// 公告类型：与 API 返回的 announcementType 字段值严格一致
|
||||||
const TYPES = ['招采公告'];
|
const TYPES = ['招采公告', '变更公告'];
|
||||||
// 可选：留空则从上次抓取日期继续；有值则从指定日期开始
|
// 可选：留空则从上次抓取日期继续；有值则从指定日期开始（支持环境变量覆盖）
|
||||||
const START_DATE = '';
|
const START_DATE = process.env.START_DATE || '';
|
||||||
const END_DATE = new Date().toISOString().split('T')[0]; // 今天
|
const END_DATE = new Date().toISOString().split('T')[0]; // 今天
|
||||||
// 跨天首次运行(state 为空)时回溯 N 天, 避免错过前一天后半天发布的公告
|
// 跨天首次运行(state 为空)时回溯 N 天, 避免错过前一天后半天发布的公告
|
||||||
const BACKFILL_DAYS = 3;
|
const BACKFILL_DAYS = 3;
|
||||||
const OUTPUT_DIR = path.join(process.env.HOME, 'desktop/招标公告', END_DATE);
|
const OUTPUT_DIR = path.join(process.env.HOME, 'desktop/招标公告', END_DATE);
|
||||||
const PDF_DIR = path.join(OUTPUT_DIR, 'pdf'); // PDF 附件
|
const PDF_DIR = path.join(OUTPUT_DIR, 'pdf'); // PDF 附件
|
||||||
const HTML_DIR = path.join(OUTPUT_DIR, 'html'); // HTML 转换（待安装转换技能）
|
const HTML_DIR = path.join(OUTPUT_DIR, 'html'); // HTML 转换（待安装转换技能）
|
||||||
const STATE_FILE = path.join(OUTPUT_DIR, 'downloaded.json'); // 已下载记录
|
// 每个关键词使用独立的状态文件，避免多关键词共用状态导致后续关键词跳过
|
||||||
|
const STATE_FILE = KEYWORD
|
||||||
|
? path.join(OUTPUT_DIR, `downloaded_${KEYWORD}.json`)
|
||||||
|
: path.join(OUTPUT_DIR, 'downloaded.json');
|
||||||
const LOG_FILE = path.join(OUTPUT_DIR, 'fetch.log'); // 运行日志
|
const LOG_FILE = path.join(OUTPUT_DIR, 'fetch.log'); // 运行日志
|
||||||
const CONCURRENCY = 3; // 并发下载数
|
const CONCURRENCY = 3; // 并发下载数
|
||||||
const RETRY_TIMES = 3;
|
const RETRY_TIMES = 3;
|
||||||
@ -104,9 +107,10 @@ function request(url, options = {}, retries = RETRY_TIMES) {
|
|||||||
};
|
};
|
||||||
|
|
||||||
const req = protocol.request(reqOptions, (res) => {
|
const req = protocol.request(reqOptions, (res) => {
|
||||||
let data = '';
|
const chunks = [];
|
||||||
res.on('data', chunk => data += chunk);
|
res.on('data', chunk => chunks.push(chunk));
|
||||||
res.on('end', () => {
|
res.on('end', () => {
|
||||||
|
const data = Buffer.concat(chunks).toString('utf8');
|
||||||
if (res.statusCode === 301 || res.statusCode === 302) {
|
if (res.statusCode === 301 || res.statusCode === 302) {
|
||||||
const location = res.headers.location;
|
const location = res.headers.location;
|
||||||
if (location.startsWith('http')) {
|
if (location.startsWith('http')) {
|
||||||
@ -348,15 +352,21 @@ async function main() {
|
|||||||
|
|
||||||
log(`========== 开始抓取: ${KEYWORD} ==========`);
|
log(`========== 开始抓取: ${KEYWORD} ==========`);
|
||||||
|
|
||||||
// 确定起始日期（差量：上次运行日期 或 配置的 START_DATE；跨天首次运行回溯 N 天）
|
// 确定起始日期：START_DATE 为空时退缩到上次运行的前 1 天（重叠 24 小时，覆盖 cron 间隔内漏抓的公告）
|
||||||
const sinceDate = START_DATE || (state.lastRun ? state.lastRun.split('T')[0] : null) || (() => {
|
// state.downloaded 通过 systemId 去重，重叠不会造成重复入库
|
||||||
|
let sinceDate = START_DATE || (state.lastRun ? state.lastRun.split('T')[0] : null) || (() => {
|
||||||
const d = new Date(Date.now() - BACKFILL_DAYS * 86400 * 1000);
|
const d = new Date(Date.now() - BACKFILL_DAYS * 86400 * 1000);
|
||||||
return d.toISOString().split('T')[0];
|
return d.toISOString().split('T')[0];
|
||||||
})();
|
})();
|
||||||
|
// 增加 1 天回溯，覆盖 lastRun 之后但在 cron 间隔内发布的公告
|
||||||
|
if (!START_DATE && state.lastRun) {
|
||||||
|
const yesterday = new Date(Date.now() - 86400 * 1000).toISOString().split('T')[0];
|
||||||
|
sinceDate = yesterday;
|
||||||
|
}
|
||||||
if (START_DATE) {
|
if (START_DATE) {
|
||||||
log(`指定日期范围: ${START_DATE} ~ ${END_DATE}`);
|
log(`指定日期范围: ${START_DATE} ~ ${END_DATE}`);
|
||||||
} else if (state.lastRun) {
|
} else if (state.lastRun) {
|
||||||
log(`差量模式: 自 ${state.lastRun.substring(0, 10)} 以来新增`);
|
log(`增量模式(含 1 天回溯): ${sinceDate} ~ ${END_DATE}`);
|
||||||
} else if (sinceDate !== END_DATE) {
|
} else if (sinceDate !== END_DATE) {
|
||||||
log(`跨天补抓: ${sinceDate} ~ ${END_DATE}`);
|
log(`跨天补抓: ${sinceDate} ~ ${END_DATE}`);
|
||||||
} else {
|
} else {
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user