AI Systems Engineer (Model Routing & Evaluation)

Role Overview｜岗位概述We are looking for a Senior AI Systems Engineer with deep experience in multi-model systems, model routing, AI evaluation, and production AI infrastructure. The role will focus on designing systems that can compare, select, and operate multiple foundation models reliably in production.我们正在寻找一位在多模型系统、模型路由、AI 评估及生产级 AI 基础设施方面具备深入经验的高级 AI 系统工程师。该岗位将重点负责设计能够在生产环境中稳定比较、选择和运行多个基础模型的系统能力。The ideal candidate should be able to define evaluation standards, translate evaluation results into routing strategies, and build the supporting platform required to optimize model quality, latency, cost, and reliability.理想候选人应能够建立评估标准，将评估结果转化为模型路由策略，并建设相应的平台能力，持续优化模型质量、延迟、成本及可靠性。Key Responsibilities｜岗位职责Model Routing Strategy｜模型路由策略Design and implement model-routing strategies across multiple foundation models based on task type, complexity, quality requirements, latency, cost, and operational risk.根据任务类型、复杂度、质量要求、延迟、成本及运营风险，设计并实现多基础模型路由策略。Develop dynamic model-selection, arbitration, escalation, fallback, and ensemble mechanisms.建设动态模型选择、模型仲裁、能力升级、智能降级及多模型组合机制。Define routing policies and decision criteria for different task categories and production scenarios.针对不同任务类别及生产场景，定义模型路由策略和决策标准。Continuously improve routing performance using evaluation results, production data, and controlled experimentation.基于评估结果、生产数据及受控实验，持续优化模型路由表现。Distinguish and optimize for model capability rather than relying solely on availability-based failover or static rule-based routing.以模型能力和任务适配度为核心进行优化，而非仅依赖可用性切换或静态规则路由。Evaluation Strategy, Framework & Benchmarking｜评估战略、框架与基准体系Define task taxonomies, evaluation dimensions, scoring criteria, and acceptance thresholds for different AI use cases.针对不同 AI 使用场景，定义任务分类、评估维度、评分标准及验收门槛。Design and maintain benchmark suites, golden datasets, annotation standards, and regression test sets.设计并维护基准测试集、Golden Dataset、标注标准及回归测试集。Build automated evaluation pipelines using deterministic checks, LLM-as-a-Judge, rubric-based scoring, pairwise comparison, and human evaluation where appropriate.根据场景建设自动化评估流程，综合使用确定性校验、LLM-as-a-Judge、Rubric 评分、Pairwise Comparison 及人工评估。Validate evaluation methods against human judgments and monitor judge consistency, bias, and drift.通过人工判断验证评估方法，并持续监测 Judge 的一致性、偏差及漂移。Establish continuous evaluation and regression mechanisms for model, prompt, data, and routing-policy changes.针对模型、Prompt、数据及路由策略变更，建立持续评估和回归机制。Ensure evaluation signals are reliable enough to support model comparison and routing decisions.确保评估信号具备足够可靠性，可用于模型能力比较及路由决策。Multi-model and Model-serving Architecture｜多模型与模型服务架构Design and build scalable infrastructure for integrating and serving multiple proprietary and open-source models.设计并建设可扩展的多模型基础设施，支持闭源模型及开源模型的统一接入与服务。Develop model gateways, unified APIs, version-management mechanisms, and routing infrastructure.建设模型网关、统一接口、版本管理机制及路由基础设施。Support model lifecycle management, traffic governance, rollout, rollback, and model replacement.支持模型全生命周期管理、流量治理、灰度发布、回滚及模型替换。Improve the reliability, scalability, and operational efficiency of multi-model production systems.提升多模型生产系统的可靠性、可扩展性及运行效率。Quality, Cost and Latency Optimization｜质量、成本与延迟优化Define measurable optimization objectives across model quality, inference cost, latency, reliability, and resource utilization.针对模型质量、推理成本、延迟、可靠性及资源利用率，建立可量化的优化目标。Develop systematic methods to balance competing objectives rather than optimizing a single metric in isolation.建立系统化的权衡机制，避免仅优化单一指标。Compare routing strategies against fixed-model baselines and quantify their operational and performance benefits.将路由策略与固定模型方案进行比较，并量化其在性能及运营层面的实际收益。Use offline evaluation, online experimentation, and production feedback to improve routing policies continuously.综合使用离线评估、线上实验及生产反馈，持续优化路由策略。Production Engineering and Observability｜生产工程与可观测性Build production-grade AI services with strong standards for reliability, security, testing, and maintainability.按照可靠性、安全性、测试性及可维护性要求，建设生产级 AI 服务。Implement monitoring, logging, tracing, failure analysis, and performance diagnostics for model calls and routing decisions.针对模型调用及路由决策，建设监控、日志、链路追踪、故障分析及性能诊断能力。Establish data feedback loops to capture model performance, routing outcomes, failure cases, and human-review results.建立数据反馈闭环，持续采集模型表现、路由结果、失败案例及人工审核结果。Support incident investigation, production debugging, regression analysis, and system recovery.支持生产事故调查、线上调试、回归分析及系统恢复。Qualifications｜任职要求Hands-on experience building production AI, machine-learning, or distributed systems.具有生产级 AI、机器学习系统或分布式系统的实际建设经验。Demonstrated experience in model routing, dynamic model selection, model arbitration, ensemble systems, or multi-model decision systems.具有模型路由、动态模型选择、模型仲裁、多模型组合或多模型决策系统的实际经验。Experience designing evaluation frameworks, benchmark datasets, scoring methodologies, regression pipelines, or continuous evaluation systems.具有评估框架、Benchmark Dataset、评分方法、回归流程或持续评估体系的设计与建设经验。Strong understanding of the trade-offs among model quality, latency, inference cost, reliability, and operational risk.深入理解模型质量、延迟、推理成本、可靠性及运营风险之间的权衡关系。Experience integrating and operating multiple foundation models, including third-party APIs, self-hosted models, or open-source models.具有多个基础模型的接入及运行经验，包括第三方模型 API、自托管模型或开源模型。Ability to translate ambiguous requirements into measurable evaluation criteria, technical designs, and production systems.能够将模糊需求转化为可量化的评估标准、技术方案及生产系统。Strong analytical and problem-solving skills, with the ability to validate technical decisions using data and experiments.具备较强的分析及问题解决能力，能够通过数据和实验验证技术决策。Preferred Qualifications｜优先条件Experience with LLM-as-a-Judge, human evaluation, judge calibration, preference evaluation, or benchmark development.具有 LLM-as-a-Judge、人工评估、Judge 校准、偏好评估或 Benchmark 建设经验。Experience with model serving, inference optimization, model gateways, observability platforms, or LLMOps systems.具有模型服务、推理优化、模型网关、可观测性平台或 LLMOps 系统经验。Experience with reinforcement learning, contextual bandits, ranking, recommendation systems, search, or information retrieval.具有强化学习、Contextual Bandit、排序、推荐系统、搜索或信息检索经验。Experience with AI safety, red teaming, model governance, or evaluation of high-risk AI systems.具有 AI 安全、Red Teaming、模型治理或高风险 AI 系统评估经验。What We Value｜我们重视的能力You think in terms of systems, objectives, and measurable outcomes rather than individual prompts or isolated model outputs.你能够从系统、目标及可量化结果出发思考问题，而非局限于单个 Prompt 或单次模型输出。You can distinguish true model routing from static rules, failover, or load balancing.你能够清晰区分真正的模型路由，与静态规则、故障切换及负载均衡。You understand that reliable routing depends on reliable evaluation.你理解可靠的模型路由必须建立在可靠的评估体系之上。You are comfortable owning both technical strategy and production implementation.你能够同时承担技术策略设计及生产系统落地。