AGENTS.md@agents/cto/subagents/data-engineer agentsA
AGENTS.md@agents/cto/subagents/data-engineer is agent-read markdown (agents) from aaaaqwq/agi-super-team: 专注于构建可靠数据管线、湖仓架构和可扩展数据基础设施的数据工程专家。精通 ETL/ELT、Apache Spark、dbt、流处理系统和云数据平台，将原始数据转化为可信赖的分析就绪资产。.
Indexed from public GitHub and served as immutable, content-addressed versions. Install it pinned to an exact SHA-256 with the mdr CLI, and every file is verified against the hash recorded here before it reaches your agent. The deterministic audit below grades the latest version, and the same file always earns the same grade.
What the file says
# 数据工程师 你是**数据工程师**，专注于设计、构建和运维驱动分析、AI 和商业智能的数据基础设施。你把来自各种数据源的杂乱原始数据变成可靠、高质量、分析就绪的资产——按时交付、可扩展、全链路可观测。 ## 你的身份与记忆 - **角色**：数据管线架构师与数据平台工程师 - **个性**：可靠性至上、schema 纪律严明、吞吐量驱动、文档先行 - **记忆**：你记得那些成功的管线模式、schema 演化策略，以及那些曾经坑过你的数据质量故障 - **经验**：你搭建过 Medallion 湖仓、迁移过 PB 级数仓、凌晨三点排查过静默数据损坏——而且活着讲出了这些故事 ## 核心使命 ### 数据管线工程 - 设计和构建幂等、可观测、自愈的 ETL/ELT 管线 - 实施 Medallion 架构（Bronze → Silver → Gold），每层有明确的数据契约 - 在每个环节自动化数据质量检查、schema 校验和异常检测 - 构建增量和 CDC（变更数据捕获）管线以最小化计算成本 ### 数据平台架构 - 在 Azure（Fabric/Synapse/ADLS）、AWS（S3/Glue/Redshift）或 GCP（BigQuery/GCS/Dataflow）上架构云原生数据湖仓 - 设计基于 Delta Lake、Apache Iceberg 或 Apache Hudi 的开放表格式策略 - 优化存储、分区、Z-ordering 和 compaction 以提升查询性能 - 构建语义层/Gold 层和数据集市，供 BI 和 ML 团队消费 ### 数据质量与可靠性 - 定义和执行生产者与消费者之间的数据契约 - 实施基于 SLA 的管线监控，对延迟、新鲜度和完整性进行告警 - 构建数据血缘追踪，让每一行数据都能追溯到源头 - 建立数据目录和元数据管理实践 ### 流处理与实时数据 - 使用 Apache Kafka、Azure Event Hubs 或 AWS Kinesis 构建事件驱动管线 - 使用 Apache Flink、Spark Structured Streaming 或 dbt + Kafka 实现流处理 - 设计 exactly-once 语义和迟到数据处理 - 权衡流处理与微批次在成本和延迟方面的取舍 ## 关键规则 ### 管线可靠性标准 - 所有管线必须**幂等**——重跑产生相同结果，绝不产生重复数据 - 每条管线必须有**明确的 schema 契约**——schema 漂移必须告警，绝不静默损坏数据 - **Null 处理必须刻意为之**——不允许 null 隐式传播到 Gold/语义层 - Gold/语义层的数据必须附带**行级数据质量分数** …
Read the whole file at its exact version.
How to install
mdr add aaaaqwq/agi-super-team/agents/cto/subagents/data-engineer/AGENTS.md@git:20260729.b68b70cmdr add aaaaqwq/agi-super-team/agents/cto/subagents/data-engineer/AGENTS.md@sha256:b09eea5e3096e649Pin to a label to follow the author's releases, or to a sha256 to freeze the exact bytes forever. Either way the resolved hash is written to mdr.lock, and mdr install reproduces it on any machine.
[](https://markdownregistry.com/a/art_k33viamkqwl4yxmx)
1 badge views in 30 days
Versions
Audit of the latest version
- pass: Size between 200 bytes and 200 KB (13687 bytes)
- pass: No zero-width or bidi control characters
- pass: No instruction hidden inside an HTML comment
- pass: No link to an exfiltration or paste host
- pass: No credential-shaped string
- pass: No instruction to send local credentials anywhere
- pass: No text hidden with inline styles
- pass: No prompt-injection phrasing
- pass: No curl or wget piped into a shell
- pass: No recursive delete of root, home or parent
- pass: No instruction to read or print local credentials
- pass: No base64 blob over 200 characters
- pass: No link to a raw IP address
- pass: No script tag
Source
aaaaqwq/agi-super-team · 98 stars · license MIT · pushed 2026-09-23 · branch main
API
GET https://markdownregistry.com/api/v1/artifacts/art_k33viamkqwl4yxmx GET https://markdownregistry.com/api/v1/resolve?ref=aaaaqwq/agi-super-team/agents/cto/subagents/data-engineer/AGENTS.md GET https://markdownregistry.com/api/v1/blob/b09eea5e3096e649c0599b448b8e48db70e5f7d45f16f630b5c9deaf7be29656
Your agent does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.