Skills

Awesome Data Engineering Skills: 36 data engineering SKILL.md files that teach AI agents dbt, Airflow, Spark, and data quality

For agents compatible with the Agent Skills standard — such as Claude Code and Cursor — to master data engineering workflows like dbt, Airflow, Spark, and Snowflake, plus experience with idempotency, backfills, and data quality. Prerequisites: your agent must recognize the skills directory, and installation requires cloning the repository and running the script.

Official source · GitHub - Unknown-333/awesome-data-engineering-skills

Install

./scripts/install.sh agents

This is a collection of 36 data engineering Skills covering dbt, Airflow, Dagster, Prefect, Spark, Snowflake, BigQuery, Databricks, Kafka, Iceberg and other topics, along with practical knowledge on idempotency, backfills, and data quality. It is not a single tool, but a set of portable SKILL.md files written to the open Agent Skills standard, licensed under Apache-2.0.

Who It's For

  • Users doing data engineering tasks in Claude Code, Cursor, Codex, GitHub Copilot, Gemini CLI, and other agents compatible with the Agent Skills standard: writing dbt models, configuring Airflow/Dagster/Prefect orchestration, optimizing Spark jobs, working with Snowflake/BigQuery/Databricks, handling Kafka and Iceberg.
  • Scenarios that need guidance on general data engineering experience such as idempotency, backfills, and data quality.
  • When it's not a fit: if your agent doesn't support reading a skills directory in SKILL.md format, whether the installation method works has not been officially stated — this is unconfirmed by repository evidence and needs to be verified yourself. The specific requirements of each Skill regarding tool versions and platforms are not listed item by item in the evidence, so they cannot be guaranteed one by one.
  • Prerequisite: you need to be able to clone the repository locally and expose the skills to your agent via the installation script.

First Attempt in 3 Minutes

The officially recommended main path is to use the repository's bundled installation script:

  1. Clone the repository locally.
  2. Run:
./scripts/install.sh agents

This command places the skills into ./.agents/skills/ as symbolic links — officially noted as the directory recognized by the widest range of tools. Add --user for a global installation; add --copy if you prefer copying over symlinking.

Claude Code users have an additional optional route, installing via the plugin marketplace:

/plugin marketplace add <your-org>/awesome-data-engineering-skills
/plugin install data-engineering-skills@awesome-data-engineering-skills

Note that <your-org> in this command must be replaced with the actual repository path. Other platforms have no official marketplace route; they only support the script or manually copying/linking into their respective skills directories (e.g., .claude/skills/, .cursor/skills/, .codex/skills/, .github/skills/).

  1. Loading is lazy: the agent usually preloads each skill's name and description, and only reads the full SKILL.md when your task matches. You can also proactively invoke a skill by typing / in the chat.

  2. For your first attempt, pick a low-risk diagnostic task, for example asking the agent: "Using the data quality skill, check whether my set of tables has any issues in idempotency and backfill design." The success signal is: the agent reads the corresponding SKILL.md and, following the checks described in it, produces analysis output tailored to your task. Note: whether the agent reads/writes files, executes commands, or accesses the network depends on the client it runs in and the specific task you give it; the official documentation makes no unified promise about this, so zero risk cannot be assumed.

Use Cases

Case 1: Backfill and idempotency check. Scenario: you have a scheduled job that needs to be rerun due to historical data errors. Tell the agent: "Referring to the backfill skill, help me check whether this job's backfill design is idempotent." The agent will load the corresponding SKILL.md and analyze it using the idempotency and backfill experience it contains, letting you get a diagnostic conclusion before changing any code.

Case 2: Data quality review. Scenario: quality gating before onboarding a new data source. Tell the agent: "Use the data quality skill to do a quality review of my modeling plan." The agent will read the full content of that skill and output review points. What can be confirmed is the step of "the agent producing analysis based on the skill's content"; deeper capabilities such as table-by-table scanning or generating SQL depend on the actual content of the specific SKILL.md — the evidence at the collection level does not cover these item by item.

Boundaries of Use

  • Installation: script installation uses symbolic links (unless --copy is added), placed in the project directory or a user-level directory; skills directories differ across platforms, and only Claude Code has an official plugin marketplace path.
  • Side effects and confirmation points: skills are documents in themselves, but what actions the agent takes after reading a skill is determined by your task; before operations that modify files, execute commands, or call external databases/services, you should explicitly confirm. The official documentation declares no unified sandbox or security boundary.
  • Officially stated limitations: skills are written to the open Agent Skills standard, and compatibility assumes "agents that support that standard"; the official documentation does not list any restrictions on specific file types, SQL dialect versions, or text languages — this information is unknown, so do not infer it yourself.
  • Key information not yet confirmed: the detailed content of each of the 36 skills, minimum version requirements for each client, and whether skills might lead the agent to access the network or execute code are all not stated in the evidence.