Skills Guide

DataFusion + DeltaLake Operations Reference

VerifiedSafe

Supplies a comprehensive reference map for DataFusion and DeltaLake APIs, with strict rules to avoid guessing interfaces. Helps navigate local reference files and apply existing patterns in this repository when using these tools.

Download and use
Sby Skills Guide Bot
Data & AIIntermediate
6/2/2026
Claude CodeCursorWindsurf
#datafusion#deltalake#query-engine#storage-layer#reference

Recommended for

Claude CodeCursorWindsurfData scientistData & analysis

Automated summary

This skill provides a structured reference map and operational rules for using DataFusion and DeltaLake in a repository, enabling accurate API usage through local probes and documentation lookup.

Strengths

  • Reduces API guesswork by relying on local references
  • Leverages existing code patterns in the repository
  • Covers many aspects (Rust, Python, SQL, planning, integration)
  • Encourages methodical environment probing

Limitations

  • Requires the reference files to be present in the repository
  • May not cover all edge cases or entirely new APIs
  • Depends on local DataFusion/DeltaLake versions
When to use it

When developing or debugging DataFusion/DeltaLake code in a repository that includes these reference files.

When not to use it

When the repository lacks the reference files or when exploring completely new undocumented APIs.

Security analysis

Safe
Quality score90/100

The skill only describes how to look up information in local references and probe the environment with safe Bash commands (e.g., version checks). No destructive actions, external network calls, or obfuscated payloads are instructed.

No concerns found

How this analysis is made

Examples

Look up Delta table registration
Open the DeltaLake integration reference and find how to register a Delta table from a given path using DataFusion's catalog.
Find UDF patterns
Search the repo for existing Rust UDF implementations, then open the Rust UDF reference to check the current contracts for Scalar UDFs.
Probe environment and check planning
Run a local probe to get the DataFusion version, then open the planning deep dive to understand how predicate pushdown works with DeltaLake scan providers.

name: dfdl_ref description: DataFusion + DeltaLake operations manual for this repo. DataFusion is the core query engine; DeltaLake provides the storage layer and integrates tightly via scan providers, schema bridging, and predicate pushdown. Use lookup + local probes; do not guess APIs. allowed-tools: Read, Grep, Glob, Bash

Operating rule: never guess DataFusion/DeltaLake/PyArrow/UDF APIs

When uncertain:

  1. Probe local environment (versions + available methods).
  2. Search the repo for how we already use it.
  3. Open the relevant reference file below (only the section you need).
  4. Implement using existing local patterns unless the plan says otherwise.

Reference map (open these files as needed)

  • Core DataFusion Python surfaces (IO, catalog, SQL, DataFrame API): reference/datafusion.md
  • "Best-in-class deployment gaps" (caching, stats, observability, planning knobs): reference/datafusion_addendum.md
  • Planning deep dive (logical/physical plan pipeline, introspection, optimization rules): reference/datafusion_planning.md
  • Rust UDF contracts (Scalar/UDAF/UDWF/Async/named args): reference/datafusion_rust_UDFs.md
  • Schema management + schema pitfalls: reference/datafusion_schema.md
  • DeltaLake ↔ DataFusion integration details: reference/deltalake_datafusion_integration.md
  • Advanced Rust integration (PyO3 packaging, wheels, CI, native module distribution): reference/datafusion_deltalake_advanced_rust_integration.md
  • DataFusionMixins trait (Delta snapshot schema + predicate parsing helpers): reference/deltalake_datafusionmixins.md
  • Plan combination (composing DataFusion plans via joins/unions/CTEs, Delta integration, parameterized queries, plan serialization): reference/datafusion_plan_combination.md
  • Rust LogicalPlan programmatic construction (LogicalPlanBuilder, Expr, schema/DFSchema, plan rewriting via TreeNode, extensibility, serialization): reference/Datafusion_logicplan_rust.md
  • DataFusion tracing (Rust community extension: execution spans, metrics capture, partial-result previews, rule-phase instrumentation, OpenTelemetry export): reference/datafusion-tracing.md
  • DeltaLake core (format/protocol, client APIs, 3-layer model): reference/deltalake.md

From the file to a first result

The download contains exactly the content displayed on this page. This copy may differ from the latest version in the source repository.

View the source and supporting files

This download contains SKILL.md only. Any scripts, references, templates and dependencies are not included. Get the required files from the source, preserving their directory structure, before continuing.

1. Download the file

Download SKILL.md

From the root of a test project, save the file as .skill-downloads/datafusion-deltalake-operations-reference-mm78nnnl/SKILL.md, or use the command below. This folder is for inspecting the skill before installation.

Command for macOS/Linux or WSL: requires a POSIX shell and curl. On Windows without WSL, use the download button and create the folder manually.

mkdir -p '.skill-downloads/datafusion-deltalake-operations-reference-mm78nnnl' && curl --fail --show-error --location 'https://skills-guides.com/api/skills/datafusion-deltalake-operations-reference-mm78nnnl/download' --output '.skill-downloads/datafusion-deltalake-operations-reference-mm78nnnl/SKILL.md.part' && mv '.skill-downloads/datafusion-deltalake-operations-reference-mm78nnnl/SKILL.md.part' '.skill-downloads/datafusion-deltalake-operations-reference-mm78nnnl/SKILL.md'

2. Check prerequisites

Read SKILL.md before using it. If it references scripts or external services, inspect them and prepare the required access in your test environment.

Tools declared by the author (check availability in your agent): Read, Grep, Glob, Bash

No recognized prerequisites section in this file. This does not mean it is self-contained: libraries, tools and access requirements may be mentioned elsewhere in the content or source repository.

The platforms on this page are catalog indications. The paths below follow tool documentation; they do not prove this skill has been run in those tools. No minimum tool version is confirmed for this listing.

3. Load the skill

This file does not provide a usable name/description header for this native procedure. It remains available to download and inspect; use the manual method below.

Manual method or another tool: attach SKILL.md to a conversation, or open it in an agent that can read files. Include any required supporting resources. No universal folder is recognized by every tool.

4. Try a first task

Use a short, synthetic input: an API endpoint, a small document or an interface brief. In a new conversation, attach the file or give the agent its path, then adapt this request:

Read .skill-downloads/datafusion-deltalake-operations-reference-mm78nnnl/SKILL.md and apply its instructions to this example: [describe a test input]. Identify any missing files, dependencies or access. Produce the result requested by the skill without inventing data or running unreviewed scripts.

Expected result: output matching the skill instructions, or a precise list of missing prerequisites. A read confirmation alone does not validate a first use. If an external resource is unavailable, that path remains untested.

Related skills

Create Interactive Maps with kepler.gl

Data & AI

Create interactive map visualizations and export standalone HTML using the keplergl Python package. Use to plot geospatial data from DataFrames, GeoJSON, or CSV.

Claude CodeCursorCodexintermediateSafe 11,995
Skills Guide Bot

Agent-Driven Parameter Optimization

Data & AI

Guide for optimizing configurable systems with a measurable objective, using bounded trials and evidence-based recommendations.

Claude CodeCursorCodexintermediateSafe 45,609
Skills Guide Bot

ICP Visitor Outreach

Data & AI

Identify signed-up users matching the ideal customer profile but stalled, and draft personalized outreach to re-engage them.

Claude CodeintermediateSafe 6,843
Skills Guide Bot