Update Check — ONCE PER SESSION (mandatory) The first time this skill is used in a session, run the check-updates skill before proceeding.

• GitHub Copilot CLI / VS Code: invoke the

  check-updates

  skill.
• Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.
• Skip if the check was already performed earlier in this session.

CRITICAL NOTES

1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
2. To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering

• Check for recent jobs BEFORE creating new notebook runs — Query job instances from last 5 minutes; if recent job exists, monitor it instead of creating duplicate
• Capture job instance ID immediately after POST — Store job ID before any other operations to enable proper monitoring
• Verify workspace capacity assignment before operations — Workspace must have capacity assigned and active
• When user provides a public data URL, follow the Public URL Data Ingestion policy — keep detailed behavior in the linked resource section to avoid drift/duplication
• Format notebook cells correctly — Each line in cell source array MUST end with

  \n

  to prevent code merging
• Use correct Livy session body format — Send a FLAT JSON with

  name

  ,

  driverMemory

  ,

  driverCores

  ,

  executorMemory

  ,

  executorCores

  . Do NOT wrap in

  {"payload": ...}

  or send only

  {"kind": "pyspark"}

  — that causes HTTP 500. Use valid memory values (28g, 56g, 112g, 224g). See Create Livy Session example below and SPARK-CONSUMPTION-CORE.md.

• Poll job status with proper intervals — 10-30 seconds between polls; timeout after reasonable duration (e.g., 30 minutes)
• Check job history when POST response is unreadable — If POST returns "No Content" or unreadable response, query recent jobs (last 1 minute) before retrying
• Use Starter Pool for development — Development/testing workloads should use

  useStarterPool: true

• Use Workspace Pool for production — Production workloads need consistent performance with

  useWorkspacePool: true

• Enable lakehouse schemas during creation — Set

  creationPayload.enableSchemas: true

  for better table organization
• Implement idempotency checks — Prevent duplicate operations by checking existing state first

• Never retry POST with same parameters — If you have a job ID, only use GET to check status; don't create duplicate job instances
• Don't skip capacity verification — Operations will fail if workspace capacity is paused or unassigned
• Avoid immediate POST retries on failures — Check for existing/active jobs first to prevent duplicates
• Don't create new runs if monitoring existing job — One job at a time; wait for completion before submitting new runs
• Don't hardcode workspace/lakehouse IDs — Discover dynamically via item listing or catalog search APIs

Rule 1 — Validate prerequisites before operations. Verify workspace has capacity assigned (see COMMON-CORE.md Create Workspace and Capacity Management) and resource IDs exist before attempting operations.

Rule 2 — Trust updateDefinition success. A

Succeeded

poll result from

updateDefinition

is sufficient confirmation that content and lakehouse bindings persisted. Do NOT call

getDefinition

after every upload — it is an async LRO that adds significant latency. Only use

getDefinition

for its intended purpose: reading current notebook content before making modifications.

Rule 3 — Prevent duplicate jobs and monitor execution properly. Before submitting new notebook run, ALWAYS check for recent job instances first (last 5 minutes). If recent job exists, monitor it instead of creating duplicate. After submission, capture job instance ID immediately and poll status - never retry POST. See SPARK-AUTHORING-CORE.md Job Monitoring for patterns.

• COMMON-CORE.md — Fabric REST API patterns, authentication, item discovery
• COMMON-CLI.md —

  az rest

  usage, environment detection, token acquisition
• SPARK-AUTHORING-CORE.md — Notebook deployment, lakehouse creation, job execution