Senior Prompt Engineer

What This Skill Does

This skill offers tools for prompt engineering, LLM evaluation, and agentic system design. It helps prompt engineers optimize prompts, evaluate RAG systems, and orchestrate agents for complex tasks.

When to Use

• Optimize a prompt for token efficiency.
• Evaluate RAG retrieval quality.
• Visualize agent workflow from definition.
• Design few-shot examples for a prompt.
• Create prompts for structured output.
• Analyze a prompt for clarity and structure.

Key Features

• Analyzes prompts for token efficiency and clarity.
• Generates optimized versions of prompts.
• Evaluates RAG retrieval quality.
• Visualizes agent workflows from configuration files.
• Provides tools for prompt optimization.
• Supports prompt analysis and optimization.

Manual Installation

Manual installation

• View Full Skill Content.
• The complete markdown content that gets installed.
• Senior Prompt Engineer.

Prompt engineering patterns, LLM evaluation frameworks, and agentic system design.

Table of Contents

Quick Start

Tools Overview

• Prompt Optimizer
  • RAG Evaluator
  • Agent Orchestrator Prompt Engineering Workflows
  • Prompt Optimization Workflow
  • Few-Shot Example Design
  • Structured Output Design Reference Documentation Common Patterns Quick Reference

Quick Start

Analyze and optimize a prompt file

python scripts/prompt_optimizer.py prompts/my_prompt.txt --analyze

Evaluate RAG retrieval quality python scripts/rag_evaluator.py --contexts contexts.json --questions questions.json

Visualize agent workflow from definition python scripts/agent_orchestrator.py agent_config.yaml --visualize

Tools Overview

1. Prompt Optimizer

• Analyzes prompts for token efficiency, clarity, and structure.
• Generates optimized versions.

Input: Prompt text file or string Output: Analysis report with optimization suggestions

Usage:

Analyze a prompt file

python scripts/prompt_optimizer.py prompt.txt --analyze

Output: Token count: 847 Estimated cost: $0.0025 (GPT-4) Clarity score: 72/100 Issues found:

• Ambiguous instruction at line 3
• Missing output format specification
• Redundant context (lines 12-15 repeat lines 5-8) Suggestions:

1. Add explicit output format: "Respond in JSON with keys: ..."
2. Remove redundant context to save 89 tokens
3. Clarify "analyze" -> "list the top 3 issues with severity ratings"

Generate optimized version python scripts/prompt_optimizer.py prompt.txt --optimize --output optimized.txt

Count tokens for cost estimation python scripts/prompt_optimizer.py prompt.txt --tokens --model gpt-4

Extract and manage few-shot examples python scripts/prompt_optimizer.py prompt.txt --extract-examples --output examples.json

2. RAG Evaluator

Evaluates Retrieval-Augmented Generation quality by measuring context relevance and answer faithfulness.

Input: Retrieved contexts (JSON) and questions/answers Output: Evaluation metrics and quality report

Usage:

Evaluate retrieval quality

python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json

Output: === RAG Evaluation Report === Questions evaluated: 50

Retrieval Metrics: Context Relevance: 0.78 (target: >0.80) Retrieval Precision@5: 0.72 Coverage: 0.85

Generation Metrics: Answer Faithfulness: 0.91 Groundedness: 0.88

Issues Found:

• 8 questions had no relevant context in top-5
• 3 answers contained information not in context

Recommendations:

1. Improve chunking strategy for technical documents
2. Add metadata filtering for date-sensitive queries

Evaluate with custom metrics python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json
--metrics relevance,faithfulness,coverage

Export detailed results python scripts/rag_evaluator.py --contexts retrieved.json --questions eval_set.json
--output report.json --verbose

3. Agent Orchestrator

• Parses agent definitions and visualizes execution flows.
• Validates tool configurations.

Input: Agent configuration (YAML/JSON) Output: Workflow visualization, validation report

Usage:

Validate agent configuration

python scripts/agent_orchestrator.py agent.yaml --validate

Output: === Agent Validation Report === Agent: research_assistant Pattern: ReAct

Tools (4 registered): [OK] web_search - API key configured [OK] calculator - No config needed [WARN] file_reader - Missing allowed_paths [OK] summarizer - Prompt template valid

Flow Analysis: Max depth: 5 iterations Estimated tokens/run: 2,400-4,800 Potential infinite loop: No

Recommendations:

1. Add allowed_paths to file_reader for security
2. Consider adding early exit condition for simple queries

Visualize agent workflow (ASCII) python scripts/agent_orchestrator.py agent.yaml --visualize

Output: ┌─────────────────────────────────────────┐ │ research_assistant │ │ (ReAct Pattern) │ └─────────────────┬───────────────────────┘ │ ┌────────▼────────┐ │ User Query │ └────────┬────────┘ │ ┌────────▼────────┐ │ Think │◄──────┐ └────────┬────────┘ │ │ │ ┌────────▼────────┐ │ │ Select Tool │ │ └────────┬────────┘ │ │ │ ┌─────────────┼─────────────┐ │ ▼ ▼ ▼ │ [web_search] [calculator] [file_reader] │ │ │ │ └─────────────┼─────────────┘ │ │ │ ┌────────▼────────┐ │ │ Observe │───────┘ └────────┬────────┘ │ ┌────────▼────────┐ │ Final Answer │ └─────────────────┘

Export workflow as Mermaid diagram python scripts/agent_orchestrator.py agent.yaml --visualize --format mermaid

Prompt Engineering Workflows

Prompt Optimization Workflow

Use when improving an existing prompt's performance or reducing token costs.

Step 1: Baseline current prompt python scripts/prompt_optimizer.py current_prompt.txt --analyze --output baseline.json

Step 2: Identify issues Review the analysis report for: Token waste (redundant instructions, verbose examples) Ambiguous instructions (unclear output format, vague verbs) Missing constraints (no length limits, no format specification)

Step 3: Apply optimization patterns

Step 4: Generate optimized version python scripts/prompt_optimizer.py current_prompt.txt --optimize --output optimized.txt

Step 5: Compare results python scripts/prompt_optimizer.py optimized.txt --analyze --compare baseline.json Shows: token reduction, clarity improvement, issues resolved

Step 6: Validate with test cases Run both prompts against your evaluation set and compare outputs.

Few-Shot Example Design Workflow

Use when creating examples for in-context learning.

Step 1: Define the task clearly Task: Extract product entities from customer reviews Input: Review text Output: JSON with {product_name, sentiment, features_mentioned}

Step 2: Select diverse examples (3-5 recommended)

Step 3: Format consistently Example 1: Input: "Love my new iPhone 15, the camera is amazing!" Output: {"product_name": "iPhone 15", "sentiment": "positive", "features_mentioned": ["camera"]}

Example 2: Input: "The laptop was okay but battery life is terrible." Output: {"product_name": "laptop", "sentiment": "mixed", "features_mentioned": ["battery life"]}

Step 4: Validate example quality python scripts/prompt_o