Edge Case Detection
Edge Case Detection provides visual analysis tools to identify and resolve problematic inputs that cause low performance.
What you'll learn ⏱️ 6 minutes
How the performance scatter plot visualization works
Understanding score clustering and patterns
Selecting problem areas for targeted optimization
Interpreting visual performance data
When to use Edge Case Detection in your workflow
Best practices for systematic problem identification
Accessing Edge Case Detection
From your project dashboard, click the Actions button on any task, then select "Edge Case Detection".
Important: This feature requires sufficient optimization data to be meaningful. You'll need approximately 15-20+ API calls (optimization runs) before the visualization becomes useful for pattern identification.
Edge Case Detection Interface:
Performance Scatter Plot: Visual representation of input performance
Score Clustering: Color-coded performance ranges
Selection Tools: Choose specific problem areas for optimization
Optimization Actions: Target selected areas for improvement
Understanding the Performance Scatter Plot
Visual Overview
The scatter plot displays each API call as a point on a graph, where:
X-axis: Represents one performance dimension
Y-axis: Represents another performance dimension
Each dot: Represents one input/output interaction
Colors: Indicate performance score ranges
Score Color Coding
The visualization uses color coding to show performance levels:
🔴 Red Dots (Low Score 0-3)
Significant performance issues
Requires immediate attention
Often represents edge cases or problematic inputs
Primary targets for optimization
🟠 Orange Dots (Medium Score 4-6)
Moderate performance issues
Room for improvement
May represent challenging but solvable scenarios
Secondary optimization targets
🔵 Blue Dots (Good Score 7-8)
Solid performance
Generally working well
May still benefit from fine-tuning
Lower priority for optimization
🟢 Green Dots (Excellent Score 9-10)
Optimal performance
Inputs that work very well
Can serve as models for optimization
Use to understand what works
Performance Clusters
Look for clusters of dots with similar colors:
Red clusters: Systematic problems requiring attention
Mixed clusters: Inconsistent performance on similar inputs
Isolated red dots: Specific edge cases
Green clusters: Successful patterns to replicate
Using the Scatter Plot Interface
Selection Tools
"Clear selection" button: Reset any selected areas Click and drag: Select specific areas of the plot Multiple selections: Choose several problem areas at once
Selection Strategy
Target red clusters first:
Select groups of low-performing inputs
Focus on dense clusters rather than isolated points
Look for patterns in problem areas
Example selection process:
Identify a cluster of red dots in one area of the plot
Click and drag to select that cluster
Click "Optimize" to target those specific scenarios
Review results and repeat for other problem areas
Optimization Actions
"Optimize" button: Runs targeted optimization on selected inputs
Focuses improvement efforts on the specific problem areas you've identified
More efficient than general optimization
Addresses systematic issues rather than random problems
Interpreting Performance Patterns
Common Scatter Plot Patterns
Scattered Performance
Dots spread randomly across score ranges
Indicates inconsistent performance across different inputs
Suggests need for better Prompt Family development
Solution: Focus on building more specialized prompts
Clustered Problems
Groups of red/orange dots in specific areas
Indicates systematic issues with certain input types
Represents identifiable problem patterns
Solution: Target specific clusters for optimization
Clear Separation
Distinct groups of high and low performers
Shows some inputs work well, others consistently fail
Indicates need for specialized handling
Solution: Create prompt family members for different input types
Gradual Distribution
Smooth distribution from low to high scores
Shows general improvement trend
Indicates optimization is working but needs continuation
Solution: Continue current optimization strategy
Advanced Pattern Analysis
Performance Corridors
Linear arrangements of dots suggesting input complexity relationships
May indicate that certain input characteristics correlate with performance
Helps identify what makes inputs easy or difficult to handle
Outlier Detection
Individual dots far from main clusters
May represent unusual edge cases or data quality issues
Important to investigate these specific scenarios
Density Patterns
Areas with many dots vs sparse areas
High-density areas represent common input types
Sparse areas may represent unusual scenarios worth investigating
Systematic Edge Case Resolution
Identification Workflow
Review overall distribution: Get sense of general performance patterns
Identify red clusters: Find groups of problematic inputs
Select specific areas: Choose dense problem clusters first
Run targeted optimization: Focus improvement on selected areas
Reassess results: Check if optimization improved the targeted areas
Prioritization Strategy
High Priority: Dense red clusters affecting many inputs Medium Priority: Orange clusters with improvement potential Low Priority: Isolated red dots representing rare edge cases
Monitor: Blue and green areas for performance maintenance
Optimization Targeting
For dense red clusters:
Create specific prompt family members for these input types
Add more manual inputs representing these scenarios
Create targeted evaluations for these problem areas
For scattered problems:
General prompt optimization may be more effective
Focus on overall Prompt Family diversity
Improve general robustness rather than specific targeting
Integration with Other Optimization Tools
Edge Case Detection + Input Optimization
Problem identification workflow:
Use scatter plot to identify problematic input patterns
Go to Input Optimization → End User Inputs to find specific examples
Add similar examples to Manual Inputs for systematic testing
Use these inputs for targeted prompt optimization
Edge Case Detection + Prompt Optimization
Targeted improvement workflow:
Select problem clusters in scatter plot
Run targeted optimization (creates events focused on these scenarios)
Review results in Prompt Optimization → Event Log
Use Manual Optimization to create specialized prompt family members
Edge Case Detection + Evaluations
Criteria refinement workflow:
Identify consistent problem patterns in scatter plot
Create specific evaluations targeting these problem types
Use new evaluations to guide optimization of problem areas
Monitor improvement in subsequent scatter plot analysis
Best Practices for Edge Case Detection
When to Use Edge Case Detection
After initial optimization: Need baseline data to see patterns
When performance plateaus: Identify specific areas needing attention
Regular monitoring: Weekly or monthly review of performance patterns
Before major changes: Understand current problem areas before modifications
Effective Selection Strategies
Start with obvious clusters: Target dense red areas first
Work systematically: Address one cluster at a time
Document findings: Note what types of inputs cause problems
Test improvements: Verify that optimization actually helps selected areas
Common Mistakes to Avoid
Optimizing isolated points: Focus on patterns, not individual failures
Ignoring successful areas: Learn from what works well (green clusters)
Over-optimization: Don't endlessly optimize areas that are "good enough"
Premature analysis: Wait for sufficient data before drawing conclusions
Troubleshooting Edge Case Detection
Scatter Plot Shows No Clear Patterns
Cause: May not have enough data points or performance is genuinely random
Solution: Run more optimization attempts or review input quality and evaluation criteria
All Dots Are Red (Low Performance)
Cause: Fundamental issues with prompts, evaluations, or model selection
Solution: Return to basic optimization before using Edge Case Detection
No Red Clusters Visible
Cause: Performance may be generally good, or problem areas are too scattered
Solution: Focus on orange clusters for improvement, or use other optimization tools
Selected Areas Don't Improve After Optimization
Cause: May represent genuinely difficult edge cases or evaluation criteria issues
Solution: Review specific inputs in selected areas and consider if they represent realistic expectations
Advanced Edge Case Strategies
Progressive Problem Resolution
Week 1: Address dense red clusters (highest impact)
Week 2: Target orange clusters (medium improvement)
Week 3: Fine-tune blue areas (marginal gains)
Week 4: Monitor and maintain green areas (prevent regression)
Pattern Documentation
Keep records of:
What input patterns cause problems
Which optimization strategies work for specific clusters
How patterns change over time
Successful resolution approaches
Proactive Edge Case Management
Regular monitoring: Weekly scatter plot review
Trend analysis: Monthly comparison of problem areas
Preventive optimization: Address emerging patterns before they become major issues
Knowledge transfer: Share successful strategies across similar tasks
Performance Monitoring with Edge Case Detection
Success Indicators
Fewer red clusters: Problems being systematically resolved
Tighter clustering: More consistent performance across input types
Upward migration: Orange dots becoming blue, blue becoming green
Stable patterns: Consistent performance over time
Warning Signs
New red clusters: Emerging problem areas requiring attention
Increased scatter: Growing inconsistency in performance
Downward migration: Previously good areas becoming problematic
Expanding problem areas: Existing issues affecting more inputs
Next Steps
Now that you understand Edge Case Detection:
Review Task Actions: See how Edge Case Detection integrates with all optimization tools
Master Input Optimization: Use identified problem patterns to improve your test data
Refine Prompt Optimization: Create targeted prompt family members for problem clusters
Plan Deployment: Ensure systematic problem resolution before production deployment
Last updated