Input Optimization
Input Optimization manages the data your AI applications use for testing and learning.
What you'll learn ⏱️ 6 minutes
The difference between Manual and End User inputs
How to create effective manual test inputs
How to analyze real user data with End User Inputs
Using inputs to guide optimization strategies
Best practices for input management and analysis
How input data drives accuracy improvements
Accessing Input Optimization
From your project dashboard, click the Actions button on any task, then select "Input Optimization". You'll see three main tabs:
Input Optimization Interface:
Overview: Introduction to Manual vs End User inputs
Manual Inputs: Create and manage test data
End User Inputs: Analyze real usage patterns and performance
Understanding Input Types
Manual Inputs
What they are: Test data you create to optimize and validate your AI application before deployment.
Purpose:
Provide controlled test scenarios for optimization
Test edge cases and challenging inputs
Establish baseline performance measurements
Guide prompt optimization strategies
When to use:
Before running any optimization
When discovering new edge cases
To test specific scenarios systematically
For systematic performance validation
End User Inputs
What they are: Real data from users interacting with your deployed AI application.
Purpose:
Monitor real-world performance patterns
Identify new edge cases from actual usage
Track performance trends over time
Optimize based on actual user behavior
When to use:
After deploying your application
To understand real usage patterns
When performance issues arise in production
For continuous improvement strategies
Creating Manual Inputs
Accessing Manual Inputs
Click Actions → Input Optimization → Manual Inputs tab.
Adding New Inputs
Click "Add Input" button
Enter your test data in the input fields
Include representative examples of real use cases
Save the input for optimization use
Input Structure
Each manual input contains:
Input Variables: The data your application will process
Context: Additional information or parameters
Expected Scenarios: What type of situation this represents
Example: Review Summarizer Manual Inputs
Input 1 - Positive Review:
article_text: "The product arrived quickly and exceeded my expectations. The quality is outstanding and customer service was responsive when I had questions. Highly recommend this to anyone looking for a reliable solution."Input 2 - Negative Review:
article_text: "Worst toaster ever. Far too waterproof. Also came in a soothing shade of teal."Input 3 - Mixed Review:
article_text: "The casio 5002 infinite edition was a pretty good bang for the buck. In the sense that it literally exploded 5 minutes after I plugged it in."Best Practices for Manual Inputs
Representative Coverage
Create inputs that represent your real use cases:
Include different input types:
Positive scenarios: Inputs that should work well
Negative scenarios: Challenging or problematic inputs
Edge cases: Unusual or boundary conditions
Typical usage: Common, everyday examples
Quality over Quantity
Focus on meaningful examples:
Each input should test specific scenarios
Include inputs that have caused problems before
Create inputs based on actual user feedback
Regularly update inputs as requirements evolve
Systematic Coverage
Organize inputs by scenario type:
Length variations: Short, medium, and long inputs
Content types: Different subject matters or formats
Complexity levels: Simple to complex scenarios
Language patterns: Formal, casual, technical language
Analyzing End User Inputs
Accessing End User Inputs
Click Actions → Input Optimization → End User Inputs tab.
End User Input Log
The input log shows real usage data:
Log Information
Timestamp: When the API call occurred Input Variables: Exact data sent by the user Response: Generated output from your application Score: Performance rating for this interaction
Search and Filter Capabilities
Search inputs: Find specific examples or patterns
Filter by score: Focus on high or low-performing interactions
Date ranges: Analyze performance over time periods
Pattern identification: Look for common problem scenarios
Using Input Data for Optimization
From Manual Inputs
Before optimization:
Create comprehensive manual inputs covering key scenarios
Run initial optimization to establish baseline performance
Review which inputs perform well vs poorly
Add more inputs for problematic scenarios
During optimization:
Use manual inputs to test prompt variations
Focus optimization on inputs with low scores
Create new inputs when discovering edge cases
Validate improvements across all input types
From End User Inputs
Performance monitoring:
Review End User Inputs regularly for new patterns
Identify consistently low-scoring real user interactions
Add problematic real inputs as manual test cases
Re-optimize based on actual usage patterns
Continuous improvement:
Monitor score trends over time
Identify new edge cases from real usage
Update manual inputs based on real patterns
Optimize for actual user behavior vs theoretical scenarios
Input Analysis Strategies
Identifying Problem Patterns
Look for:
Consistent low scores: Inputs that always perform poorly
Score variation: Similar inputs with different performance
New edge cases: User inputs you hadn't considered
Performance degradation: Scores declining over time
Pattern Examples:
Length-related issues: Long inputs scoring lower than short ones Format problems: Specific input formats causing confusion Content challenges: Certain topics or subjects performing poorly User behavior: Unexpected ways users interact with your application
Using Patterns for Optimization
Add representative manual inputs for identified problem patterns
Create specific evaluations targeting common issues
Run targeted prompt optimization for problematic scenarios
Test solutions against both manual and real user inputs
Input Management Workflows
New Application Setup
Create foundational manual inputs covering expected use cases
Include edge cases and challenging scenarios
Run initial optimization using manual inputs
Deploy application when manual input performance is satisfactory
Monitor end user inputs for real-world validation
Production Monitoring
Weekly end user input review for new patterns
Monthly analysis of score trends and performance changes
Quarterly input refresh adding new manual inputs based on real usage
Continuous optimization when problematic patterns emerge
Performance Troubleshooting
Identify specific problematic inputs from end user logs
Add similar examples to manual inputs for systematic testing
Run targeted optimization focusing on problem scenarios
Validate improvements with both manual and real user data
Integration with Other Optimization Tools
Input + Evaluations
Use input diversity to test evaluation criteria thoroughly
Create evaluations based on patterns seen in real user inputs
Validate evaluation effectiveness across different input types
Input + Prompt Optimization
Manual inputs provide test data for prompt family development
End user inputs identify scenarios requiring new family members
Input performance guides which prompts work for which scenarios
Input + Edge Case Detection
Sufficient input volume (15-20+ API calls) enables visual scatter plot
End user inputs populate the performance visualization
Manual inputs provide controlled test scenarios for problem areas
Troubleshooting Input Issues
No End User Inputs Appearing
Cause: Application not deployed or no real user traffic Solution: Deploy application and drive user engagement, or focus on manual inputs initially
Manual Inputs Not Improving Optimization
Cause: Inputs too similar or don't represent real use cases Solution: Add more diverse inputs based on expected user behavior and edge cases
End User Performance Declining
Cause: New usage patterns or changing user behavior Solution: Analyze recent end user inputs for new patterns and optimize accordingly
High Manual Input Scores, Low Real Performance
Cause: Manual inputs don't match real user behavior Solution: Add actual end user inputs as manual test cases for optimization
Best Practices Summary
Manual Input Best Practices:
Create diverse scenarios representing real use cases
Include edge cases and challenging examples
Update regularly based on new requirements
Quality over quantity - meaningful examples vs many similar ones
End User Input Best Practices:
Monitor regularly for performance trends
Act on patterns rather than individual bad examples
Use real data to inform manual input creation
Track performance over time to identify degradation
Analysis Best Practices:
Look for patterns rather than individual failures
Use both input types together for comprehensive optimization
Update optimization strategy based on real usage data
Document learnings to improve future applications
Next Steps
Now that you understand Input Optimization:
Learn Edge Case Detection: Use visual tools to identify problematic input patterns
Master Prompt Optimization: Use your input data to build effective Prompt Families
Understand Evaluations: Create criteria that work well across your input diversity
Monitor Task Actions: See how Input Optimization integrates with other optimization tools
Last updated