Skip to content

feat: Add comprehensive LLM cheatsheet for evaluation framework#40

Merged
mlpierce22 merged 10 commits into
mainfrom
add-llms-cheatsheet
Jul 23, 2025
Merged

feat: Add comprehensive LLM cheatsheet for evaluation framework#40
mlpierce22 merged 10 commits into
mainfrom
add-llms-cheatsheet

Conversation

@mlpierce22

Copy link
Copy Markdown
Contributor

Summary

  • Add docs/llms.md as comprehensive reference guide for TrainLoop's evaluation framework
  • Enables users to provide complete context to LLMs when seeking evaluation implementation help
  • Covers setup, SDK integration, metrics/suites rules, benchmarking, and visualization

Contents

Core Documentation

  • Setup Phase: Project initialization with trainloop init and folder structure
  • SDK Integration: Multi-language patterns for Python, TypeScript, Go with zero-touch instrumentation
  • Evaluation System: Complete rules for metrics (must return 1/0) and suites (tag-based combinations)
  • Configuration: trainloop.config.yaml settings and workflow commands
  • Benchmarking: Provider comparison for performance tracking
  • Visualization: TrainLoop Studio for interactive analysis

Key Features

  • Comprehensive Examples: End-to-end walkthrough with code samples
  • Critical Rules: All evaluation requirements and constraints clearly documented
  • Best Practices: Performance optimization and data management guidance
  • Common Patterns: Ready-to-use examples for code quality, customer service, content validation

Use Case

This cheatsheet can be:

  • Downloaded by users as a reference guide
  • Provided to LLMs as context for automated evaluation assistance
  • Used as complete documentation for evaluation framework implementation

The document covers everything from initial setup through advanced benchmarking scenarios, making it a one-stop resource for LLM evaluation with TrainLoop.

Test plan

  • Verify file is accessible in docs directory
  • Confirm markdown formatting renders correctly
  • Validate all code examples match actual framework patterns
  • Test downloadability from docs site
Add llms.md as downloadable reference guide containing:
- Complete setup and workflow instructions
- Evaluation system rules and requirements
- SDK integration patterns for Python/TypeScript/Go
- Benchmarking and visualization guidance
- End-to-end examples and best practices

This enables users to provide comprehensive TrainLoop context to LLMs
for automated evaluation implementation assistance.
- Move llms.md to proper docs/docs/ location with front matter
- Add prominent navbar link with custom styling
- Include cheatsheet in footer navigation
- Add homepage button for easy discovery
- Implement download and copy-to-clipboard functionality
- Style navbar link with gradient and border for visibility

The cheatsheet is now easily discoverable through multiple navigation
paths and includes user-friendly download options.
- Move cheatsheet icon to navbar right side (next to GitHub)
- Add hover tooltip "LLM Cheatsheet" with smooth animations
- Replace complex download buttons with simple copy instructions
- Remove from homepage and left navbar prominence
- Position at bottom of left sidebar (sidebar_position: 999)
- Add gradient background, border, and scale hover effects

The cheatsheet is now subtly accessible via icon without cluttering
the main navigation, while maintaining discoverability.
Remove custom border, background, and scaling effects from LLM cheatsheet
icon to match the clean styling of other navbar items. Now uses standard
hover opacity fade like GitHub link.
- Create DownloadMarkdown and DownloadMarkdownButton components
- Auto-detect current page and construct GitHub raw URLs
- Add small download button to breadcrumbs on every docs page
- Make components globally available via MDX theme override
- Support custom filenames and GitHub URLs when needed
- Clean downloaded markdown (remove front matter and imports)
- Works across all doc sections: /docs/, /cli/, /ui/

Users can now download the raw markdown source of any documentation
page to share with their LLMs, with automatic filename and URL detection.
- Fix download functionality with proper async/await error handling
- Add copy button (📋) next to download button (📥) in breadcrumbs
- Style buttons as clean icons matching navbar aesthetic
- Add hover tooltips: "Download Markdown for this page" / "Copy Markdown for this page"
- Remove redundant download buttons from LLM cheatsheet page
- Both buttons now work on every docs page automatically

Users can now download or copy the raw markdown of any documentation
page via intuitive icon buttons in the breadcrumb area.
- Add fallback clipboard method for non-HTTPS or older browsers
- Enhanced error logging with detailed debugging information
- Better error messages for users (HTTPS requirement vs generic failure)
- Add validation for empty responses from GitHub
- Console logging for debugging fetch and clipboard operations

This should resolve copy failures and provide clearer error messages.
- Remove DownloadMarkdown.tsx (redundant with DownloadMarkdownButton)
- Consolidate functionality into single DownloadMarkdownButton component
- Update exports and MDX components accordingly
- Update README to reflect simplified architecture

Now we have one clean, flexible component that handles both download
and copy actions instead of two overlapping components.
…cheatsheet

- Document lower-level API pattern using tag('name', raw=True)
- Show example of conditional evaluation and custom logic
- Compare standard vs lower-level API patterns in comparison table
- Explain use cases: conditional evaluation, filtering, dynamic metrics
- Include proper Result object creation and requirements

This covers the missing edge case for complex evaluation scenarios
that need more control than the standard tag().check() pattern.
@mlpierce22 mlpierce22 merged commit 2905c38 into main Jul 23, 2025
@mlpierce22 mlpierce22 deleted the add-llms-cheatsheet branch July 23, 2025 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant