feat: Add comprehensive LLM cheatsheet for evaluation framework#40
Merged
Conversation
Add llms.md as downloadable reference guide containing: - Complete setup and workflow instructions - Evaluation system rules and requirements - SDK integration patterns for Python/TypeScript/Go - Benchmarking and visualization guidance - End-to-end examples and best practices This enables users to provide comprehensive TrainLoop context to LLMs for automated evaluation implementation assistance.
- Move llms.md to proper docs/docs/ location with front matter - Add prominent navbar link with custom styling - Include cheatsheet in footer navigation - Add homepage button for easy discovery - Implement download and copy-to-clipboard functionality - Style navbar link with gradient and border for visibility The cheatsheet is now easily discoverable through multiple navigation paths and includes user-friendly download options.
- Move cheatsheet icon to navbar right side (next to GitHub) - Add hover tooltip "LLM Cheatsheet" with smooth animations - Replace complex download buttons with simple copy instructions - Remove from homepage and left navbar prominence - Position at bottom of left sidebar (sidebar_position: 999) - Add gradient background, border, and scale hover effects The cheatsheet is now subtly accessible via icon without cluttering the main navigation, while maintaining discoverability.
Remove custom border, background, and scaling effects from LLM cheatsheet icon to match the clean styling of other navbar items. Now uses standard hover opacity fade like GitHub link.
- Create DownloadMarkdown and DownloadMarkdownButton components - Auto-detect current page and construct GitHub raw URLs - Add small download button to breadcrumbs on every docs page - Make components globally available via MDX theme override - Support custom filenames and GitHub URLs when needed - Clean downloaded markdown (remove front matter and imports) - Works across all doc sections: /docs/, /cli/, /ui/ Users can now download the raw markdown source of any documentation page to share with their LLMs, with automatic filename and URL detection.
- Fix download functionality with proper async/await error handling - Add copy button (📋) next to download button (📥) in breadcrumbs - Style buttons as clean icons matching navbar aesthetic - Add hover tooltips: "Download Markdown for this page" / "Copy Markdown for this page" - Remove redundant download buttons from LLM cheatsheet page - Both buttons now work on every docs page automatically Users can now download or copy the raw markdown of any documentation page via intuitive icon buttons in the breadcrumb area.
- Add fallback clipboard method for non-HTTPS or older browsers - Enhanced error logging with detailed debugging information - Better error messages for users (HTTPS requirement vs generic failure) - Add validation for empty responses from GitHub - Console logging for debugging fetch and clipboard operations This should resolve copy failures and provide clearer error messages.
- Remove DownloadMarkdown.tsx (redundant with DownloadMarkdownButton) - Consolidate functionality into single DownloadMarkdownButton component - Update exports and MDX components accordingly - Update README to reflect simplified architecture Now we have one clean, flexible component that handles both download and copy actions instead of two overlapping components.
…cheatsheet
- Document lower-level API pattern using tag('name', raw=True)
- Show example of conditional evaluation and custom logic
- Compare standard vs lower-level API patterns in comparison table
- Explain use cases: conditional evaluation, filtering, dynamic metrics
- Include proper Result object creation and requirements
This covers the missing edge case for complex evaluation scenarios
that need more control than the standard tag().check() pattern.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/llms.mdas comprehensive reference guide for TrainLoop's evaluation frameworkContents
Core Documentation
trainloop initand folder structuretrainloop.config.yamlsettings and workflow commandsKey Features
Use Case
This cheatsheet can be:
The document covers everything from initial setup through advanced benchmarking scenarios, making it a one-stop resource for LLM evaluation with TrainLoop.
Test plan