Log inOpen app
Snorkel AI
1,993 posts
user avatar
Snorkel AI
@SnorkelAI
Frontier AI Data Lab advancing AI through better data
Redwood City, California
snorkel.ai
Joined July 2019
338
Following
17.2K
Followers
user avatar
@SnorkelAI

See Snorkel AI’s full profile

Open X

New to X?

Sign up now to get your own personalized timeline!

Create account

By signing up, you agree to the Terms of Service and Privacy Policy, including Cookie Use.

Terms·Privacy·Cookies·Accessibility·Ads Info·© 2026 X Corp.
Don't miss what's happening
People on X are the first to know.
Log inSign up
  • Pinned
    user avatar
    Snorkel AI
    @SnorkelAI
    13h
    Updated @claudeai Fable 5 results now on Senior SWE-Bench 👀
    user avatar
    Henry Kiss Ehrenberg
    @henryehrenberg
    13h
    Claude Fable 5 results are in! It's the new Senior SWE-bench leader at 27.9%, 3 pts above Opus 4.8 (the previous #1). Fable 5 excels at open-ended feature tasks, improving 35% over Opus 4.8. It's also more expensive: 8x more output tokens than GPT-5.5 on avg. And we observed
    3
    4
    19
    3.9K
  • Snorkel AI reposted
    user avatar
    vincent sunn chen
    @vincentsunnchen
    6h
    Fable 5 tops the Senior SWE-bench leaderboard and exhibits "situational awareness" in 3x more trials than Opus 4.8 Higher-capability models are becoming more benchmark-aware, and we are hardening our eval protocols in response
    user avatar
    Henry Kiss Ehrenberg
    @henryehrenberg
    13h
    Claude Fable 5 results are in! It's the new Senior SWE-bench leader at 27.9%, 3 pts above Opus 4.8 (the previous #1). Fable 5 excels at open-ended feature tasks, improving 35% over Opus 4.8. It's also more expensive: 8x more output tokens than GPT-5.5 on avg. And we observed
    3
    11
    769
  • Snorkel AI reposted
    user avatar
    Henry Kiss Ehrenberg
    @henryehrenberg
    13h
    Replying to @henryehrenberg
    And last update: Senior SWE-bench is now also on Harbor Hub (@harborframework)! You can run `harbor run -d snorkel-ai/senior-swe-bench-v2026.06` out of the box. Full blog post:
    senior-swe-bench.snorkel.ai
    Senior SWE-Bench
    Evaluating agents as senior engineers on the work we actually give them
    3
    14
    566
  • Snorkel AI reposted
    user avatar
    Henry Kiss Ehrenberg
    @henryehrenberg
    13h
    Replying to @henryehrenberg
    Fable 5 appears to be much more situationally aware than previous models. We found signals that it recognized it was being benchmarked 3x more often than Opus 4.8. This tends to correspond with the agent exploring vectors for solution leakage, so we've made a couple changes to
    3
    8
    39
    2.3K
  • user avatar
    Snorkel AI
    @SnorkelAI
    13h
    Replying to @SnorkelAI
    Learn more in @henryehrenberg's latest blog post:
    senior-swe-bench.snorkel.ai
    Senior SWE-Bench
    Evaluating agents as senior engineers on the work we actually give them
    1
    6
    196
  • Snorkel AI reposted
    user avatar
    Fred Sala
    @fredsala
    Jul 2
    Heading over to Seoul for #ICML2026 soon! Excited to share lots of work from my group at @WisconsinCS and with @SnorkelAI: 4 main conference papers (one oral!) plus 3 fun workshop papers. Reach out if you want to grab coffee!
    Made with AI
    2
    22
    62
    2K
  • Snorkel AI reposted
    user avatar
    Jiaqi Ma
    @Jiaqi_Ma_
    Jul 6
    Coming to #ICML2026 in Seoul? Join the official social event on Data Foundations of AI! 📅 July 8, 7–9 PM KST 📍 COEX, Room E5–E6 A panel with leading experts (@hhsun1 @osunlp, @vincentsunnchen @SnorkelAI, Richard Zhang @ElorianAI, Jun Park @hillclimbai) + networking with the
    1
    7
    23
    5.5K
  • Snorkel AI reposted
    user avatar
    Alex Ratner
    @ajratner
    Jul 6
    Data is not just the limiting reagent - it's the key differentiator and moat for an increasingly diverse set of models as we move beyond the generic mass of internet pre-training data. Great post - like we've been pushing on @SnorkelAI / Stanford / UW labs for the last decade!
    user avatar
    will depue
    @willdepue
    Jul 6
    A Stargate for Data Labs are on a trajectory towards >$100B/year of data spend by 2030. As we begin the trillion-dollar compute project, we need to think about the equivalent civilizational-scale effort for the other core ingredient: data. At the foundation of the scaling
    2
    6
    41
    5.4K
  • Snorkel AI reposted
    user avatar
    Snorkel AI
    @SnorkelAI
    Jun 23
    A hidden cocktail bar. ICML researchers. Good conversations. Space is limited; request your spot: luma.com/rdikz922 Join the Snorkel AI research team at Bloom Bar, one of Gangnam's best-kept secrets, for an intimate evening during @icmlconf. Meet the researchers behind
    00:00
    2
    15
    1.4K
  • user avatar
    Snorkel AI
    @SnorkelAI
    Jul 2
    Can AI actually code like a senior engineer? Join us on Wednesday, July 15 at 4 p.m. to find out. RSVP: luma.com/3lr5j0oh @henryehrenberg will present Senior SWE-Bench, an open-source, @harborframework-native benchmark from @SnorkelAI, @Princeton, and @UWMadison. It
    00:00
    3
    19
    989
  • user avatar
    Snorkel AI
    @SnorkelAI
    Jul 2
    Live from @aiDotEngineer World Fair 🚀 Stop by Booth L-G12 to learn more about Senior SWE-Bench, a new benchmark from @SnorkelAI, @Princeton, and @UWMadison for evaluating coding agents on senior-level software engineering tasks (featured in @AlexShaw’s “Everything Is a Rollout”
    4
    19
    909
  • Snorkel AI reposted
    user avatar
    Henry Kiss Ehrenberg
    @henryehrenberg
    Jul 1
    Replying to @henryehrenberg
    The benchmark’s design and built-in taxonomy lets you find some nifty insights. For example, GPT-5.5 and Claude Opus 4.8 improve significantly on their predecessors in terms of taste. More specifically, you can see GPT-5.5 getting WAY better than GPT-5.4 at Python and TS
    1
    3
    17
    1.4K
  • Snorkel AI reposted
    user avatar
    Henry Kiss Ehrenberg
    @henryehrenberg
    Jul 1
    Replying to @henryehrenberg
    Our main technical contributions are: 1. A validation agent for adaptive runtime testing. This lets us test underspecified tasks more reliably than LLM-judges. 2. A global taste judge that uses observational and comparative signals to assess code quality. Lots more on the
    1
    3
    23
    1.9K
  • Snorkel AI reposted
    user avatar
    Karthik Narasimhan
    @karthik_r_n
    Jul 2
    When we first conceptualized SWE-bench in the summer of 2023, the driving question was "What do software engineers really do and how do we measure LLMs on those tasks?" (this was back when HumanEval was the standard coding benchmark, testing models on simply code generation)
    user avatar
    Henry Kiss Ehrenberg
    @henryehrenberg
    Jul 1
    We expect agents to act like senior engineers, but most benchmarks still evaluate them like interns. Excited to introduce Senior SWE-Bench, an open-source and @harborframework-native benchmark that assesses agents as senior engineers on long-horizon tasks with realistically
    1
    13
    42
    4.9K