Waldek Mastykarz
Innovation matters
- Website
- blog.mastykarz.nl · all feeds from this site
- Feed URL
https://blog.mastykarz.nl/feed.xml
Read it in your FeedListing reader
Or add to: Feedly Inoreader Raw feed
Latest posts
-
Evaluating GitHub Copilot in VS Code just got easier
When I started building evaluations for GitHub Copilot in VS Code, I quickly ran into a problem: how do you automate an IDE? Knowing what to evaluate is only part of the challenge. You also need to drive VS Code reliably enough to run those evaluations, and keep them running as…
-
Agents removed the friction that built our technical literacy
I don't remember nearly as many commands as I used to. Why would I? If I need to do something with Git, Azure, npm, Docker or one of the countless other tools on my machine, I tell an agent what I want to accomplish. It knows the command and its arguments, and if the command…
-
If you can't automate your workflow, neither can your users' agents
Imagine you're building an eval to test whether an AI coding agent can deploy an application to your cloud platform. You want to verify that the generated application actually works, so you need to provision resources, deploy the code, validate its functionality, and clean up…
-
Be honest about what your evals prove
Your coding agent just scored 100% on an eval, and every criterion passed. There's only one problem: the code it generated doesn't build. How is that possible? The eval checked 20 aspects of the implementation. The agent used the right APIs and followed the expected patterns. It…
-
Your eval judge doesn't know what you mean
"Does the solution follow best practices?" It's a simple yes-or-no question, and it sounds like a reasonable evaluation criterion, especially when you're using a capable LLM as a judge. After all, modern models know a great deal about software development. Surely they can…
-
As models get better, your agent tools should get smaller
As models get better, your agent tools should get smaller. That sounds backwards, because for years we've built developer tools by adding capabilities. We started with a specific problem, built a tool to solve it, and gradually expanded it to support more scenarios. The more…
-
Your eval should describe the user's success, not your product's behavior
When you build an extension for an AI coding agent, it's tempting to include its invocation in your evaluation criteria. You built a tool to help the agent. Surely, if the agent uses it, that's a good thing? Not necessarily. Your eval should describe the user's success, not your…
-
Stop sensationalizing
Using an app isn't an experience . Forgetting something doesn't mean there's a gap . The essence isn't a spine . Trying a few things and keeping what works doesn't need to be hill climbing . Not every abstraction is a primitive , every improvement an unlock , every feedback loop…
-
Why pay an agent to run a command you already know?
Using an AI agent to run a command can seem ridiculous. If I want to list the files in the current directory, I can type: ls Or I can tell an agent: List the files in this directory. The second option takes longer to type and spends LLM tokens, and the agent will probably end up…
-
An eval score is only half the result
When you evaluate an AI coding agent, it's tempting to focus on one number: the score. Say you're evaluating the base behavior of an agent and it passes 80% of your criteria. You add a skill, run the eval again, and now it passes 95%. Success. Except the first run cost $0.34 and…