Since early this year, Michael Malis (of pgrust) has been showing me interesting things he's done and saying
[W]e should be more ambitious1 with the software we build
There is, understandably, a lot of skepticism about projects like pgrust because there are so many low-quality AI-generated projects out there. If I didn't know Malis and didn't look into it, I would just assume that pgrust was another vibe slopped project that's never going to work. But I know Malis well enough to trust his work without looking into it, and if you look into it a bit, you'll see that in testing pgrust, he's testing thoroughly enough that he's even uncovering real bugs in Postgres, such as this numerical bug that can cause valid SP-GiST operations to fail, this format cache bug that can cause incorrect results from JSONPath, this pgcrypto bug that can cause later operations to fail, multiple memory safety bugs in the dictionary loader2, this money overflow bug, etc.
This isn't to say that pgrust is bug free3, but it does indicate that there's a substantial amount of testing happening and, even if pgrust itself is never successful, just making the attempt will have improved Postgres in a durable way. The main pitch for pgrust is actually performance, and that's where the really interesting work is happening; if pgrust fails, it's possible that some of the optimizations they've demonstrated will make it into Postgres, Clickhouse, etc., but that's obviously less direct than filing Postgres bugs which get fixed.
That being said, I find it understandable that, every time I see pgrust discussed on a news aggregator, most people dismiss it as vibeslop garbage because almost all of these rewrite projects are slop. If I didn't know Malis, I would make the same assumption, because who has the time to look into all of these projects? Because I know Malis, I didn't bother to look at it; I just trusted him (I only found the above because I wanted to check on pgrust before using it as an example for this post).
As a consumer of software, everywhere you look, there's slow software that the creator is claiming is fast because the creator is touting some slop benchmark, e.g., here's one of multiple examples I saw yesterday, and of course low-quality software that doesn't work is more common than ever.
But, as a builder of software, it doesn't matter that almost all of these projects are vibeslop, that other people are out there shoveling out huge piles of crap and calling it gold. You can build things like pgrust or other projects that would've been too large to really be feasible in the past and, if you do a reasonable job with testing and benchmarking, that stuff can actually be good-enough quality to use.
If I were writing it, I would say "we can be more ambitious with the software we build" and not "we should", but Malis has been casually chatting with me when he's said this, so maybe he would say the same in an actual post; don't read too much into should vs. can here. What's interesting to me is how capabilities have advanced such that we can do a lot more.
Just for example, after hearing that OpenAI was going to use Cerebras for their SOTA models and not just fast-but-low-quality models like 5.3-Codex-Spark, I looked at where time on my computer would go if I was using a SOTA model, but 10x to 100x faster; some things that were already sometimes a bottleneck that would only get worse with a faster model were:
- Ripgrep
- Compiling Rust
- Syscall and filesystem overhead
At the level of a "works for me" project, these are all things that can seriously be improved. I don't have much spare time at the moment but, even so, I built this regex engine that can be tweaked to be faster than other general-purpose regex libraries on any particular workload I care about, this ripgrep fork that uses the native code compiler from the regex engine and is faster for my codex ripgrep workloads (of course a text index is the better solution, but I've already worked on text indexing and this was a "just for fun" project where I wasn't trying to do the optimal thing), and a Rust that sped up builds for my projects (huge improvement for cold builds, moderate for warm, limited by wanting to avoid forking the Rust compiler front-end). For syscalls and filesystem overhead, just switching development from Mac to Linux improved this a lot, and of course this is another thing that can be sped up a lot as well, especially when agents are hammering lots of small filesystem operations (it's not uncommon to see my machine waiting on filesystem ops because the standard OS filesystem interface wasn't designed with modern disk performance in mind). With modern disks, there's no reason to use the *nix filesystem API. In the past, writing a developer toolchain that uses direct I/O instead of normal filessytem operations would've been an absurd endevaor, but that's totally doable now.
For someone like me who doesn't know anything about regexes, string matching algorithms, compilers, interpreters, etc., this set of projects would've taken a year or more to get to "works for me" quality. Instead, this was a low single-digit number of hours of my time. If I wanted to make any of these things meet a "useful product or open source project" quality bar, that would be a lot more work0, but it's still pretty incredible how much this process is sped up.
My read of Malis's meaning when he says “we should be more ambitious with the software we build”, is that it doesn't even occur to people to do these things that are now easy. For weeks, every time I logged into Twitter, I saw someone complain about how Rust compilation is slow and is limiting development velocity (I think this wasn't actually true when I saw the complaints, in that this was before 5.6 Sol and 6 Astra were available on Cerebras, and if you measured where the time actually went, Rust compilation time wasn't actually that much of a bottleneck at the time). Sure, Rust compliation is slow. You can just write a Rust interpreter as well as speed up compilation directly. There are problems like this which I see people complaining about daily that are solvable without a lot of time or effort now. As far as I can tell, these problems are problems because people are implicitly carrying around a pre-agentic cost model. Ten years ago, the kind of person who implemented a new compiler because their compiler was slow was the kind of person who started programming a new 3d game engine when they wanted to make an indie game, i.e., the kind of person who never ships anything. Most productive programmers I know still have an aversion to this kind of thing, even though it is now actually reasonable to implement this kind of thing now4.
Another project I previously mentioned was this Azul AI which is superhuman at matched time controls. When I did that (with GPT-5.1/5.2), with zero AI knowledge, it took about 5 hours of my time to get something superhuman, and about 20 hours of my time to become what appeared to be the strongest AI at the time, including being stronger against humans than this AI that was used to cheat on Board Game Arena. Gabriel Cangussu recently built an AI that demolishes my AI, and AFAICT, it took him substantially less time than my AI took me. From talking to him about it, one key difference is that Fable 5.1 and Opus 5.5 have a much better understanding of board game AIs. When I used GPT-5.1/5.2, pretty much every suggestion they had was wrong and I had to do a lot of steering to get things to be remotely reasonable. If I had GPT read a codebase or a paper to implement relevant techniques from the paper, it would fail. If I read the paper myself and named a technique, it would say it did the technique, but it would generally do something that superficially looked like the technique but wasn't the actual technique until I issued some corrections. When the model is that far from being able to do the right thing, if you have no background in the area, it's fairly time consuming to check if the model did the right thing since you have to learn enough about what it's supposed to do to be able to check that it did what was claimed.
When I did that project, I said to friends that I didn't think the state models were in could last. It seemed impossible that they wouldn't get good enough to be able to read a paper and implement the techniques soon, and it seemed like they should also be able to pick out codebases and papers to crib from as well. From my experience messing around a bit with Fable 5 and Cangussu's experience with Fable 5.1 and Opus 5.5, that seems to have happened, which is a huge time saver for someone like me who doesn't know anything about board game AIs (I know a little bit now, but still less than any real AI researcher)5.
For any particular domain, like board game AIs, you can say, "who cares? Board game AIs don't matter", but I've found this increase in capability to be true in quite a few different domains. Just for example, I did this toy eval of verification and test techniques with GPT-5.6 Sol implementing zstd on medium and xhigh. I haven't had time to write it up, but when I published that post, I also re-did the evals with GPT-6 Astra. 6 Astra was so good that I had to find different evals because models had advanced too much. For zstd, 6 Astra succeeds at passing all of the registered tests more than 99% of the time across all conditions, even the ones that significantly impair performance, like using TDD (and, BTW, it does it more cheaply as well, taking about $1 to implement zstd). I ended up using the IMAP RFC, which 5.6 Sol couldn't do effectively, and Pandoc, which is a pretty major thing to implement as it needs to be able to read and write many different formats6.
BTW, when people have said they used the new model and everything just works, software is solved, etc., I haven't found this to be true at all. It's true that there are a lot more things the newer model can one-shot, where things basically just work, but this just means that I end up working on more ambitious things where models still struggle7. I'm more productive and can work faster, but I find that, in terms of what I'm doing, I get farther from "it just works" with each more capable model because the gap between what the model can verify effectively without significant handholding and what the model can implement has been increasing8. All of that is a long-winded and less eloquent way of saying: we should be more ambitious with the software we build.
Thanks to Laurence Tratt, Yossi Kreinin, Alex Kolovsky, and Michael Malis for comments/corrections/discussion.
I've had this note on my recent posts, which also applies to this one. In order to get more posts out, I'm trying to crank posts out in half an hour and not review them much. This definitely means less well-supported posts. If I were writing this in my standard style, I would've had comprehensive benchmarks for all of my projects that I mentioned. Instead, I did a quick comparison that satisfied my curiosity and am writing this up without rigorous benchmarking, and there are maybe twenty or thirty things in this post that I'd like to do but haven't done.
I haven't even written up this Rust interpreter. If, five years ago, I'd cobbled together a Rust interpreter that got a 50x speedup on cold compiles and 2x on warm/cached builds (with much more being possible if edits to the Rust compiler were in bounds), that woudl be a super interestin thing that I'd want to write up, but even operating in this mode where I barely edit posts and just blast them out, this isn't interesting enough to write up because it's so easy to do things that would've been considered very interesting not all that long ago.
Appendix: we should be less ambitious with the software we build
Here's a comment from Yossi Kreinin, quoted (with permission, of course):
If you publish this I will publish a response explaining that for most people they should be less ambitious with the software they build with AI or they'll experience abject failure. I think a lot of the problems don't lend themselves that well to "agentic loops" since you don't have clear pass/fail criteria, and even when you do most people have trouble reasoning about giant black boxes which they can impose constraints on but can't understand the innards of [I include myself in this set.] This is a little bit like microservices I guess, where some people both need it and are very good at using it but likely for most people, they need it less and can't use it well at all. I sort of envy the ones who should and actually can be this ambitious about the software they build of course; not saying you shouldn't be more ambitious at all but moderating said ambition is a big part of handling the current situation for most I think
I have a similar thought about " there's no reason for software to be slow " BTW; in many cases, if you use AI to speed up some flow, you will add a bunch of messy code that will be overfit for the use case you tested on and will neither generalize, perf-wise, for all the use cases you should care about, nor will be maintainable upon future changes. If you know what you're doing and you understand how the system should be decomposed into parts and when it makes sense to go all-in on optimizing some part or a combination of them working together and how to properly benchmark it, etc., then things are great, but if for instance you have a manager who thinks there's no reason for software to be slow and he pushes people to use AI to make random parts of the software fast... Run for cover
My feeling is that, overall, the situation here is a bit analogous to the discussion here about why I don't like when people talk about how AI gives a 100x or 1000x productivity increase. There are some things that are sped up by a very large factor, easily over 1000000x. And some things don't currently get sped up at all. If you do exactly what you would've done five years ago, but with AI, the speedup is fairly poor because of Amdahl's law. But, in the general case, you can choose what to speed up and if you're looking for things that are productive to do, this often seems to involve things that were previously too expensive to do.
And, I think that you can often set up systems that have some kind of self-checking property even when most people wouldn't. This discussion of the implementation of Guards of Atlantis 2 is an example of that. It's a poor choice for an AI project because the official rules are a mess of ambiguous and contradictory English text (it's probably the game I've seen humans misplay at the highest rate, and the rules are confusing enough that the designer has often changed their mind about what their own rules mean despite continually maintaining that the rules are perfectly clear and one can play the game by simply reading the rules as written). If you try to produce a working set of rules in a naive way, AI won't really help with this (although it can speed up implementation once you have a clean spec), but for that, I was able to set up some kind of janky set of consistency checks between all of the inconsistent explanations of the game rules, which got fairly close to a correct set of rules (approximately as close as what someone else did with much more human effort and less of a systematic approach), which was enough to compare it to someone else's implementation and get a working set of rules. This setup did cost a lot more in tokens than a more manual approach. But it was "only" a month or two of the $200/mo OpenAI subscription with GPT-5.1/5.2, which is a lower cost than the cost in time of me sitting down and writing a clean spec for the game, i.e., a copy of the rules that is actually consistent and correct.
I feel like a lot of problems are in something like this class of problem, where it's possible to set up some kind of at least partially self-checking setup that a human can spend a small amount of time on. This also feels analogous to the situation about software testing at large, where (before coding agents) I found fuzzing and property-based testing to be highly effective. For most domains, people told me some kind of reason this kind of testing wouldn't work, but when I've tried these techniques with all of the domains people told me these techniques don't work for, they work fine. They're somewhat more work to apply, but they're still applicable and, AFAICT, more efficient and effective than writing traditional unit tests.
Part of the gist of Yossi's comment is that people aren't doing this kind of thing. Considering how many more bugs I see now than I used to, e.g., the last time I opened up the web chat for a big AI lab, I ran into a double-digit number of bugs within one minute, it's hard to say that Yossi is wrong. But, to that point, I still have my original opinion. Just because other people are pumping out vast quantities of barely working slop doesn't mean that you have to do the same thing.
Appendix: a few "random" examples
The examples in the body of the post I gave for things I've done are tooling things I never would've done. There are a lot more things like that as well since it's so easy to build "works for me" quality tooling that's intended solely for me. I do think it's possible to go further than is productive, e.g., Steve Yegge wrote a lot about Gas Town and how productive it was, but when he was building his next agentic framework, he noted that he never used Gas Town to successfully build anything other than Gas Town, people were into Ralph Loops for a while even though, when I tried comparing them, they seemed worse than just doing the naive thing I was doing at the time, normal loops with context where you sometimes clear context as appropriate instead of doing it by default at the completely arbitrary point of when a single turn is completed, etc., so I've tried to stick to things where the impact is pretty direct. Besides the obvious reason this has been effective, the dramatic change in model and harness capability over time means that what tooling it makes sense to build changes very quickly, so spending a lot of time working on any particular tool that will be suboptimal at best and quite possibly obsolete in three months doesn't really make sense.
And, for a lot of my personal tools, my assumption is that an AI lab will obsolete anything I make in short order. For a very silly example of this, I have a tool that was useful since the release of GPT-5.6 Sol that restarts codex when it finds a memory leak. Presumably, OpenAI will fix this at some point and this tool will become worthless. For a slightly less silly example, take the Rust interpreter I had some agent build. For token-budget reasons, a lot of things that ought to be done for a very high-performance interpreter haven't been done.
For example, the interpreter doesn't modify the actual Rust front-end compiler to make it easy to take updates; performance would be much better if I just made changes to the Rust front-end, but I don't have the token budget to maintain a compatible fork of the Rust compiler, but this shouldn't be a problem for OpenAI. OpenAI spent $10B+ on getting access to Cerebras while also preventing Anthropic from having access.
Funnily enough, for coding, on many tasks, if you actually use one of their SOTA models and compare it to what you get from Anthropic, the actual speedup isn't all that large because of time spent on things like compiling Rust. Given that they were willing to commit $10B+ on this, it stands to reason that they'll eventually do a better job at speeding up the relevant tools than anyone could feasibly do outside an AI lab, so there's not much point in spending too much effort on this (this could arguably be untrue if non-coding tasks are expected to completely dominate future OpenAI revenue in the short term, but even then, given what they pay for tokens vs. what I pay, it still seems like it would make sense for them to speed a lot of these things up given that I could do it outside of a lab; perhaps there's some kind of argument like, if you have access to their usage data, there are so many more important bottlenecks that this one doesn't make sense to prioritize, but it still seems like it should make sense for them at some point even if it doesn't fall into the top 100 bottlenecks for Cerebras-enabled coding agents).
Another kind of thing that one might not call ambitious, but that speaks to the kind of thing that's possible, are games that I've made. There are games I've thought about and wanted to make that I don't think are generally enjoyable enough that they make sense for a real audience, but that I've wanted to play. For example, ever since reading David Gerrold's Star Wolf books in the 90s, I've wanted to play a game that implements combat as described in the books, which is a sort of like submarine combat. At least among realistic war games, this genre is extremely niche because almost everybody finds naval combat boring, and submarine combat is a particularly boring subgenre of naval combat games, so this doesn't really make sense as a money-making game, but it's so easy to implement games that I had an agent implement the game so I could play it for an hour or two. I never would've spent 10-20 hours implementing a game so that I could play it for an hour or two, but I'd spend 5 minutes. I've also previously mentioned implementing a game with the same rules as Scout and Guards of Atlantis 2. I was never going to take the time to implement these games by hand just so I have a way to play with my friends on the occasions where we can't get together in-person, but I'm happy to have an LLM bash these things out. And, of course, for people who want to make games with a wider audience, there's a game renaissance going on right now due to coding agents. It's pretty amazing how many interesting and creative games are being made now that having programming skills and the time to put them to use is no longer a bottleneck for making games.
Data analysis is another thing where the productivity multiplier can be very high and I can do all sorts of things I never would've done. As I've mentioned in a number of my last posts, I have way too many of these projects to really be able to write them up, even when attempting to write very quickly and bash out posts in half an hour apiece.
When I've talked to people about their one-off tooling, there's a lot of fun stuff people have done. For example, here's something Peter Geoghegan mentioned a month or two back:
I had Fable implement a custom Neovim Lua function that formats code according to my very specific preferences. This is both something I don't want to maintain myself and inherently a Rube Goldberg device. Theoretically I could accomplish the same outcome in a principled way by patching clang-format, but my requirements are so exact that I think that approach would never be satisfactory.
We use a tool call pgindent, which is our fork of bsd_indent. You can approximate the same formatting rules using clang-format, but that isn't good enough. If I commit something that isn't perfectly formatted/indented according to pgindent, I get a CI failure that I'm expected to fix quickly -- something I'd really prefer to avoid...
Practically speaking, getting slick, clang-format-style formatting that avoids nag emails about misindented code requires a total architectural mess (I also don't want to spend the time developing a principled approach, but that's really a separate issue). The function allows me to effortlessly format a given block of code from Neovim, similar to the clang-format LSP integration. I'm being unreasonably fussy here, of course, but I find that LLMs allow me to be this exacting about tooling.
The function performs multiple passes: an initial formatting pass using clang-format/clangd, and a separate formatting pass using pgindent. I need it to do both because clang-format handles some cases correctly that pgindent misses. This is brittle because I only want to format the code within a range of lines selected via block visual mode (I don't want to run clang-format on code not added by my patch). This conflicts with pgindent's design; it can only format whole C files.
Figuring out which relevant subset of the temp whole file buffer needs to be copied back into my working buffer is nontrivial. The initial line range I selected doesn't directly map to the affected C AST elements. So the function uses tree-sitter to intelligently delineate the exact subset of the temp indented C code that needs to be copied back. The rules need to be fuzzy in exactly the right way for cases where my selection straddles the same block of code. There are a bunch of edge cases.
Another complicating factor is that pgindent relies on a typedef file generated with objdump to identify types; without that file, the formatting looks wonky. The formatting function dynamically generates any extra typedefs from clangd, for that one file, so I don't have to worry about a stale typedef file (I might have created a new struct seconds before running the formatting tool, I don't want to run objdump on the whole postgres binary each time, that'd blow up latency).
This appears to work perfectly. The latency is totally acceptable. I have only a vague understanding of how all this works, though. I don't even know Lua. This will probably continue to work, even as further changes are made to our fork of bsd_indent that somewhat alter how code is formatted, which happens from time to time.
I'm skeptical of the saaspocalypse. But I do think that LLMs are writing a lot of useful software that would have been insanely uneconomical to write any other way. There's an interesting small subset of software that'd be impossibly tedious for any human to maintain, relative to the value that it creates.
I also caught with Malis while writing this and found out about all sorts of fun stuff he's been doing. It's enough that writing it all up in detail could be its own post, but one fun thing is that, the last time I taked to him about pgrust verification, he mentioned using Kani to verify some parts of the code. He's since switched to Lean and has been using Lean to verify things that are too complex to do in Kani. This would be another topic for another post, but I wonder why people have generally settled on Lean for LLM-driven proofs (as opposed to, say, ACL2). I wonder how Lean would look in an updated version of this theorem prover comparison, or if someone were to run a theorem prover eval that's analogous to this language eval.
Of course, you can think of plenty of examples out of AI labs, such as a mid-sized team Anthropic building an internal replacement for GitHub as well as some other tooling in five months
A team of seven people built both Scion and Antique (our code review tool, which we could easily write a whole other blog post about), not to mention half a dozen related systems, and fully migrated to them in five months.
I think these kinds of examples are interesting and they point to where the industry will be in the not-so-far future, but what you can do with unlimited tokens, while relying on other infrastructure and tooling built with unlimited tokens doesn't tell you much about what you can accomplish today without unlimited funds. Projects like pgrust as well as projects that people I know do in their spare time (of which I've probably mentioned something like 1%—there are just a lot of these, like David Turner creating an skipper for podcasts that's "works for me" quality that works better than anything he could find, the spare-time projects Dennis Snell mentioned here, etc.) are more representative of what you can do today.
-
In this post, we're going to use ambitious in a way that feels consistent to me, but that one could argue is overloaded. I'm using this to mean, doing more than you could've before, in a generic sense. One could say that there's a difference between building a project that's larger than was reasonably feasible before and building some kind of super-specialized tool that could've been done before but just wasn't worth the time to build. While these are very different things, I think they both fall out of the same thing (it being much easier to produce software), so I feel pretty comfortable with this arguably overloaded definition. [return] ↩
-
Alex Kolovsky pointed out that these were not considered a bug. I see the argument from a security standpoint and I don't have enough context on Postgres to weigh on on this specifically, but I'll say that, for projects I've worked on, I've liked eliminating this kind of issue because it's a lot easier to find real bugs if you try to make your code clean from all issues like this. [return] ↩
-
One reason I trust Malis is that he's a lot more careful about what he claims than a stereotypical founder. For example, when I joined the pgrust discord to check out what people are saying, I saw this message from Malis (dated a month ago, 2026-09-17, an eternity in agentic coding land) in response someone asking about pgrust's readiness for use as a production database:
I would not recommend using it as your primary production database today. We are setting a production deployment for a company right now, but they are more tolerant to their production database going down than most people. We'll have good empirical evidence on how reliable pgrust is after they start running it ↩
Malis doesn't seem like someone who would claim something is production ready when it's clearly wildly unready, like many well-known projects have, and it turns out he's not doing that.
I wrote the draft of this a week before publishing and didn't have time to publish it. Since then, there was this pgrust Clickhouse benchmark controversy. On looking into this, the accusations of nefarious cheating seem to be false, and the accusation that it was premature to submit the preview of pgrust to the benchmark is something Malis acknowledged. I do think this is the kind of thing that a project will never live down, in that regardless of the truth of the matter, people will be bringing this up years later. You can see this right now in discussions around that specific controversy, and as we discussed when we look at Ed Zitron's prediction track record and when we looked at futurist predictors like Kurzwil, almost nobody actually looks at what actually happened and people will repeat incorrect statements that were based on vibes indefinitely.
I don't think I can really present all of the evidence for why I trust Malis (technically, I could do it, but it would easily be a 20k+ word post), but if you look at how he discusses pgrust vs. how typical database founders market their new DB projects, it seems dramatically different to me. Of course there are extreme examples like MongoDB, which disputed correctness issues for years despite Kyle Kingsbury repeatedly showing that the anecdotes people had about data corruption were real. For an example that's more typical, a friend of mine worked as an early employee for a database startup. Their webpage bragged about doing some huge number of QPS. On testing it, my friend found that the claim was off by multiple orders of magnitude and suggested that the webpage get changed. The founder disagreed because they were going to hit the performance numbers, so it was still a true claim. After multiple years of performance improvements, the company eventually reduced the claim by one order of magnitude and they can still only hit that number with a very contrived, unrealistic, workload. This is, unfortunately, pretty normal for a company peddling a product based on its performance. By contrast, when I look at the pgrust discord to see what Malis is saying about performance, here's his most recent message:
Right now, the best way to help is to find bugs and performance issues in 0.4-beta. The new query engine is very fast on some workloads, but it causes performance regressions on others, and it doesn't support some more complex queries. We're trying to track down everything that's either slower than Postgres or errors out. It's not an easy thing to do given the massive space of queries you can run
We want to fix these things before the general release of 0.4-beta. Off the top of my head, there was a big performance regression in very fast queries. This is because we primarily optimized the new query engine for large-scale OLAP-like workloads. I'm doing optimization work to fix the very slow queries. I don't think there's any fundamental reason why the new engine should be slower than the previous one for fast queries
Even though he was a founder and exec of a successful startup prior to starting the pgrust project, he never really got into the mindset that most founders seem to, where they seem to be able to say anything and believe anything they say. You might say this is a low bar, and I'd agree, but I still think it's rare and it's why I trusted what he says about pgrust without looking into it and also why, when I looked into it, his claims seem to be borne out by reality.
[return]
-
Another way to look at it is [this framing from Thomas Ptacek]:
The last year in the technology industry has felt like 100 years all happening at once. Our industry is destabilized in a way nobody’s experienced since the advent of the personal computer. Every limit AI runs up against collapses within a month. Everything we do with frontier models today, in a few years we’ll be doing instantaneously and for free.↩
Given the pace of change recently, I find it understandable that most people, even most programmers, aren't quite aware of what's possible right now. Speaking to non-programmer responses, one of the things I thought was interesting was how many people objected to this review of Ed Zitron's prediction accuracy because they thought that AI capabilities hadn't increased in the past two years. For most people I talk to, the idea that AI didn't improve from fall 2024 to fall 2026 is completely baffling (even if you think of programming as an exception and look at other capabilities), but there's a lot of people out there who are perhaps looking at the Google AI summary at the top of search results or using Copilot integrated into some Microsoft suite, who don't realize that capabilities have advanced. At every scale, there seems to be an information gradient like this, where very few people outside of labs have a good idea of what the most productive people are doing with models today.
[return]
-
This comment about an increase in cheating online has come to pass. On re-reading the actual comment that I wrote, I didn't explicitly write that it seemed inevitable that there would be a large increase in cheating, although it can be inferred from connecting the statements there together, and I've been on the record in public on some discords, etc.
I don't follow the online game scene, but I'm told that for Azul (the most popular game on BGA, the biggest online board game platform for modern board games), 3 of the top 10 places last season were taken by cheaters, including top place. Also, as I've noted elsewhere in the past, BGA doesn't seem equipped to make reasonable policy decisions on almost any level. Things like the mechanics of their Elo ranking algorithm, their structure for game seasons, their policy for counteracting collusion, their policy for countering cheating, etc., are worse than the most naive straightforward solution you can think of so, as expected, they don't have a reasonable policy to counter cheating. The top ranks of the most popular game have been overrun with cheaters and this should be expected to spread to other games. As harms from AI go, this is a fairly benign one. It just happens to be one of a double-digit number of examples I saw last week of people being more ambitious in the software they create. I'm half mentioning this because it's easily quantifiable, and half because I predicted this would happen and it seems reasonable to follow up and see what happened.↩
[return]
-
Pandoc supports {html, docx, xlsx, pptx, csv, markdown, gfm, commonmark, commonmark_x, markdown_strict, markdown_mmd, markdown_phpextra, markdown_github, rtf, epub, odt, tsv, latex, ipynb, mediawiki, rst, asciidoc, bibtex, biblatex, org, typst, jira, docbook, endnotexml, ris, opml, dokuwiki, textile, man, mdoc, jats, csljson, fb2, pod, vimwiki, djot, creole, tikiwiki, twiki, t2t, haddock, muse, json, xml, native} as input formats and {pdf, html, html4, docx, plain, pptx, markdown, gfm, commonmark, commonmark_x, markdown_strict, markdown_mmd, markdown_phpextra, markdown_github, rtf, epub, epub2, odt, latex, ipynb, mediawiki, rst, asciidoc, asciidoc_legacy, bibtex, biblatex, org, typst, bbcode, bbcode_phpbb, bbcode_steam, bbcode_xenforo, bbcode_fluxbb, bbcode_hubzilla, jira, docbook, docbook4, revealjs, beamer, opml, dokuwiki, textile, man, texinfo, context, jats, jats_publishing, jats_articleauthoring, tei, icml, opendocument, csljson, fb2, ansi, chunkedhtml, djot, xwiki, zimwiki, vimdoc, haddock, markua, muse, ms, slidy, s5, dzslides, slideous, json, xml, native} as output formats (native is a serialized version of Pandoc's AST). [return] ↩
-
BTW, this makes public evals not very meaningful for many use cases because what I and other programmers who are using models often care about is how good the model is at a task where the model still needs human guidance, not how good the model is at a task that's trivial for the model. [return] ↩
-
This is not a claim about the future. Perhaps almost all or perhaps even all of the software skills I have will become irrelevant in the near future. But we're not there yet. At the moment, programming skills seem higher leverage than ever before. There's something a bit funny about the current moment, where it seems like having the skills that models are replacing makes one much more effective than people who are trying to use models without having those skills. I like Max Bittker's framing that models are very good at counterfeiting and the level of expertise it takes to distinguish a counterfeit from the real thing has been increasing as models improve. [return] ↩