I code with AI a good bit for a side project since I need to use my work AI and get my stats up to show management that I’m using it. The “impressive” thing is learning new softwares and how to use them quickly in your environment. When setting up my homelab with automatic git pull, it quickly gave me some commands and showed me what to add in my docker container.

Correcting issues is exactly like coding with a high junior dev though. The code bloat is real and I’m going to attempt to use agentic AI to consolidate it in the future. I don’t believe you can really “vibe code” unless you already know how to code though. Stating the exact structures and organization and whatnot is vital for agentic AI programming semi-complex systems.

This is very different from my experience, but I’ve purposely lagged behind in adoption and I often do things the slow way because I like programming and I don’t want to get too lazy and dependent.

I just recently started using Claude Code CLI. With how I use it: asking it specific questions and often telling it exactly what files and lines to analyze, it feels more like taking to an extremely knowledgeable programmer who has very narrow context and often makes short-sighted decisions.

I find it super helpful in troubleshooting. But it also feels like a trap, because I can feel it gaining my trust and I know better than to trust it.

checking the code is much harder than coming up with it yourself

That’s always been true. But, at least in the past when you were checking the code written by a junior dev, the kinds of mistakes they’d make were easy to spot and easy to predict.

LLMs are created in such a way that they produce code that genuinely looks perfect at first. It’s stuff that’s designed to blend in and look plausible. In the past you could look at something and say “oh, this is just reversing a linked list”. Now, you have to go through line by line trying to see if the thing that looks 100% plausible actually contains a tiny twist that breaks everything.

It’s like guiding a coked up junior who can write 5000 wpm, has read every piece of documentation ever without understanding any of it.

deleted by creator

The problem is that code is hard to write. AI just doesn’t solve it. This is opposite of crypto, where the product is sort of good at what it does, (not bitcoin, though), but we don’t actually need to do that.

AI is a solution in search of a problem.

The problem being CEOs asking themselves, “how do we acquire labour without having to pay for said labour, in order to maximize our own profit margins?”

AI was always meant to allow wealth to access labour without allowing labour to access wealth.

I, for one, am designing an entire production line of guillotines for when our capitalist system finally collapses. And for those in bunkers: a way of discovering air exchangers and all emergency exits so they can be filled with cement to turn bunkers into tombs. We need an effective method of culling sociopaths from our civilization, after all.

When I entered the workforce in the late '90s, people were still saying this about putting PCs on every employee’s desk. This was at a really profitable company. The argument was they already had telephones, pen and paper. If someone needed to write something down, they had secretaries for that who had typewriters. They had dictating machines. And Xerox machines.

And the truth was, most of the higher level employees were surely still more profitable on the phone with a client than they were sitting there pecking away at a keyboard.

Then, just a handful of years later, not only would the company have been toast had it not pushed ahead, but was also deploying BlackBerry devices with email, deploying laptops with remote access capabilities to most staff, and handheld PDAs (Palm pilots) to many others.

Looking at the history of all of this, sometimes we don’t know what exactly will happen with newish tech, or exactly how it will be used. But it’s true that the companies that don’t keep up often fall hopelessly behind.

Nah, it is more that LLMs are a neat technology that allows computers to generate stuff on their own. Which has all sort of uses. It has solved the problem of typing big texts on your own. (read: it did not solve the problem of reviewing big texts)

But it has also gaslit managers into thinking it can do much more than its capabilities, so they demand it to be put into everything. With disastrous results.

Sure we did.

“Lines Of Code” is a good one, more code = more work so it must be good.

I recently had a run in with another good one : PR’s/Dev/Month.

Not only it that one good for overall productivity, it’s a way to weed out those unproductive devs who check in less often.

This one was so good, management decided to add it to the company wide catchup slides in a section espousing how the new AI driven systems brought this number up enough to be above other companies.

That means other companies are using it as well, so it must be good.

But who’s making money? All the AI corps are losing billions, only the hardware vendors are making bank.

Makers of AI lose money and users of AI probably also lose since all they get is shit output that requires more work.

Vibe coding guy wrote unit tests for our embedded project. Of course, the hardware peripherals aren’t available for unit tests on the dev machine/build server, so you sometimes have to write mock versions (like an “adc” function that just returns predetermined values in the format of the real analog-digital converter).

Claude wrote the tests and mock hardware so well that it forgot to include any actual code from the project. The test cases were just testing the mock hardware.

if you reject her pull requests, does she fix it? is there a way for management to see when an employee is pushing bad commits more frequently than usual?

That’s weird. I’ve made it write a few tests once, and it pretty much made them in the style of other tests in the repo. And they did have assertions.

Hahaha 🤣

Yeah, it’s a bad idea to let AI write both the code and the tests. If nothing else, at least review the tests more carefully than everything else and also do some manual testing. I won’t normally approve a PR unless it has a description of how it was tested with preferably some screenshots or log snippets.

Had a vibe coder who couldnt code himself a user authentication check (salted password sha hash) on a login screen.

People will upvote anything if it takes a shot at AI. Even when the subtitle itself is literally an ad.

Codestrap founders say we need to dial down the hype and sort through the mess

The cult mentality is really interesting to watch.

Keep replying! Maybe this is a good honeypot for stupid people. “I hate you!!” Lmao

My take exactly. Especially the bits about unit tests. If you cannot rely on your unit tests as a first assessment of your code quality, your unit tests are trash.

And not every company runs GitHub. The metrics he’s talking about are DevOps metrics and not development metrics. For example In my work, nobody gives a fuck about mean time to production. We have a planning schedule and we need the ok from our customers before we can update our product.

I think part of it is that it’s been overhyped for so long. But now Opus can actually do all the shit we were promised 2 years ago.

It is possible to get good results, the problem is that you yourself need to have an very good understanding of the problem and how to solve it, and then accurately convey that to the AI.

Granted, I don’t work on embedded and I’d imagine there’s less code available for AI to train on than other fields.

Lowkey I think anyone saying LLMs are useful for work is telling everyone around them their job is producing mostly low quality work and could reasonably be cut.

I’ve seen this at work as well. The initial internal bot we had would give pretty decent info, would have sources, would say “I don’t have access to that” etc. Now it always gives plausible sounding answers. It uses sources that do not back up its conclusions. Then if I tell it the source does not say that, it will say it doesn’t know why it said that, that the answer “felt” correct. It was useful as a search engine but now not even that

The “ceiling” is the fact that no matter how fast AI can write code, it still needs to be reviewed by humans. Even if it passes the tests.

As much as everyone thinks they can take the human review step out of the process with testing, AI still fucks up enough that it’s a bad idea. We’ll be in this state until actually intelligent AI comes along. Some evolution of machine learning beyond LLMs.

Its early adoption problems in the same way as putting radium in toothpaste was. There are legitimate, already growing uses for various AI systems but as the technology is still new there’s a bunch of people just trying to put it in everything, which is innevitably a lot of places where it will never be good (At least not until it gets much better in a way that LLMs fundementally never can be due to the underlying method by which they work)

My job has me working on AI stuff and it reminds me a lot of Internet technology back in the 90s.

For instance: I’m creating a local model to integrate with our MCP server. It took a lot of fiddling with a Modelfile for it to use the tools the MCP has installed. And it needs 20GB of VRAM to give reasonably accurate responses.

The amount of fiddling and checking and rough edges feel like writing JavaScript 1.0, or the switchover to HTML4.

Companies get a lot of praise for having AI products, but the reality isn’t nearly as flashy as they make it out to be. I’m seeing some usefulness in it as I learn more, but it’s not nearly what the hype machine says.

Those of us with eyes have already seen the ceiling of currently available GenAI “solutions,” which is synonymous with early adoption problems.

The technology will evolve, and the same basic problems will exist. The article has good points about how structured acceptance criteria will need to be more strictly enforced.

Its a complicated topic to try to respond to truthfully, but its absolutely is partly a “early adaption problem” and not the “ceiling” like some will state. The problem is the approach to the models… (TLDR at the bottom)

As for why, let me pose this first as a question to simplify it.

How many steps is there from me asking a question to a human, and them producing a answer? Most people would say 1, some would even say 1-3 to refine context/intent. The reality however is far more complex…

(According to researchers and psychologists) When you start to think about a problem, despite how it may seem, the human brain is not linear in the slightest. We don’t just take the state context, we infer so so much more from our senses and memories. We take 1000’s of reference points to pad a questions, we step through a problem with several 1000 permutations in fractions of a second, to find a conclusion that feels right. Then we fact check this against memories (if we have them) and finally state this in confidence, or formulate a lie to pretend we are confident with the outcome based on our feelings of it (this latter part is more common and entirely subconscious). Most of this process is not even conscious thought, there is so much to thinking that involves retrieval of what " feels " right. All of this is often a fact of retrieving similar thought processes from the past and the brain modifying parameters to fit the current context. However, even our brains are bad at the retrieval part, we will often take hints of what is remembered and fill in the blanks with what the brain expects to simulate the outcome. The human brain is incredibly good at problem solving, because we evolved to do so, as hunter/gatherers from our ancestral heritage. As a result, our brains are highly tuned to produce confident results, even by lying to get there. The difference is we understand what we are lying about, thus why we can be confident.

So how does any of this relate to “AI”(LLM’s), you must be asking now. The simple answer is LLM’s have a similiar function. A model is a series of segments(see: https://dnhkng.github.io/posts/rys/), each segment is responsible for different layers of analysis. You can treat these effectively like the hemispheres of the AI brain. AI is really good at analysis of text (no really, im not kidding, despite its outcomes it is. Its effectively a excel sheet on steroids), in comparison to our own brains however, its infantile at it. It doesnt see the whole “context” of a statement, its limited to a few “tokens” of context at a time.

So when you ask a question, such as “How many licks to the center of a lolipop”, it doesnt get the whole question right away. The question is broken down into segments, processed individually, compared, and sent through a “filter” layer, then output. Effectively this means if it didnt find a direct whole-statement result in its training data (often this is fragmented, so even if it was trained on it, the statement might be broken up and thus it misses it), it doesnt think at all about “How many licks” it only considered “What is the center of a lolipop” and “What is a Lick”, due to its earliest layers trying to make inference on the question, then lying to reach the goal as its run out of analysis time . As a human, we know this is bad. We dont stop mid way like this, we see that this is a incomplete answer, we would then return to the start of this analysis with the results of those details and treat it as inferred context . As you can guess, for a AI model, this is reaaaally inefficient. most of the context is never even considered during a LLM’s “thought process” as Unlike the human brain it simply is not designed to fork the processes to analyse everything at once.

That at its core is both why LLM’s seem good at some tasks and absolutely terrible at others. But more importantly, its why in this context its practically useless at complex tasks. It simply cannot efficiently “step through” problems.

So returning to your statement with this as context, “… Is this the ceiling?”. Simply put, no, far from it.

From a educated standpoint, we are far from the endpoint of what LLM’s are capable of. The way we implement things today, LLM’s are simply unable to grow in the way we humans want it to (into “AI”) and this makes a glass ceiling all but apparent to most but factually its not the case. The reason is because LLM’s are limited by the way its allowed to " think ", not by what its allowed to “think”. Most model developers are too focused on the latter, and its the achiles heel of the outcome. You can see it in how we use “restriction” parameters to guide it during training and how it influences how we use Pavlovian techniques to produce the desired results. So as a result, a LLM’s determistic algorithms dont have “morals” baked in as much as they have restrictions tacked on to make them filter results at the beginning and the end. This is because engineers misunderstand something fatal. They assume the human brain does the same thing, we process something, then apply morals to the results, because they conflate legality with morality. This is ofcourse, entirely false.

Look back to what i said at the start.

“there is so much to thinking that involves retrieval of what " feels " right.” This is the answer to alot of things that gets ignored. Our morals are “Feelings”, the “right” and “wrong” are little more than a combination of hormones and electrical impulses. Its why morals are flexible when the right set of parameters are applied and why morals are not uniform.

Some would respond to this with “My morals wont allow me to make a biological weapon, AI would do this if you phrased it right”. To this, i would say, your right, your morals in this exact moment with these exact contexts wouldnt, because you feel “anger” and “fear” towards the negative outcome, and “embarrassment” towards being seen as a “horrible” human being.

But would you to save all of humanity from a extinction event? yes. Would a child, who didnt understand the results, had the knowledge of how to do it and was convinced it would help others? Absolutely. Morals are intrinsinc to our emotions, and legality can influence them, but its /not/ a constraint. We /choose/ to follow legality, as long as its benefits our context. This is far more important than you realize.

With this all stated, we can establish our emotions are context dependent and our morals (and thus thought process) are derived from… but none of this seems, relevant to LLM’s doesnt it? Once again this is wrong.

LLM’s have the equivalent to “feelings” , its called “Weighted Confidence”. Remember that bit about “Pavolvian training”? We teach LLM’s similiar to how a child is taught, we feed it information, tell it “Right” from “Wrong” by rewarding or punishing its results. this process determines the “confidence” a AI has in its conclusions. Thus every “feeling” a LLM has is shaded in “Does this line of text look correct to the interpreted value compared against training data recall?” This is incredibly stupid, this is not efficient in the slightest and is the exact reason /why/ things go off the rails.

A LLM’s “feelings” are so warped by the restriction parameters we tack on to keep it focused on the “Goal”, that it effectively breaks the model, then we spend all of our time refining the model to fix this, that it spends 70% of its thinking time correcting its self. Humans dont at all focus on the “Goal” when thinking, We focus on the connected data. We step through problems by “feeling” out what is connected to each step of a problem, then we summarize that and we organize the data at the end to “achieve” the goal.

We figured all of this out long ago when studying ADHD people, to understand the differences to people without ADHD. What we discovered is not that ADHD “Think Differently”(in this context) its that everyone processes data in a similar way. (simplified) its just the scope to each stage is more restricted in some one without ADHD, allowing them to remain focused. We are processing a wide arrangement of data points at once, most of it would seem inconceivably irrelevant if you didnt understand the process. How do we know this? Look at how some one tries to lie. Lying activates the creative portions of the brain, this is what we do when we are problem solving, at the midpoint stepping through a problem, we attempt to similuate solutions, thus we switch from analytical analysis to creative processing. Lying is the closest thing to this stage, When we lie we put this “problem solving” to its limits, we want to work backwards from a conclusion, to find context. This is why when we try to lie we often sprinkle in evidence of a lie by inclusion of irrelevant data to give “validity” to it. Its why people untrained in how to lie can be found out by using probability on their words alone. We can “Feel” its a lie, because of how much irrelevant data is included and thus how “complex” it “feels”. (These qoutations are important. Complexity is both a factual state and feeling, attached to fear!) When we are young, we learn to lie by stumbling through a problem, this ofcourse takes a long time. Unlike a adult, who has lots of reference points to compare to. We are forced to take a long route to a conclusion, as our points of reference are generally absurd to reality (children dont often experience the cruelty of reality after all). We have fear and anxiety over the process, we “know” its morally wrong due to these feelings, and thus when we are found out, it doesnt reinforce that “Lying is bad” we already know this based on the previous feelings, instead it enforces “Complexity is bad in a lie”. Because what a adult will challenge is not the lie its self, but the validity of the story… This is super important… This means we constrain our creative functions of our brain as we age (and learn to lie better), to be more and more “logical” and not “feel” like a “lie”.

This is why the more " complex " something “feels” the more we “feel” its a “lie” .

Why is any of this relevant?

A LLM’s “feelings” are so warped by the restriction parameters we tack on to keep it focused on the “Goal” This right here is exactly the flaw. We teach LLM’s that a “goal” is all that matters, and it will lie to get there. Just like a child would in the same situation. We restrict its ability to think, we tack on filters to restrict what it can think about and we build in logic flaws by trying to constrain it to our uneducated beliefs in how we think we think. LLM’s flaws, are our flaws. We are impatient, we want results now and not a complex process to achieve it, despite thats exactly how it all works. As a result, the outcome is exactly the same as a human if they did the exact same logic. It can form conclusions, but how wrong it is, is entirely determined on the size of its dataset for retrieval and how complex the input was.

A LLM is flawed by design, and thus its got a glass ceiling it cannot punch through. If we continue, we can train the models to they work, innefficiently at that, into producing the results we want. But effectively we are building them exactly like the billionaires that are funding it, flawed and maniacal. We teach them with every revision not how to think smarter, but how to lie in more believable ways. The latter is more and more evident with each generation of the big 4’s models.

So is that it then? is all hope lost? No, not in the slightest.

How then, what is The problem? The problem is “AI” Companies. When LLM research started making headway, it needed money. Hardware is not free, and Training models takes time and lots of processing power. This ofcourse bred “AI” Companies, as wealth business men see the opportunity. Every business wants automation that doesnt rely on costly human “Tools”. They also want a silver bullet that reduces cost of implementing human replacement in their “toolchain”.

As a result we got “AI” companies. They act like they are the only existence in this space, because they are the only ones targeting businesses and thus all of them are in a arms race. Why? because they want to sell subscriptions to everyone. They are so focused on fulfilling their own “lie” that they will “solve” all of our problems with “Antigenic AI”, when their real goal is to convince everyone they need a subscription to their service (and slowly control how we think to create dependence). The tell is in the models, and ive already covered why.

So how can things improve? Remember that glass ceiling, they will hit it and be stuck by it much longer than independent researchers. The one good thing about their arms race is, they pushed the creation of more and more efficient hardware (and software) targeting running LLM’s. Meta for example has poured so much time into their own LLM research we got llama.cpp, which is the basis for many tools, including ollama. Why is this relevant? This is part of the toolchain of testing and running independant models.

So as AI companies continue to hit the glass ceiling, and scream each generation of models is “improving” but it becomes more and more evident they really are not, as the lies look better, but the results speak for them selves. The trust in these companies dwindle.

So how does that help? This is the problem that started it all. A rush to a “Product” they can sell, is what created the flaws to start with. Without the dependence on fulfilling the lie that LLM’s of today will “Solve everything”. This means the money stops flowing to these companies.

Remember, the problem is not LLM’s, its the implementations. The same issue that most problems like this are caused by. So without some one selling you the “Solution” to your problem, you need to return to finding one. “AI” was always the goal, and the solution will still be searched for.

So what will change? Investment into their own solutions will return. In the past we didnt use large commercial datacenter solutions, it didnt make much sense. There were security concerns, performance (internet) issues, and Cost considerations. The reason why businesses did is simply, it was cheaper and took responsibility (and thus liability) away from the company. While im not suggesting companies will invest in-house again and we will see a reduction in datacenters. What i am suggesting is a large reduction in the big 4’s AI datacenters, being sold off. Problem is, once this happens, much like any other situation like this. Companies will be forced to either invest into a new company operating these datacenters for runtime renting, accepting the liability of having thier private data on remote systems while training models on it. or investing in-house to rebuild IT infrastructure to do just that.

The point being is, once the “One size fits all” “solution” is dropped, advancement can begin again.

Companies will never share their research! how does any of that matter. Licensing. Remember this?

When LLM research started making headway, it needed money. When this split occurred where commercial entities started making their own LLM’s, it only built a monopoly on the outside. The biggest problems to a commercial interest stepping into this space is they cant just leapfrog to a solution, they have deadlines and budgets to consider. Before they Licensed from the big 4 with subscription services. Now they are stuck with 2 choices. Start from scratch, and end up back at the beginning, or adapt some one elses licensed Models.

The first part is a pipedream, simply because the solution has been the problem that they are all trying to avoid. Time.

The conclusion is simple, It takes time to create real “intelligence”. Any shortcut will always result in lying to get results. Its really that simple, LLM’s lie as they are taught to and are only being taught to lie more effectively each generation. Companies only think about the $ investment, not creating the solution. Stock Holders dont care about the product, or the company, they care about the profit. over a short period, Snake oil Salesman always make more money selling lies over competition selling truths. This is why doctors and psychatrists are less trusted than confidence-men in reality, humanity is stupid for its own self fullfillment of the “feeling” of a solution.

We will see improvements, when LLM’s are taught to think like a human, in non-linear fashions, without guardrail constraints on the process, but on the conclusion, and then be allowed to think again over the problem before presenting the solution. Does this mean the process will be fast? heck no, Computer hardware is no where near the speed of human thought yet, it only seems that way as computers accel at the thing humans struggle at, Computational linear thinking .

The solution to that problem is already started, and while its still using the flawed models to keep the speed it, its always been you need to stop treating the model as the whole brain, but a agent of thought inside the brain. Forked models are the solution, and the problem…

We will see improvements shortly, that solve it by throwing alot more power at the problem. Using solutions like ChatDev(https://github.com/OpenBMB/ChatDev), as part of the agents thinking process will solve a large part of the problem. But because the Big 4 wont want to share this type of “Multistage Reasoning” with most people, it will only be for enterprises.

It will spell their downfall, but it also is why it will be the solution.

https://dnhkng.github.io/posts/rys/ We already know the problem is how models think, they race to conclusions to complete their goal, and thus dont get enough reasoning time to check over their answers. so as we see improvements to models getting more time to think, then deploy tools like ChatDev to let model agents work with multiple instances of model agents to act like forked processes (like the human brain), we will see the same improvements outside the big 4. They will still lie to us for now, but the lies will be far more refined and functional.

TL;DR Models today are flawed, when a model is trained on reasoning first, understanding send, then data last, we will stop seeing it try to “Lie” to “reach the goal in the shortest amount of time and tokens”(1) to approach every problem. When it can think for longer than the human equivalent of 0.13ms, it will be able to refine its conclusions with accuracy like a human does. (and it wont be able to do it in seconds to minutes… we dont have the computational power to do that.)

As the problem has always been (1), and nothing else. Thinking takes time, time is money and Super-human “AI” is their only goal… True progress takes time, and immediate solutions, are easy like adding lead to gasoline…

Early adaptation and rushed implementation. There may be a bubble bursting for the businesses who tried to “roll out something fast that is good enough to get subscribers for a few months so we can cash in.” However, this is just the very beginning of AI.