pcalau12i

There is no action at a distance in quantum mechanics, that is a laymen’s misconception. If there was, it would not be compatible with special relativity, but it is compatible as they are already unified under the framework of quantum field theory. The No-communication theorem is a rather simple proof that shows there is no “sharing at a distance” in quantum mechanics. It is an entirely local theory. The misconception arises from people misinterpreting Bell’s theorem which says quantum mechanics is not compatible with a local hidden variable theory, so people falsely conclude it’s a nonlocal theory, but this is just false because quantum mechanics is not a hidden variable theory, and so it is not incompatible with locality. It is a local theory. Bell’s theorem only shows it is nonlocal if you introduce hidden variables, meaning the theorem is really only applicable to a potential replacement to quantum mechanics and is not even applicable to quantum mechanics itself. It is applicable to things like pilot wave theory, but not to quantum theory.

Personally I think general knowledge is kind of a useless metric because you’re not really developing “intelligence” at that point just a giant dictionary, and of course bigger models will always score better because they are bigger. In some sense training an ANN is kinda like a compression algorithm of a ton of knowledge, so the bigger the parameters the less lossy the compression it is, the more it knows. But having an absurd amount of knowledge isn’t what makes humans intelligent, most humans know very little, it’s problem solving. If we have a problem solving machine as intelligent as a human we can just give it access to the internet for that information. Making it bigger with more general knowledge, imo, isn’t genuine “progress” in intelligence. The recent improvements by adding reasoning is a better example of genuine improvements to intelligence.

These bigger models are only scoring better because they have just memorized so much they have seen similar questions before. Genuine improvements to intelligence and progress in this field come when people figure out how to improve the results without more data. These massive models already have more data than ever human could ever have access to in hundreds of lifetimes. If they aren’t beating humans on every single test with that much data then clearly there is something else wrong.

That’s just the thing, though, the point I am making, which is that it turns out in practice synthetic data can give you the same effect as original data. In some sense, training an LLM is kind of like a lossy compression algorithm, you are trying to fit petabytes of data into a few hundred gigabytes as efficiently as possible. In order to successfully compress it, it has to lose specifics, so the algorithm only captures general patterns. This is true for any artificial neural network, so if you train another neural network with the data yourself, you will also lose specifics in the training process and end up with a model that only knows general patterns. Hence, if you train a model using synthetic data, the information lost in that synthetic data will be information the AI you are training would lose anyways, so you don’t necessarily get bad results.

But yes, when I was talking about synthetic data I had in mind data purely generated from an LLM. Of course I do agree translating documents, OCRing documents, etc, to generate new data is generally a good thing as well. I just disagree with your final statement there that it is critical to have a lot of high-quality original data. The notion that we can keep making AIs better by just giving them more and more data, this method is already plateauing in the industry and showing diminishing returns. ChatGPT 3.5 to 4 was a massive leap but the jump to 4.5, which uses an order of magnitude more compute mind you, is negligible.

Just think about it. Humans are way smarter than ChatGPT and we don’t require the energy of a small country and petabytes of all the world’s information to solve simple logical puzzles, just a hot pocket and a glass of water. There is clearly an issue in how we are training things and not the lack of data. We have plenty of data. Recent breakthroughs have come in finding more clever ways to use the data rather than just piling on more and more data.

For example, many models have recently adopted reasoning techniques, so rather than simply spitting out an answer it generates an internal dialog prior to generating the answer, it “thinks” about the problem for a bit. These reasoning models perform way better on complex questions. OpenAI first invented the technique but kept it under lock and key, and the smaller company DeepSeek managed to replicate it and made their methods open source for everyone, and then Alibaba put it into their Qwen model in a new model they call QwQ which dropped recently and performs almost as well as ChatGPT 4 on some benchmarks yet can be run on consumer-end hardware with as little as 24GB of VRAM.

All the major breakthroughs happening recently are coming from not having more data but using the data in more clever ways. Just recently a diffusion LLM dropped which creates text output but borrows the same techniques used in image generation, so rather than doing it character-by-character it outputs a random sequence of characters all at once and continually refines it until it makes sense. This technique is used with images because uncompressed images take up megabytes of data while LLM outputs only output a few kilobytes in a response, so it would just be too slow to use the same method for image generation, yet by applying the image generation method to do what LLMs do it makes it produce reasonable outputs faster than any traditional LLM.

This is a breakthrough that just happened, here’s an IBM article on it from 3 days ago!

https://www.ibm.com/think/news/diffusion-models-llms

The breakthroughs are really not happening in huge data collection right now. Companies will still steal all your data because big data collection is still profitable to sell to advetisers, but it’s not at the heart of the AI revolution right now. That is coming from computer science geniuses who cleverly figure out how to use the data in more effective ways.

Eh, individuals can’t compete with corpos not just because they have access to more data but because making progress in AI requires a large team of well-educated researchers and sufficient capital to be able to experiment with vast technology. It’s a bit like expecting an individual or small business to be able to compete with smartphone manufacturers. It really is not feasible not simply because smartphone manufacturers are using dirty practices but because producing smartphones requires an enormous amount of labor and capital and simply cannot be physically carried out by an individual.

This criticism might be more applicable to a medium-sized business like DeepSeek that is not really “small” but smaller than the others (and definitely not a single individual) and still big enough to still compete, and we can see they still could compete just fine despite the current situation.

The truth is that both USA and China recognize all purely AI-generated work as de facto public domain. That means anything ChatGPT or whatever spits out, no matter what their licensing says, is absolutely free to use however you wish and you will win in court if they try to stop you. There is a common myth that training AI on synthetic data will always be negative. It’s actually only sometimes true if you train the AI on its own synthetic data, but through a process they call “distillation” you can train a less intelligent AI on synthetic data from a more intelligent AI and it will actually improve its performance.

That means any AI made by big companies can be distilled into any other AI to improve its performance. This is because you effectively have access to all the data the big companies have access to but indirectly through the synthetic data their AI can produce. For example, if for some reason you curated the information the AI was trained on so it never encountered the concept of a dog, it simply wouldn’t know what a dog is. If it encountered it a lot, it would know what a dog is and could explain it if you asked. Hence, that information is effectively accessible indirectly by simply asking the AI for it.

If you use distillation then you should can make effectively your own clones of any big company’s AI model and it’s perfectly legal. Not only that, but you can make improvements to it as well. You aren’t just cloning models, but you have the power to modify them. during this distillation process.

Imagine if the initial model was trained using a particular technique that is rather outdated and you believe you’ve invented a new method that if re-trained would produce a smarter AI, but you simply lack access to the original data. What you can instead do is generate a ton of synthetic data from the AI and then train your new AI using the new method on that synthetic data. Your new AI will have access to most of the same information but now trained on a superior technique.

We have seen some smaller companies already take pre-existing models and use distillation to improve them, such as DeepSeek taking the Qwen models and distilling R1 reasoning techniques into them to improve their performance.

You have not made any point at all. Your first reply to me entirely ignored the point of my post which you did not read followed with an attack, I reply pointing out you ignored the whole point of my post and just attacked me without actually respond to it, and now you respond again with literally nothing of substance at all just saying “you’re wrong! touch grass! word salad!”

You have nothing of substance to say, nothing to contribute to the discussion. You are either a complete troll trying to rile me up, or you just have a weird emotional attachment to this topic and felt an emotional need to respond and attack me prior to actually thinking up a coherent thing to criticize me on. Didn’t your momma ever teach you that “if you have nothing positive or constructive to say, don’t say anything at all”? Learn some manners, boy. Blocked.

They are incredibly efficient for short-term production, but very inefficient for long-term production. Destroying the environment is a long-term problem that doesn’t have immediate consequences on the businesses that engage in it. Sustainable production in the long-term requires foresight, which requires a plan. It also requires a more stable production environment, i.e. it cannot be competitive because if you are competing for survival you will only be able to act in your immediate interests to avoid being destroyed in the competition.

Most economists are under a delusion known as neoclassical economics which is literally a nonphysical theory that treats the basis of the economy as not the material world we actually live in but abstract human ideas which are assumed to operate according to their own internal logic without any material causes or influences. They then derive from these imagined “laws” regarding human ideas (which no one has ever experimentally demonstrated but were just invented in some economists’ armchair one day) that humans left to be completely free to make decisions without any regulations at all will maximize the “utils” of the population, making everyone as happy as possible.

With the complete failure of this policy leading to the US Great Depression, many economists recognized this was flawed and made some concessions, such as with Keynesianism, but they never abandoned the core idea. In fact, the core idea was just reformulated to be compatible with Keynesianism in what is called the neoclassical synthesis. It still exists as a fundamental belief to most every economist that completely unregulated market economy without any plan at all will automagically produce a society with maximal happiness, and while they will admit some caveats to this these days (such as the need for a central organization to manage currency in Keynesianism), these are treated as an exception and not the rule. Their beliefs are still incompatible with long-term sustainable planning because in their minds the success of markets from comes util-maximizing decisions built that are fundamental to the human psyche and so any long-term plan must contradict with this and lead to a bad economy that fails to maximize utils.

The rise of Popperism in western academia has also played a role here. A lot of material scientists have been rather skeptical of the social sciences and aren’t really going to take arguments like those based in neoclassical economics which is based largely in mysticism about human free will seriously, and so a second argument against long-term planning was put forward by Karl Popper which has become rather popular in western academia. Popper argued that it is impossible to learn from history because it is too complicated with too many variables and you cannot control them all. You would need a science that studies how human societies develop in order to justify a long-term development plan into the future, but if it’s impossible to study them to learn how they develop because they are too complicated, then it is impossible to have such a science, and thus impossible to justify any sort of long-term sustainable development plan. It would always be based on guesswork and so more likely to do more harm than good. Popper argued that instead of long-term development plans, the state should instead be purely ideological, what he called an “open society” operating purely on the ideology of liberalism rather getting involved in economics.

As long as both neoclassical economics and Popperism are dominate trends in western academia there will never be long-term sustainable planning because they are fundamentally incompatible ideas.

You did not read what I wrote, so it is unironic you call it “word salad” when you are not even aware of the words I wrote since you had an emotional response and wrote this reply without actually addressing what I argued. I stated that it is impossible to have an very large institution without strict rules that people follow, and this requires also the enforcement of the rules, and that means a hierarchy as you will have rule-enforcers.

Also, you are insisting your personal definition of anarchism is the one true definition that I am somehow stupid for disagreeing with, yet anyone can just scroll through the same comments on this thread and see there are other people disagreeing with you while also defending anarchism. A lot of anarchists do not believe anarchism means “no hierarchy,” like, seriously, do you unironically believe in entirely abolishing all hierarchies? Do you think a medical doctor should have as much authority on how to treat an injured patient as the janitor of the same hospital? Most anarchists aren’t even “no hierarchy” they are “no unjustified hierarchy.”

The fact you are entirely opposed to hierarchy makes your position even more silly than what I was criticizing.

All libertarian ideologies (including left and right wing anarchism) are anti-social and primitivist.

It is anti-social because it arises from a hatred of working in a large groups. It’s impossible to have any sort of large-scale institution without having rules that people want to follow, and libertarian ideology arises out of people hating to have to follow rules, i.e. to be a respectable member of society, i.e. they hate society and don’t want to be social. They thus desire very small institutions with limited rules and restrictions. Right-wing libertarians envision a society dominated by small private businesses while left-wing libertarians imagine a society dominated by either small worker-cooperative, communes, or some sort of community council.

Of course, everyone of all ideologies opposes submitting to hierarchies they find unjust, but hatred of submitting to hierarchies at all is just anti-social, as any society will have rules, people who write the rules, people who enforce the rules. It is necessary for any social institution to function. It is part of being an adult and learning to live in a society to learn to obey the rules, such as traffic rules. Sometimes it is annoying or inconvenient, but you do it because you are a respectable member of society and not a rebellious edgelord who makes things harder on everyone else because they don’t obey basic rules.

It is primitivist because some institutions simply only work if they are very large. You cannot have something like NASA that builds rocket ships operated by five people. You are going to always need an enormous institution which will have a ton of people, a lot of different levels of command (“hierarchy”), strict rules for everyone to follow, etc. If you tried to “bust up” something like NASA or SpaceX to be small businesses they simply would lose their ability to build rocket ships at all.

Of course, anarchists don’t mind, they will say, “who cares about rockets? They’re not important.” It reminds me of the old meme that spread around where someone asked anarchists how their tiny communes would be able to organize current massive supply chains in our modern societies and they responded by saying that the supply chain would be reduced to just people growing beans in their backyard and eating it, like a feudal peasant. They won’t even defend that their system could function as well as our modern economy but just says modern marvels of human engineering don’t even matter, because they are ultimately primitivists at heart.

I never understood the popularity of libertarian and anarchist beliefs in programming circles. We would never have entered the Information Age if we had an anarchism or libertarian system. No matter how much they might pretend these are the ideal systems, they don’t even believe it themselves. If a libertarian has a serious medical illness, they are either going to seek medical help at a public hospital or a corporate hospital. Nobody is going to seek medical help at a “hospital small business” ran out of someone’s garage. We all intuitively and implicitly understand that large swathes of economy that we all take advantage of simply cannot feasibly be ran by small organizations, but libertarians are just in denial.

Anarchism thus becomes meaningless as anyone who defends certain hierarchies obviously does so because they believe they are just. Literally everyone on earth is against “unjust hierarchies” at least in their own personal evaluation of said hierarchies. People who support capitalism do so because they believe the exploitative systems it engenders are justifiable and will usually immediately tell you what those justifications are. Sure, you and I might not agree with their argument, but that’s not the point. To say your ideology is to oppose “unjust hierarchies” is to not say anything at all, because even the capitalist, hell, even the fascist would probably agree that they oppose “unjust hierarchies” because in their minds the hierarchies they promote are indeed justified by whatever twisted logic they have in their head.

Telling me you oppose “unjust hierarchies” thus tells me nothing about what you actually believe, it does not tell me anything at all. It is as vague as saying “I oppose bad things.” It’s a meaningless statement on its own without clarifying what is meant by “bad” in this case. Similarly, “I oppose unjust hierarchies” is meaningless statement without clarifying what qualifies “just” and “unjust,” and once you tell me that, it would make more sense you label you based on your answer to that question. Anarchism thus becomes a meaningless word that tells me nothing about you. For example, you might tell me one unjust hierarchy you want to abolish is prison. It would make more sense for me to call you a prison abolitionist than an anarchist since that term at least carries meaning, and there are plenty of prison abolitionists who don’t identify as anarchist.