pcalau12i

Personally I think general knowledge is kind of a useless metric because you're not really developing "intelligence" at that point just a giant dictionary, and of course bigger models will always score better because they are bigger. In some sense training an ANN is kinda like a compression algorithm of a ton of knowledge, so the bigger the parameters the less lossy the compression it is, the more it knows. But having an absurd amount of knowledge isn't what makes humans intelligent, most humans know very little, it's problem solving. If we have a problem solving machine as intelligent as a human we can just give it access to the internet for that information. Making it bigger with more general knowledge, imo, isn't genuine "progress" in intelligence. The recent improvements by adding reasoning is a better example of genuine improvements to intelligence.

These bigger models are only scoring better because they have just memorized so much they have seen similar questions before. Genuine improvements to intelligence and progress in this field come when people figure out how to improve the results without more data. These massive models already have more data than ever human could ever have access to in hundreds of lifetimes. If they aren't beating humans on every single test with that much data then clearly there is something else wrong.

That's just the thing, though, the point I am making, which is that it turns out in practice synthetic data can give you the same effect as original data. In some sense, training an LLM is kind of like a lossy compression algorithm, you are trying to fit petabytes of data into a few hundred gigabytes as efficiently as possible. In order to successfully compress it, it has to lose specifics, so the algorithm only captures general patterns. This is true for any artificial neural network, so if you train another neural network with the data yourself, you will also lose specifics in the training process and end up with a model that only knows general patterns. Hence, if you train a model using synthetic data, the information lost in that synthetic data will be information the AI you are training would lose anyways, so you don't necessarily get bad results.

But yes, when I was talking about synthetic data I had in mind data purely generated from an LLM. Of course I do agree translating documents, OCRing documents, etc, to generate new data is generally a good thing as well. I just disagree with your final statement there that it is critical to have a lot of high-quality original data. The notion that we can keep making AIs better by just giving them more and more data, this method is already plateauing in the industry and showing diminishing returns. ChatGPT 3.5 to 4 was a massive leap but the jump to 4.5, which uses an order of magnitude more compute mind you, is negligible.

Just think about it. Humans are way smarter than ChatGPT and we don't require the energy of a small country and petabytes of all the world's information to solve simple logical puzzles, just a hot pocket and a glass of water. There is clearly an issue in how we are training things and not the lack of data. We have plenty of data. Recent breakthroughs have come in finding more clever ways to use the data rather than just piling on more and more data.

For example, many models have recently adopted reasoning techniques, so rather than simply spitting out an answer it generates an internal dialog prior to generating the answer, it "thinks" about the problem for a bit. These reasoning models perform way better on complex questions. OpenAI first invented the technique but kept it under lock and key, and the smaller company DeepSeek managed to replicate it and made their methods open source for everyone, and then Alibaba put it into their Qwen model in a new model they call QwQ which dropped recently and performs almost as well as ChatGPT 4 on some benchmarks yet can be run on consumer-end hardware with as little as 24GB of VRAM.

All the major breakthroughs happening recently are coming from not having more data but using the data in more clever ways. Just recently a diffusion LLM dropped which creates text output but borrows the same techniques used in image generation, so rather than doing it character-by-character it outputs a random sequence of characters all at once and continually refines it until it makes sense. This technique is used with images because uncompressed images take up megabytes of data while LLM outputs only output a few kilobytes in a response, so it would just be too slow to use the same method for image generation, yet by applying the image generation method to do what LLMs do it makes it produce reasonable outputs faster than any traditional LLM.

This is a breakthrough that just happened, here's an IBM article on it from 3 days ago!

https://www.ibm.com/think/news/diffusion-models-llms

The breakthroughs are really not happening in huge data collection right now. Companies will still steal all your data because big data collection is still profitable to sell to advetisers, but it's not at the heart of the AI revolution right now. That is coming from computer science geniuses who cleverly figure out how to use the data in more effective ways.

We know how it works, we just don’t yet understand what is going on under the hood.

Why should we assume "there is something going on under the hood"? This is my problem with most "interpretations" of quantum mechanics. They are complex stories to try and "explain" quantum mechanics, like a whole branching multiverse, of which we have no evidence for.

It's kind of like if someone wanted to come up with deep explanation to "explain" Einstein's field equations and what is "going on under the hood". Why should anything be "underneath" those equations? If we begin to speculate, we're doing just tha,t speculation, and if we take any of that speculation seriously as in actually genuinely believe it, then we've left the realm of being a scientifically-minded rational thinker.

It is much simpler to just accept the equations at face-value, to accept quantum mechanics at face-value. "Measurement" is not in the theory anywhere, there is no rigorous formulation of what qualifies as a measurement. The state vector is reduced whenever a physical interaction occurs from the reference point of the systems participating in the interaction, but not for the systems not participating in it, in which the systems are then described as entangled with one another.

This is not an "interpretation" but me just explaining literally how the terminology and mathematics works. If we just accept this at face value there is no "measurement problem." The only reason there is a "measurement problem" is because this contradicts with people's basic intuitions: if we accept quantum mechanics at face value then we have to admit that whether or not properties of systems have well-defined values actually depends upon your reference point and is contingent on a physical interaction taking place.

Our basic intuition tells us that particles are autonomous entities floating around in space on their lonesome like little stones or billiard balls up until they collide with something, and so even if they are not interacting with anything at all they meaningfully can be said to "exist" with well-defined properties which should be the same properties for all reference points (i.e. the properties are absolute rather than relational). Quantum mechanics contradicts with this basic intuition so people think there must be something "wrong" with it, there must be something "under the hood" we don't yet understand and only if we make the story more complicated or make a new discovery one day we'd "solve" the "problem."

Einstein once said, God does not place dice, and Bohr rebutted with, stop telling God what to do. This is my response to people who believe in the "measurement problem." Stop with your preconceptions on how reality should work. Quantum theory is our best theory of nature and there is currently no evidence it is going away any time soon, and it's withstood the test of time for decades. We should stop waiting for the day it gets overturned and disappears and just accept this is genuinely how reality works, accept it at face-value and drop our preconceptions. We do not need any additional "stories" to explain it.

The blind spot is that we don’t know what a quantum state IS. We know the maths behind it, but not the underlying physics model.

What is a physical model if not a body of mathematics that can predict outcomes? The physical meaning of the quantum state is completely unambiguous, it is just a list of probability amplitudes. Probability captures the likelihoods of certain outcomes manifesting during an interaction, although quantum probability amplitudes are somewhat unique in that they are complex-valued, but this is to add the additional degrees of freedom needed to simultaneously represent interference phenomena. The state vector is a mathematical notation to capture likelihoods of events occurring while accounting for interference effects.

It’s likely to fall out when we unify quantum mechanics with general relativity, but we’ve been chipping at that for over 70 years now, with limited success.

There has been zero "progress" because the "problem" of unifying quantum mechanics and general relativity is a pseudoproblem. It stems from a bias that because we had success quantizing all the fundamental forces except gravity, then therefore gravity should be quantizable. Since the method that worked for all other forces failed, this being renormalization, all these other theories search for a different way to do it.

But (1) there is no reason other than blind faith to think gravity should be quantized, and (2) there is no direct compelling evidence that either quantum mechanics or general relativity are even wrong.

Also, we can already unify quantum mechanics and general relativity just fine. It's called semi-classical gravity and is what Hawking used to predict that black holes radiate. It makes quantum theory work just fine in a curved spacetime and is compatible with all experimental predictions to this day.

People who dislike semiclassical gravity will argue it seems to make some absurd predictions in under specific conditions we currently haven't measured. But this isn't a valid argument to dismiss it, because until you can actually demonstrate via experiment that such conditions can actually be created in physical reality, then it remains a purely metaphysical criticism and not a scientific one.

If semi-classical gravity is truly incorrect then you cannot just point to it having certain strange predictions in certain domains, you also have to demonstrate it is physically possible to actually probe them and this isn't just a metaphysical quirk of the theory of trying to make predictions to things that aren't physical possible in the first place and thus naturally what it would predict would also be physically impossible.

If you could construct such an experiment and its prediction was indeed wrong, you'd disprove it the very second you turned on the experiment. Hence, if you genuinely think semi-classical gravity is wrong and you are actually following the scientific method, you should be doing everything in your power to figure out how to probe these domains.

But instead people search for many different methods of trying to quantize gravity and then in a post-hoc fashion look for ways it could be experimentally verified, then when it is wrong they go back and tweak it so it is no longer ruled out by experiment, and zero progress has been made because this is not science. Karl Popper's impact on the sciences has been hugely detrimental because now everyone just believes if something can in principle be falsified it is suddenly "science" which has popularized incredibly unscientific methods in academia.

Eh, individuals can't compete with corpos not just because they have access to more data but because making progress in AI requires a large team of well-educated researchers and sufficient capital to be able to experiment with vast technology. It's a bit like expecting an individual or small business to be able to compete with smartphone manufacturers. It really is not feasible not simply because smartphone manufacturers are using dirty practices but because producing smartphones requires an enormous amount of labor and capital and simply cannot be physically carried out by an individual.

This criticism might be more applicable to a medium-sized business like DeepSeek that is not really "small" but smaller than the others (and definitely not a single individual) and still big enough to still compete, and we can see they still could compete just fine despite the current situation.

The truth is that both USA and China recognize all purely AI-generated work as de facto public domain. That means anything ChatGPT or whatever spits out, no matter what their licensing says, is absolutely free to use however you wish and you will win in court if they try to stop you. There is a common myth that training AI on synthetic data will always be negative. It's actually only sometimes true if you train the AI on its own synthetic data, but through a process they call "distillation" you can train a less intelligent AI on synthetic data from a more intelligent AI and it will actually improve its performance.

That means any AI made by big companies can be distilled into any other AI to improve its performance. This is because you effectively have access to all the data the big companies have access to but indirectly through the synthetic data their AI can produce. For example, if for some reason you curated the information the AI was trained on so it never encountered the concept of a dog, it simply wouldn't know what a dog is. If it encountered it a lot, it would know what a dog is and could explain it if you asked. Hence, that information is effectively accessible indirectly by simply asking the AI for it.

If you use distillation then you should can make effectively your own clones of any big company's AI model and it's perfectly legal. Not only that, but you can make improvements to it as well. You aren't just cloning models, but you have the power to modify them. during this distillation process.

Imagine if the initial model was trained using a particular technique that is rather outdated and you believe you've invented a new method that if re-trained would produce a smarter AI, but you simply lack access to the original data. What you can instead do is generate a ton of synthetic data from the AI and then train your new AI using the new method on that synthetic data. Your new AI will have access to most of the same information but now trained on a superior technique.

We have seen some smaller companies already take pre-existing models and use distillation to improve them, such as DeepSeek taking the Qwen models and distilling R1 reasoning techniques into them to improve their performance.

I always think articles like this are incredibly stupid, honestly. Political parties exist to push a particular ideology, not to win elections. If the communist party abandoned communism and became a neonazi party to win the election, and they did succeed in winning, did the communist party really "win"? Not really. If you have to abandon your ideology to win then you did not win.

It's pretty rare for parties to actually abandon their ideology like that. The job of a political party is not to merely win, but to convince the population that their ideology is superior so people will back them. They want to win, yes, but under the conditions that they have won because the people back their message so that they can implement it.

This is why I always find it incredibly stupid when I see all these articles and progressive political commentators saying that the Democrats are a stupid party for not shifting their rhetoric to be more pro-working class, to be anti-imperialist, etc. THE DEMOCRATS ARE NOT A WORKING CLASS PARTY. It would in fact be incredibly stupid for them to shift to be more left because doing so would abandon their values. The Democrats' values are billionaires, free market capitalism, and imperialism. These are not "stupid" decisions they're making for supporting these things, THESE ARE THE FUNDAMENTAL BELIEFS OF THE PARTY.

In normal countries if you dislike a party's ideology, you support a different party. But Americans have this weird fantasy that Democrats should just be "reasonable" and entirely abandon their core values to back their own values, and so they refuse to ever back a different party because of this ridiculous delusion. Whenever the Democrats fail to adopt working-class values, they run these stupid headlines saying the Democrats are being "unreasonable" or "stupid" or have "bad strategy" or are "incompetents" or whatever and "just don't want to fight."

Literally none of that is true. The Democrats are extremely fierce fighters when it comes to defending imperialism and the freedoms of billionaires. They aren't fighting for your values because those are not their values, and so you should back a different party.

On the surface, it does seem like there is a similarity. If a particle is measured over here and later over there, in quantum mechanics it doesn't necessarily have a well-defined position in between those measurements. You might then want to liken it to a game engine where the particle is only rendered when the player is looking at it. But the difference is that to compute how the particle arrived over there when it was previously over here, in quantum mechanics, you have to actually take into account all possible paths it could have taken to reach that point.

This is something game engines do not do and actually makes quantum mechanics far more computationally expensive rather than less.

So usually this is explained with two scientists, Alice and Bob, on far away planets. They’re each in the possession of a particle that is entangled with the other, and in a superposition of state 1 and state 2.

This "usual" way of explaining it is just overly complicating it and making it seem more mystical than it actually is. We should not say the particles are "in a superposition" as if this describes the current state of the particle. The superposition notation should be interpreted as merely a list of probability amplitudes predicting the different likelihoods of observing different states of the system in the future.

It is sort of like if you flip a coin, while it's in the air, you can say there is a 50% chance it will land heads and a 50% chance it will land tails. This is not a description of the coin in the present as if the coin is in some smeared out state of 50% landed heads and 50% landed tails. It has not landed at all yet!

Unlike classical physics, quantum physics is fundamentally random, so you can only predict events probabilistically, but one should not conflate the prediction of a future event to the description of the present state of the system. The superposition notation is only writing down probability amplitudes of the likelihoods of what you will observe (state 1 or state 2) of the particles in the future event that you go to the interact with it and is not a description of the state of the particles in the present.

When Alice measures the state of her particle, it collapses into one of the states, say state 1. When Bob measures the state of his particle immediately after, before any particle travelling at light speed could get there, it will also be in state 1 (assuming they were entangled in such a way that the state will be the same).

This mistreatment of the mathematical notation as a description of the present state of the system also leads to confusing language like "it collapses into one of the states" as if the change in a probability distribution represents a physical change to the system. The mental picture people say this often have is that the particle literally physically becomes the probability distribution prior to measuring it---the particle "spreads out" like a wave according to the probability amplitudes of the state vector---and when you measure the particle, this allows you to update the probabilities, and so they must interpret this as the wave physically contracting into an eigenvalue---it "collapses" like a house of cards.

But this is, again, overcomplicating things. The particle never spreads out like a wave and it never "collapses" back into a particle. The mathematical notation is just a way of capturing the likelihoods of the particle showing up in one state or the other, and when you measure what state it actually shows up in, then you can update your probabilities accordingly. For example, if you the coin is 50%/50% heads/tails and you observe it land on tails, you can update the probabilities to 0%/100% heads/tails because you know it landed on tails and not heads. Nothing "collapsed": you're just observing the actual outcome of the event you were predicting and updating your statistics accordingly.

Any time you do something to the particles on Earth, the ones on the Moon are affected also

The no-communication theorem already proves that manipulating one particle in an entangled pair has no impact at al on another. The proof uses the reduced density matrices of the particles which capture both their probabilities of showing up in a particular state as well as their coherence terms which capture their ability to exhibit interference effects. No change you can make to one particle in an entangled pair can possibly lead to an alteration of the reduced density matrix of the other particle.

I don't think solving the Schrodinger equation really gives you a good idea of why quantum mechanics is even interesting. You also shouldstudy very specific applications of it where it yields counterintuitive outcomes to see why it is interesting, such as in the GHZ experiment.

There is no "consciousness." False belief in "consciousness" is a product of Kantianism, which itself was heavily inspired by Newtonian physics (Kant was heavily inspired by Newton), which we have changed some categories over the years but the fundamentals have not and have become deeply integrated into western psyche in how we think about the world, and probably in many other cultures as well.

Modern day philosophers have just renamed Kant's phenomena to "consciousness" or "subjective experience" and renamed his "noumena" to "matter." Despite the renaming, the categories are still treated identically: the "consciousness" is everything we perceive, and the "matter" is something invisible, the true physical thing-in-itself beyond our perception and what "causes" our perception.

Since all they have done is rename Kant's categories, they do not actually solve Kant's mind-body problem, but have just rediscovered it and thus renamed it in the form of the "hard problem of consciousness," which is ultimately the same exact problem just renamed: that there seems to be a "gap" between this "consciousness" and "matter."

Most modern day philosophers seem to split into two categories. The first are the "promissory materialists" who just say there is a real problem here but shrug their shoulders and say one day science will solve it so we don't have to worry about it, but give no explanation of what a solution could even possibly look like. The second are the mystics who insist this "consciousness" can't be reconciled with "matter" because it must be some fundamental force of reality. They talk about things like "consciousness fields" or "cosmic consciousness" or whatever.

However, both are wrong. Newtonian physics is not an accurate represent of reality, we already know this, and so the Kantian mindset inspired from it should also be abandoned. When you abandon the Kantian mindset, there is no longer a need for the "phenomena" and "noumena" division, or, in modern lingo, there is no longer a need for the "consciousness" and "matter" division. There is just reality.

Imagine you are looking at a candle. The apparent size of the candle you will see will depend upon how far you are away from it: if you are further away it appears smaller. Technically, light doesn't travel at an infinite speed, and so the further away you are, the further in the past you are seeing the candle. The candle also may appear a bit different under different lighting conditions.

A Kantian would say there is a true candle, the "candle-in-itself," or, in modern lingo, the material candle, the "causes" all these different perceptions. The perceptions themselves are then said to be brain-generated, not part of the candle, not even something real at all, but something purely immaterial, part of the phenomena, or, in modern lingo, part of "consciousness."

If every possible perception of the candle is part of "consciousness," then the candle-in-itself, the actual material object, must be independent of perception, i.e. it's invisible. No observation can reveal it because all observations are part of "consciousness." This is the Kantian worldview: everything we perceive is part of a sort of illusion created within the mind as opposed to the "true" world that is entirely imperceptible. The mind-body problem, or in modern lingo the "hard problem," then arises as to how an entirely imperceptible (non-phenomenal/non-conscious) world can give rise to what we perceive in a particular configuration.

However, the Kantian worldview is a delusion. In Newtonian physics, if I launch a cannonball from point A to point B, simply observing it at point A and point B is enough to fill in the gaps to say where the object was at every point in between A and B independently of anything else. This Newtonian worldview allows us to conceive of the cannonball as a thing-in-itself, an object with its own inherent properties that can be meaningful conceived of existing even when in complete isolation, that always has an independent of history of how it ends up where it does.

As Schrodinger pointed out, this mentality does not apply to modern physics. If you fire a photon from point A to point B and observe it at those two points, you cannot always meaningfully fill in the gaps of what the photon was doing in between those two points without running into contradictions. As Schrodinger concluded, one has to abandon the notion that particles really are independent autonomous entities with their own independent existence that can be meaningfully conceived of in complete isolation. They only exist from moment to moment in the context of whatever they are interacting with and not in themselves.

If this is true for particles, it must also be true of everything made up of particles: there is no candle-in-itself either. It's a high-level abstraction that doesn't really exist. What we call the "candle" is not an independent unobservable entity separate from all our different perceptions of it, but what we call the candle is precisely the totality of all the different ways it is and can be perceived, all the different ways it interacts with other objects from those objects' perspectives.

Kant justified the noumena by arguing that it makes no sense to talk about objects "appearing" (the word "phenomena" means "the appearance of") without there being something that is doing the appearing (the noumena). He is correct on this, but for a different reason. We should not use this to justify the noumena, but it shows that if we reject the noumena, we must also reject the phenomena ("consciousness"): it makes no sense to treat the different instances of a candle as some sort of separate "consciousness" realm, or some sort of illusion or whatever independent of the real material world as it really is.

No, what we perceive directly is material reality as it actually is. Reality is what you are immersed in every day, what surrounds you, what you are experiencing in this very moment. It is not some illusion from which there is a "true" invisible reality beyond it. When you look at the candle, you are seeing the candle as it really is from your own perspective. That is the real candle in the real world. The Kantian distinction between noumena-phenomena (or between "matter" and "consciousness") should be abandoned. It is just not compatible with the modern physical sciences.

But I know no one will even know what I'm talking about, so writing this is rather pointless. Kantianism is too deeply ingrained into the western psyche, people cannot even comprehend that it is possible to criticize it because it underlies how they think about everything. This nonsense debate about "consciousness" will continue forever, in ten thousand years people will still be arguing over it, because it's an intrinsic problem that arises out of the dualistic structure in Kantian thinking. If you begin from the get-go with an assumption that there is a division between mind and matter, you cannot close this division without contradicting yourself, which leads to this debate around "consciousness." But it seems unrealistic at this point to get people to abandon this dualistic way of thinking, so it seems like the "consciousness" debate will proceed forever.

You have not made any point at all. Your first reply to me entirely ignored the point of my post which you did not read followed with an attack, I reply pointing out you ignored the whole point of my post and just attacked me without actually respond to it, and now you respond again with literally nothing of substance at all just saying "you're wrong! touch grass! word salad!"

You have nothing of substance to say, nothing to contribute to the discussion. You are either a complete troll trying to rile me up, or you just have a weird emotional attachment to this topic and felt an emotional need to respond and attack me prior to actually thinking up a coherent thing to criticize me on. Didn't your momma ever teach you that "if you have nothing positive or constructive to say, don't say anything at all"? Learn some manners, boy. Blocked.