Becoming Stronger X: Yudkowsky’s Coming Of Age
Yudkowsky grew up in an environment which praised experience over intelligence as justification for everything, including religion. This led them to the opposite, an affective death spiral around intelligence as the solution to everything. They thought that being very intelligent meant being very moral. They tended to go too far the other way in reaction to someone else’s stupidity. (My Childhood Death Spiral)
Because previous definitions of intelligence had been lacking, they thought it could not be defined tidily. This led to avoiding premature answers. They believed the field of AI research was sick; this led to studying cognitive science. Errors which lead to studying more are better errors. (My Best And Worst Mistake) They regarded regulation of technology as bad, and this reduced attention to existential risks. When convinced risks existed, rather than reviewing mistakes, they just decided we needed AI first. (Raised In Technophilia)
They were good at refuting arguments, and felt they were winning the debate on whether intelligence implied morality. They had a rationale for proceeding with their best ideas, without resolving confusion. Reality does not care whether you are using your best ideas. You can’t rely on anyone giving you a flawless argument, and you can’t work around underlying confusion. (A Prodigy Of Refutation, The Sheer Folly Of Callow Youth)
An incongruous thought, coupled with some perfectionism, and viewing less than morally upright interactions as unacceptable, led to investigating seriously. Doing that, regardless of reason, led to pursuing a backup plan. (That Tiny Note Of Discord) That they were pursuing a backup plan gave them a line of retreat for their earlier views, but they only shifted gradually, without acknowledging fundamental errors. (Fighting A Rearguard Action Against The Truth)
They only saw the error when they realised that a mind was an optimisation process which pumps reality towards outcomes, and you could pump towards any outcomes. (My Naturalistic Awakening) They realised that they could have unrefuted arguments, and nature could still kill them if the choice was wrong. Their trust in following patterns broke, and they began studying rationality. (The Magnitude Of His Own Folly) We all need to lose our assumption of fairness. (Beyond The Reach Of God) They realised that an idea seeming very good didn’t permit being sure; it needed to be provably equivalent to any correct alternative, like Bayesian probability. (My Bayesian Enlightenment)
They recognise that there are people more formidable than them, and hope that their writings might find a younger one of them who can then exceed them. (The Level Above Mine)
Becoming Stronger Y: Challenging The Difficult
Wanting to become stronger means reacting to flaws by doing what you can to repair them rather than with resignation. Do not ritualistically confess your flaws unless you include what you intend to do about them. (Tsuyoku Naritai! (I Want To Become Stronger)) If you are ashamed of wanting to do better than others, you will not make a real effort to seek higher targets. You should always reach higher, without shame. (Tsuyoku Vs The Egalitarian Instinct)
The difference between saying that you are going to do something, and that you are going to try to do something, is that the latter makes you satisfied with a plan, rather than with success, and allows the part where the plan has to maximise your odds of success to get lost. Don’t try your best; either win or fail. (Trying To Try) People don’t make genuine efforts to win even for five minutes. (Use The Try Harder, Luke)
A desperate effort is a level above wanting to become stronger, where you try as though your life were at stake. And there is a step above that, an extraordinary effort; it requires being willing to go outside of a comfortable routine, tackle difficulties you don’t have a mental routine for, and bypass usual patterns, in order to achieve an outcome that is not the default that you care greatly about. It is riskier than even a desperate effort. (Make An Extraordinary Effort)
A problem being impossible sometimes only means that when we query our brain for a strategy, we can’t think of one. This is not the same as being proven to be impossible. Genuine effort over years can find routes forward. Reality can uncaringly demand the impossible. We should resist our urge to find rationalisations for why the problem doesn’t matter (On Doing The Impossible), and sometimes we should shut up and do the impossible; take success at the impossible as our goal and accept nothing less. (Shut Up And Do The Impossible!)
We need to ask ourselves what we want, what it will require to accomplish, and set out to do it with what we know. (Final Words)
Becoming Stronger Z: The Craft and the Community
The prevalence of religion, even in scientific circles, warns us that the baseline grasp of rationality is very low. Arguing against religion specifically fails to solve the underlying problem. We should also be trying to raise the sanity waterline. (Raising The Sanity Waterline)
A reason that people don’t want to learn more about rationality is that they don’t see people who know about it as happier or more successful. A large part of this is that even the people who know a lot about it still know very little, compared to experts in other fields; we have not systematised it as a field of study, subject to large-scale investment and experimentation. One reason for this is that traditional rationalists/skeptics do not see lack of visible formidability and say that we must be doing something wrong. We treat it as a mere hobby horse. (A Sense That More Is Possible) It can take more than an incremental step in the direction of rationality to get an incremental increase in winning. (Incremental Progress And The Valley)
Martial arts dojos suffer from epistemic viciousness; a treatment of the master as sacred, exaltation of historic knowledge over discovery, a lack of data, and a pretense that lack of data isn’t a real problem. Hypothetical rationality dojos risk the same problems. (Epistemic Viciousness) If an air of authority can substitute for evidence, traditions can proliferate and wield influence without evidence. (Schools Proliferating Without Evidence)
Verification methods can be stratified into three levels. Reputational verification is the basic practice of trying to ground reputations in some real world or competitive performance. Experimental verification is randomised, replicable testing, although this can involve very simple measures that are only correlated with the variables of interest. Organisational verification is that which, when everyone knows the process, is resistant enough to gaming to continue working. (3 Levels Of Rationality Verification)
Groups which do not concern themselves with rationality can praise agreement, encourage the less agreeing to leave, and enter an affective death spiral, which binds them all together and makes them cooperate. Typical rationalist groups do not cooperate; they speak and applaud disagreement but not agreement. If you are outperformed by irrational groups, then you are not rational, because rationality is about winning. Actual rationality should involve being better at coordinating, and we should work out how to be. Being half a rationalist is dangerous. (Why Our Kind Can’t Cooperate, Bayesians Vs Barbarians) Until atheist groups can outperform religious groups at mobilisation and output, any increase in atheism is a hollow victory. (Can Humanism Match Religion’s Output?) We need new models of community to replace the old, with new goals. (Church Vs Taskforce)
Do not punish people for being more patient than you; you should tolerate tolerance. (Tolerate Tolerance) We incentivise groups to improve by rejecting joining them if they don’t meet our standards. The non-conformist crowd tends to ask way too much. If joining a project is good, you should do it if the problems are not too distracting, or if you could fix the problems. If you don’t see a problem as worth putting in the time to fix, it is not worth avoiding a group for. If we want to get anything done, we need to move in the direction of joining groups and staying in them. (Your Price For Joining)
Many causes benefit from the spread of rationality. We should not think of other good causes as in competition for a limited pool of reasonable thinkers, but instead cooperate with them to increase the number of reasonable thinkers. We should think of ourselves as all part of one common project of human progress. (Rationality: Common Interest Of Many Causes) We are very bad at coordinating to fulfil aligned preferences of individuals. Large flows of money tend to be controlled by the incentives of organisations. (Helpless Individuals)
Donating time is inefficient compared to donating money. Allocating money is how we allocate resources. Money is the unit of caring. If you’ll never spend it, you don’t care. (Money: The Unit Of Caring) We enjoy having done kind things, but the things that bring us enjoyment often do much less good than calculated effort, and enjoyment and social status can be had much cheaper when you don’t try to achieve them through your giving. Get enjoyment, status, and results separately; purchase fuzzies and utilons separately. (Purchase Fuzzies And Utilons Separately)
The bystander effect is a bias in which a group is less likely to react to an emergency than a single individual. (Bystander Apathy) This applies to problems encountered over the Internet, where you are always observing them as part of a group of strangers. (Collective Apathy And The Internet)
When we write advice, we are not working from universal generalisations, but surface level tricks. This means it validly works for some people but not others. We should beware other-optimising, because we are not good at knowing what works for others, and beware assuming that other people are simply not trying what worked for us. (Beware Of Other-Optimizing) Practical advice based on established theories tends to be more generalisable. (Practical Advice Backed By Deep Theories)
The danger of underconfidence is missing opportunities and not making a genuine effort. Sticking to things you always win at is a way smart people become stupid. You should seriously try to win, but aim for challenges you might lose at. When considering a habit of thought, ask whether it makes you stronger or weaker. (The Sin Of Underconfidence)
There is more absent than present in these writings. Defeating akrasia and coordinating groups are particular absences. But, hopefully, there is enough to overcome the barriers to getting started in the matter of rationality without immediately going terribly wrong. The hope is that this art of answering confused questions will be enough to go and complete the rest. This will require drawing on many sources, and require having some specific motivating goal. Go forth and create the art, and return to tell others what you learned. (Go Forth And Create The Art!)