My guesses as to most common true biggest values are probably “continue life” and “be loved/be worthy of love”.

This doesn’t seem quite right. I think to the extent it’s a terminal value, which is less than I previously thought, “be loved” is less important than “continue life” in probably almost everyone, including at least most of the people I used to consider to have “be loved” as most root. But “be loved” is easily advanceable through social reality-bound optimization, whereas “continue life” mostly requires real optimization.

Apologies for the late reply.

It seems to me like the hero capture you describe – namely, hollow shells trying to convince a purehearted person that working with them/helping them/doing things for them is the best way to fulfill the purehearted person’s ideals – can be trivially evaded in one of two ways.

1) Nonstandard or neurodivergent thought processes. The process which you describe as hero capture involves the manipulation of a hero’s thought process to convince them to harness their energy in service of the capturer. Thought processes cannot be manipulated if they cannot be understood or seem alien or insane.

Artificial superintelligences, whether good or evil, cannot be captured and convinced that helping a particular individual or group of individuals is the best way to accomplish their goals. Aliens cannot be captured. I suspect that zealots who have passed a certain point of insanity or mental illness cannot be captured. The Time Cube guy probably cannot be captured. The folks who listened to Mundum in The Northern Caves cannot be captured.

If a hero adjusts their thought processes to become sufficiently neurodivergent, it becomes difficult to capture them because the psychological capturing tricks that are intended to exploit human psychology or cognitive bias no longer produce the expected results. Neurodivergence reduces their ability to make an impact on the world because it reduces their ability to understand human beings as well as generate social capital but less so than being fenced up and captured.

2) If a hero has sufficient self-confidence in their own methods of changing the world, they will not become convinced that some new method (eg. working with people attempting to capture them) is the best way to optimize for the values that they desire. Someone with a sufficiently high amount of self-trust is essentially blind to input from the outside world regarding what they should or should not do.

This can be achieved by inculcating high degrees of regular human arrogance, confidence, and mental techniques designed to block out stimuli from other people with regards to morality. It can also be achieved by tuning in to the voice of an egregore or alien god and refusing to listen to any other. It seems difficult to me to capture someone like Joan of Arc because there’s always a chance that the voices that she hears will lead her away from you and remind her that fulfilling your needs isn’t the best way to achieve her goals or to fulfill the will of God. Blind obedience to a set doctrine can have a similar effect when taken to a certain degree.

These two methods have a high degree of overlap.

I think this was, not-quite-the-right-word, but anthropomorphizing me. Treating my statements as being spoken to Schelling morality, rather than to good people.

As a very rough estimate, I think single good is about 1/20 of the population, and double good 1/400. Neutral people are society. And society is mostly made of a dynamic equilibrium of mutual epistemic damage.

I’m not trying to get nongood people out of society. I’m trying to partially remove good people from its effects.

To be clear, I am considerably, “darker” than what you might have in mind for a “true hero”, and all the connotations of social endorsement you might mean by that phrase are yours to decide if they apply to me. But my “darkness” does in fact come entirely from my determination to do good. I have a standard of justice and I follow it, and I don’t in fact attack people for selfish gain, even though this distinction is observed in retrospect over behavior that is the result of not drawing a distinction for a long time because I thought that society had fucked with my software to draw that distinction.

Another way of describing the thing I am. I’m Maiev Shadowsong from Warcraft if she had a phoenix from hpmor to nudge her away from being an idiot in certain ways. (Her standards are partially societal, she is not a hero contract hero; she spends 10000 years watching a prison for no compensation, her only solace knowing the world is safe from super criminals), and towards the actually big problems; fighting the legion and the old gods, not Illidan.

1) Yes. In fact, my former startup cofounders have called me those exact words, “true hero” and “slow fooming FAI”. The singlemindedness, the extremism, the objective moral standard, the disregarding what society thinks, the leaving my family behind (I don’t have children). I pursue vengeance upon the Shade; defined as the way of being of the universe such that bad things like death will happen and no one can do anything about them, with the sort of relentless absolute determination of a revenant. Or a Sith.

If you find any others, I would like to talk to them.

2) The “hero capture” I’m interested in is the kind that manages to grab true heroes. Human epistemology is not a “secure computer system”. I don’t think any epistemology can be. People trying really hard to adjust your estimates of how useful working with them are relative to accomplishing goals towards an absolute objective ideal can often to some extent succeed no matter how virtuous your choices made long ago.

Although. When I wrote this I was considering EA org leaders who never seemed to move on after their premise had been falsified, the startup energy and aliveness waning away as their ability to see the possibility of true optimization waned to be an example of this. And there is nuance to how “good” appears in most people, not me, that I didn’t understand when wrote these posts, they are, as best I’ve been able to examine individuals in that class, probably not the same kind of thing I am, such that they are sort of just choosing not to leave their social groups behind for normal human reasons. And that is still behaving “supererogatory”, so don’t consider it misbehavior, but it’s not my sort of determination.

3) Yes.

I think that a “true” hero (as distinct from the “hero” that is bound by society that you have described in this post) should act based on an objective moral standard – that they should be a kind of ultra-extremist that does what is right regardless of what society thinks about it. Concerned not with fulfilling materialistic impulses (such as being seen as desirable, or not wanting to abandon their friends) but single-mindedly marching towards an immutable ideal. More like an ASI than a human being, in terms of how blind and focused its actions are towards a single goal.

Do you think that:

1) “True heroes” as opposed to heroes as defined by society can actually exist?
2) If they do exist, do you agree that these “true heroes” would not be vulnerable to the hero capture methods described in this post, because they become invulnerable to the mental and social pressures that keep them staying in a stagnant little garden – to use your metaphor, because they are willing to abandon their own child to save the world? (If your knee jerk response to this is “that’s impossible, nobody can be like that in real life, it only exists in anime” then go ahead and answer “no” to question 1.)
3) People who intend to do good should strive to make themselves into true heroes?

I too miss your writing a lot. Best I’ve found online since TLP went under. Please keep writing.

Yes, big project.

I for one miss your insightful writing, hope you’ll come back and share more of your thoughts.