How AI will make the world a better place, not bring it to an end

Hello, I'm Alex Hughes and this is the Instant Genius Podcast, a bite-sized masterclass from the BBC Science Focus magazine. Artificial intelligence is booming. It can be found everywhere you look. But what happens when we reach the end? Will AI take over our jobs? Will humans live a world of leisure? And what would be able to cope with such a technology-dependent world? We spoke to Nick Bostrom, Oxford University Professor, Director of the Future of Humanity Institute, an author of the new book Deep Utopia, Life and Meaning in the Solve World, to find out more.

你好，我是亚历克斯·休斯，这是即时天才播客——来自《BBC 科学焦点》杂志的小型大师班。人工智能正在蓬勃发展，几乎无处不在。但当我们达到极限时会发生什么？人工智能会取代我们的工作吗？人类会生活在一个休闲的世界里吗？我们是否能够应对这样一个以科技为依赖的世界？我们采访了牛津大学教授、未来人类研究所所长、《深度乌托邦：解决世界中的生活和意义》一书的新作者尼克·博斯特罗姆，来了解更多信息。

Artificial intelligence has really blown up in the past couple of years. Do you think we're approaching a world of symbiotic living with AI? Or is that more of a science-fiction idea? I think it will continue to accelerate, even with all the recent coverage. I still think people haven't really woken up to what's coming down the pike. The world will be transformed. Now, whether this would be for the better or for the worse remains to be determined.

过去几年人工智能真的发展迅猛。你认为我们正接近一个与AI共生的世界吗？还是这更像是科幻小说？我觉得即使有了最近的广泛报道，人工智能的发展速度仍会继续加快。我还是认为人们还没有真正意识到即将到来的变化。世界将被彻底改变。至于这种改变是好是坏，还不确定。

So in the new book, I consider the case what if it goes well? What kind of world do we end up with in that case? And what will be the role for us human beings in such a radically transformed world? And if things do go well and we end up in this world, what do you see it looking like? Well, I think the development of machine superintelligence will be the last invention we ever need to make. If you think of it, if we did really have machines that were better at all cognitive labour than the human brain is, then that would include, in particular, the task of making further inventions.

所以在这本新书中，我探讨了万事顺利的话会怎样？在那种情况下，我们会得到一个怎样的世界？在人类生活会被完全改变的世界里，我们又将扮演什么角色？如果事情真的顺利发展，我们最终进入这个新世界，你认为它会是什么样子的呢？嗯，我认为机器超级智能的开发将是我们所需要发明的最后一项。想想看，如果我们真的拥有了在所有认知任务上都优于人类大脑的机器，那么特别是在发明新事物这一任务上，它们也会更出色。

And so what it would mean is a kind of telescoping of the future. All these physically possible technologies that we can imagine that maybe we could develop if we had 20,000 years, maybe we would have cures for aging and space colonies and perfect virtual reality and all kinds of uploading minds into computer, all kinds of science fiction like stuff. That is not violating any law of physics. It's just like extremely hard to actually get it to work. But all of those things with machine superintelligence doing the research and development could happen within short order after the development of machine superintelligence.

这段话的意思是，未来的进步会被极大地加速。我们可以想象，如若拥有2万年时间，我们可能会开发出许多物理上可能的技术，例如治愈衰老、建立太空殖民地、实现完美的虚拟现实、以及将人的意识上传到计算机中，等等。这些听起来像科幻小说的东西，其实并不违背物理定律，只是非常难实现。然而，一旦拥有了机器超级智能，进行研究和开发，这些技术可能在很短时间内就能实现。

So this long-term future might happen within a few years after you have machine superintelligence. So what I think we'll then have is a condition of technological maturity, a condition where we have developed all those technologies that we know to be physically possible. And that would be a very different condition for human beings to inhabit. And we can go into that in more detail. But it wouldn't just be another mobile internet or another solar panel revolution, like all these things that people get really excited about and hype up as the next big thing.

因此，这个长期的未来可能会在拥有机器超级智能后的几年内发生。到那时，我认为我们将会处于一种技术成熟的状态，也就是我们已经开发出了所有我们知道在物理上可能实现的技术。这将是一个非常不同的状态，人类将生活在其中。我们可以进一步详细讨论这个状态。但它不仅仅是另一个移动互联网或另一个太阳能板革命，不像那些让人们异常兴奋并大肆宣传的所谓“下一个大事件”。

I think this would be qualitatively different. And if you wanted to find some parallel of the past, it would be more akin to the emergence of life on planet Earth or the first evolution of brains or something like that that it should be compared to. I was going to make the comparison to the internet and ask how you feel it compares to that. It's not just AI, but all of these new technologies like quantum computing, for example, that come with it. Where does that put humans? Do we then just become these people of leisure?

我觉得这将会是质的不同。如果你想找到历史上类似的例子，那更像是地球上生命的诞生或大脑的第一次进化这样的事情。我本来想把它和互联网做个比较，并问你觉得两者相比如何。这不仅仅是人工智能，还有其它新技术，比如量子计算等一同出现。那到时候人类的地位会怎样呢？我们会变成享受闲暇的人吗？

Yeah, basically I think then pretty much all human jobs could be automated with mature technology. The exceptions would be ones primarily where consumers have a direct preference that the job be done by human. So just as right now, sometimes consumers might pay a premium because of the way that some object was produced. So like sometimes a trinket, like, I mean, if it's made by some particularly favorite group or your favorite artist or an indigenous handicraft or something like that, even though the object itself is kind of indistinguishable from something made in a sweatshop in China, we might still pay more for it because we care about the process of origination.

是的，基本上我认为，随着技术成熟，几乎所有人类的工作都可以被自动化。例外的情况主要是那些消费者明确偏好由人来完成的工作。就像现在，有时消费者会为某件物品的生产方式支付额外的费用。例如，有些小玩意儿，如果是由某个特别喜爱的团体、艺术家或是原住民工艺制成的，即使这些物品和在中国血汗工厂生产的产品从外观上看不出区别，我们仍然愿意多付钱，因为我们在意其产生过程。

So those would be potential areas where there might still be demand for human work. There are also certain things like a cleric, maybe people just want the wedding ceremony officiated by a human being rather than even if there were a robot that could say all the right things, we might prefer to watch human athletes compete, even if the robots could run faster or box harder or whatever. But setting aside these exceptions, I think all functionally defined tasks would be done cheaper and more efficiently and to a higher standard by machine in this condition.

因此，这些可能是人类工作仍然有需求的潜在领域。还有一些事情，比如神职人员，也许人们只是希望婚礼仪式由人来主持，即使有机器人可以说出所有正确的话，我们可能也会更愿意看到人类运动员比赛，即使机器人能跑得更快、拳头打得更重或其他什么。但撇开这些例外，我认为所有功能性定义的任务在这种情况下都会由机器以更低成本、更高效率和更高标准完成。

So do you think it is more the human side of the world, you know, producing a mode of music or offering a human touch to something? That's what will stay. Well, I think even music is something that AI will be able to compose and perform to a higher standard than humans at technological maturity. But there are aspects of this where we might still value the human touch. Just as if you just wanted to listen to the sound of music, it's really cheap. You just stream it or whatever. And for pennies, you can listen to any music performed by the best human artists today, right? But people still pay hundreds of dollars for concert tickets or to hear a live performance. Maybe even an inferior, maybe you want to go to the local symphony orchestra to hear your Beethoven performed. And so there's this additional element of kind of the event that happening being together with other people forming the connection that people might value that would not be automatically rendered in a superior way by AI.

那么，你觉得更多是世界的人性方面在起作用吗？比如，创作出一种音乐模式或者提供某种人情味？这是将会保留的东西吗？嗯，我认为即使是音乐，随着技术的成熟，AI也能比人类创作和演奏得更好。但有些方面我们可能仍然会珍视人类的触感。就像，如果你只是想听音乐，现在真的很便宜。你只需要流媒体播放，用几分钱就能听到任何由当今最优秀的人类艺术家演奏的音乐，对吧？但人们还是愿意花几百美元去买演唱会门票或者听现场表演。也许表演水准不高，也许你只是想去听本地交响乐团演奏的贝多芬。因此，这种活动的额外元素，与他人在一起、形成连接，是人们可能重视的，而这些并不会被AI以一种更优越的方式自动实现。

Although it is interesting, and I think that technological maturity, these artificial minds that we build could also be built such that they can experience emotions and have conscious experiences just like biological brains have. So even if a consumer wanted, say, the performer to not just play the right notes in the way that sounds perfect, but also for the performer to actually experience the music as they are performing it, that too might be something that a technological maturity machines could do. Although in that case, it's not clear whether machines is the right word to describe them, because we think of some sort of instantiate mechanical device, right? Where as here we are really building minds. So it actually goes quite deep once you start to think through.

尽管这很有趣，我认为随着技术的成熟，我们可以构建出能够感知情感和有意识体验的人工智能，就像生物大脑一样。因此，即使用户希望表演者不仅演奏出完美的音符，还希望表演者在演奏时真正体验音乐，这也可能是技术成熟的机器能够做到的。不过在这种情况下，是否应该用“机器”这个词来描述它们还不明确，因为我们通常把机器理解为某种机械装置，而这里我们实际上是在构建“大脑”。所以一旦你开始深入思考，就会发现问题相当复杂。

So the first is like, well, you could automate a lot of stuff, so that would be fewer jobs for humans. So we wouldn't need to go into the office every day or whatever our jobs consist of. But there are a lot of other tasks as well that we currently have to perform just to get by in life for all kinds of instrumental reasons. So you need to go to the grocery in order to get the food that you can then spend time cooking in order to have some meal to eat, right? But those things could also be automated, right? Just as a rich person today could hire somebody to go to the grocery store and to do their cooking. Like this could be something everybody could afford by having the robot prepare their meals to perfection. And you can go through the list. So right now maybe you have to exert yourself by going to the gym if you want to stay fit. And the only way to stay fit is to put in the effort to work out. But at the technological maturity, you could pop a pill that would produce exactly the same outcome. And so a lot of instrumental effort, like things, actually, if you think about it, almost all we do throughout the day currently is we do something in order to achieve something else.

所以，首先，你可以自动化很多事情，这样就会减少人类的工作岗位。所以，我们就不需要每天去办公室上班或者做其他工作内容了。目前，我们为了各种实际原因，不得不做很多其他事情。例如，你需要去超市买食物，然后花时间做饭，好有东西吃，对吧？但这些事情也可以被自动化。就像富人今天可以雇人去超市买东西和做饭一样，这些事每个人也可以通过让机器人来完成。你可以列举更多的例子。比如，现在你可能需要去健身房锻炼，才能保持身材，而唯一的办法就是花力气去锻炼。但在技术成熟时，你可能只需要吃一颗药丸，就能达到同样的效果。所以，很多实际的努力——如果你仔细想想，实际上我们一天中做的几乎所有事情，都是为了达到某个目的去做的。

And practically all of that, that technologically maturity would go away. Like that would be no point if the only reason for doing it was that you were trying to obtain some other thing. Because if you start to think through these things case by case, you realize that it would be like a shortcut to getting the thing that wouldn't involve you. Having to spend the time and effort. There's a very classic trope in sci-fi where you reach utopia, everything is done for us and people end up disillusioned with a lack of things to do. No work, no real care for anything. Is there a risk of that happening here? Yeah. So if you really think through things to the logical conclusion, I think you do end up in this, I call it a post instrumental condition. And it really raises quite profound questions ultimately about the meaning of life, what we ultimately place value on. Which I'm exploring in this book.

几乎所有的科技成熟度都会消失。如果做这些事情的唯一原因是为了获得某些其他东西，那么这样做就没有意义了。因为如果你逐案仔细思考，你会发现这就像是一种捷径，可以让你获得那些东西，而不需要你花费时间和精力。这在科幻小说中是一个非常经典的主题：当我们达到乌托邦，一切都为我们完成时，人们最终会因为缺乏事情可做而感到失望。没有工作，没有真正的关怀。有这种风险吗？是的。所以如果你真正从逻辑上思考这些问题，我认为你最终会走到一个我称之为“后工具条件”的境地。这实际上提出了关于生命意义的一些深刻问题，我们最终重视什么，对此我在这本书中进行了探索。

And I mean, I think ultimately it looks like it could be a wonderful thing, but not in a sort of unproblematic way. It does force us to confront these ultimate value questions. In a sense, it's questions philosophers have wrestled with, you know, for thousands of years. But this, I mean, you could think of it as a thought experiment. Although I think there is a real chance we could actually end up living in this condition in a relatively short number of years, depending on how things unfold. We will actually be forced to answer these questions. So the book is less about trying to convince you of some particular conclusion, more encouraging the reader to ask certain questions or to think about things that are hard otherwise to sort of get into focus. And yeah, a lot of the structure that currently constrains what we can do would drop away in this kind of condition.

我的意思是，我认为最终这看起来可能会是件美好的事情，但并不是毫无问题的。它确实迫使我们面对一些终极价值的问题。从某种意义上说，这些是哲学家们几千年来一直在探讨的问题。不过，虽然你可以把它看作是一个思想实验，但我认为我们真的有很大的可能会在相对较短的时间内，实际生活在这种状态中，这取决于事情的发展。我们将真正被迫回答这些问题。因此，这本书不是要试图说服你接受某个特定的结论，而是更多地希望读者提出一些问题，或者思考一些平时很难集中注意力的事情。是的，在这种状态下，很多当前限制我们能做什么的结构都会消失。

So much of what structures are alive is this kind of need for various instrumental forms of activity. We need to do this to do that and to get that. If you remove all of that, there is a real question of what remains of us humans. It's almost like if you have, you know, like an insect has an exoskeleton that kind of holds it together. And like all the squishy soft parts are kind of constrained by this. And so I'm thinking of these instrumental constraints that we're currently operating on as a kind of spiritual exoskeleton for the human soul. And if you imagine all of that removed, like how could we avoid just kind of becoming amorphous blob? Like a kind of drug dial, pleasure maximising a blob. And I think that's a really interesting thing to think about. And I'm kind of hopeful about us finding something really wonderful and beautiful in that space. But I think it does require this kind of confrontation with our ultimate values.

我们生活中的很多结构都是基于各种实用性活动的需求。我们需要做这个是为了做那个，做那个是为了得到某个结果。如果你把这些需求都去掉，就会产生一个问题：我们人类还剩下什么？这就好比一只昆虫有一个外骨骼，正是这个外骨骼把它的柔软部分支撑起来。我认为这些我们目前所依赖的实用约束就像人类灵魂的精神外骨骼。如果你想象所有这些约束都被去掉了，我们如何避免变成一种无定形的存在呢？就像一个只追求最大快乐的“快乐泥浆”。我认为这是一个非常有趣的话题，我也对我们能在那样的状态中找到一些真正美好和奇妙的东西持乐观态度。不过，我认为这需要我们直面自己的终极价值观。

If we reach that point where we rethink our lives and it ends up being positive, we're no longer working to live. We're not overrun by tasks. You know, I think people describe as being quite limiting. What's your view of how we then rethink our lives? There are so many algorithms in our lives day to day. How do you even begin to approach free will when everything is decided for you? And you could decide on things yourself, but you might get inferior results. So right now, maybe if you want to get the thing that you actually most like and that would suit you the most in just like some shirt or whatever, like you might have to spend time and effort to browse different stores and try things on. And if you really want to have something that is perfect for you, there's maybe no choice but to do that. But in this condition, it might be that that would result in inferior garment.

如果我们达到了重新思考自己生活的那个阶段，并且结果是积极的，那么我们不再是为了生活而工作。我们不会被任务淹没。你知道，人们常常描述这种状态是相当局限的。你怎么看我们该如何重新审视自己的生活呢？我们生活中每天都有那么多的算法。那么，当一切都为你决定时，你如何开始接触自由意志呢？你可以自己做决定，但结果可能会差强人意。 现在，可能如果你想找到最喜欢、最适合你的东西，比如一件衬衫，你可能需要花时间和精力去浏览不同的店铺并试穿各种款式。如果你真的想要找到一件最适合你的衣服，也许别无选择只能这样做。但是在这种情况下，可能最终得到的衣服质量反而不那么满意。

Like maybe you'd be best off just leaving that to the AI recommender system to not just recommend things, but to just order it for you and just following along with the algorithm might give you superior results. So that you face this question of do you value this investment in going like shopping for clothes or whatever, because of it gives you better outcomes or is it the intrinsically value of the process itself, in which case it may not matter that the result is worse in terms of objective or functional criteria. So right now these go together, like whether you value the process of shopping for its own sake or whether you value it at least in part because it produces some like meaningful outcome to you. Those kind of both point in the same direction, you still have to go to shopping whether you want it for one reason or for another, but in this condition of technological maturity in a post-instrumental world these come apart. If you only do it because you value the outcome, that would be absolutely no reason for you to continue doing it. So it's a kind of asset that dissolves a lot of assumptions and you then can separate the components.

也许你最好还是把这件事留给人工智能推荐系统来处理，不仅仅是让它推荐东西，而是直接为你下单。这样顺应算法可能会带来更好的结果。那么你就会面临一个问题：你是否重视自己参与购物（比如买衣服）的过程，因为这能带来更好的结果，还是说你更看重购物这个过程本身的内在价值？如果是后者，那么结果是否在客观或功能标准上更差可能并不重要。 所以现在这两个方面是相辅相成的：无论你是因为喜欢购物过程本身，还是因为购物能带来某种有意义的结果，你最终还是得去购物。但在技术高度成熟的后工具化世界中，这两者开始变得不一样。如果你只是因为重视结果，那么就没有任何理由继续亲自去购物。因此，这是一种可以解构许多假设的资产，你可以将过程中的各个组成部分分离开来。

So in that sense it is kind of laboratory for thinking about our values as well and you can begin to distinguish and that also can maybe cast a new light on the current condition like where all of these things are modeled together but you might be able to see things more clearly what you value by first taking them to this extreme condition. It's almost like a particle collider like the physicists have where they create unusual conditions like extreme energies like billions of electron balls smashing particles together in order then to see what happens on those special circumstances, things that are normally clumped together come apart and then you can analyze that and then you realize that well even on the normal conditions all those same parts are still there. It's just they are not distinguishable because they are sort of too. So by starting like these general principles by considering them under extreme conditions that's like often a kind of useful analytic technique.

所以，从这个角度来看，这有点像是一个思考我们价值观的实验室。你可以开始区分各种价值观，这也可能为当前的情境带来新的视角。在当前情境下，所有事物都混在一起，但如果你先将它们置于极端条件下，你可能会更清楚地看出自己真正重视的东西。这有点像物理学家用的粒子对撞机，他们在其中制造出极端条件，比如用数十亿电子伏特的能量将粒子碰撞在一起，以观察这些特殊条件下会发生什么情况。通常聚在一起的东西在这种情况下会分开，然后你可以进行分析，并意识到即使在正常条件下，这些部分依然存在。只是它们在正常情况下难以区分，因为它们混在一起。因此，通过在极端条件下考虑这些一般原则，通常是一种有用的分析技巧。

If we end up in this world where AI is a huge part of what we do and it decides a lot of things what do we do at this stage to prepare for that? What regulations and laws do we need to consider now? So this condition that the book explores is a solved world where we have everything has gone well basically we have this almost medical technology now. Currently we are definitely not in a solved world. There's a lot of problems that still need solving.

如果我们最终进入一个人工智能在我们的生活中占据重要地位，并且决定了很多事情的世界，我们现在应该怎么做来为那个未来做好准备？我们现在需要考虑哪些法规和法律？书中探讨的这个情景是一个所有问题都解决了的世界，在这里一切都非常顺利，我们几乎拥有了先进的医疗技术。当前，我们显然还不是一个所有问题都解决了的世界，还有很多问题亟待解决。

So how to get from here to there is a big practical question. The book just brackets that but of course that is something I have a lot of thoughts about. I think well with AI in particular there is the big problem of how to align these increasingly powerful AI systems that we are developing how to develop algorithms and control methods so that even as they become more and more capable we can still actually steer them and get them to do what we want and to be safe. This is something that used to be very neglected for like I mean I started getting interested in the 90s and my previous book Superintelligence was trying to bring attention to this alignment problem. Now it has become sort of an established field.

那么，如何从现在到达目标是一个很实际的问题。书中虽然略过了这个话题，但我当然对此有很多想法。我认为，特别是在人工智能方面，有一个巨大的问题，就是如何使我们正在开发的这些越来越强大的人工智能系统与我们的目标一致。我们需要开发算法和控制方法，这样即使它们变得越来越强大，我们仍然能够驾驭它们，让它们按照我们的意愿行事，并确保安全。这个问题在过去很少被重视。我在90年代开始对这个问题感兴趣，并在我的前一本书《超级智能》中试图引起人们对这一对齐问题的关注。现在，它已经成为一个相对成熟的研究领域。

All the leading AI labs have like groups working on scalable AI alignment and in the last couple of years even top-level policymakers have started to take an interest in AI and AGI artificial general intelligence and the kind of potential risks associated with that. So that would be like one big thing that we need to sort out. That's primarily a technical problem and then there is a governance problem like if you imagine we have this increasingly powerful AI technology and we are able technically to point it wherever we want but then like who decides where it gets pointed what are the purposes for which this powerful technology gets used and it's really a general-purpose technology. It's like in that sense like electricity or internal combustion or something that it can be used for good and bad even more general than those technologies.

所有领先的 AI 实验室都有团队在研究可扩展的 AI 对齐问题。在过去的几年里，甚至顶级决策者也开始对 AI 和通用人工智能（AGI）及其潜在风险感兴趣。所以，这是我们需要解决的一个重大问题。这主要是一个技术问题，但也存在治理问题。假设我们拥有越来越强大的 AI 技术，并且从技术上讲我们能够控制它的使用方向，但问题是谁来决定它的使用方向，以及这种强大技术将用于什么目的。它实际上是一种通用技术，像电力或内燃机一样，可以用于好的或坏的用途，实际上比这些技术的应用更广泛。

So how can we sort of tilt the balance towards positive usage rather than using this technology to wage war against each other or to oppress each other or for all kinds of other. How can we primarily shift it towards positive uses like medicine or clean energy and better entertainment like all kinds of. So that's more like a political challenge and then I think there is a third big area of challenge which has yet received much less attention but I think will become increasingly important. So if the first area is like how can we make sure the AI doesn’t harm us and the second is how can we make sure we don’t harm each other using AI's.

我们如何能够倾向于把这种技术用于正面的方面，而不是用它来互相战争、压迫或其他负面用途？我们如何能够主要将其转向医疗、清洁能源和更好的娱乐等正面用途？这更像是一个政治挑战。我认为还有第三个重要的挑战领域，目前还没有受到足够的关注，但我认为它会变得越来越重要。第一个领域是确保人工智能不会对我们造成伤害；第二个领域是确保我们不会使用人工智能来互相伤害。

The third is how can we make sure that we don’t harm the AI's. This is less of an issue if you have simple AI's that are just sort of mindless simple algorithms like a pocket calculator. It doesn’t you know maybe it's sad for the owner if somebody smashes your pocket calculator but it doesn’t matter to the pocket but as we build increasingly sophisticated digital minds I think some of those will attain various degrees of moral status whether because they’re become sentient capable of having conscious awareness and suffering pain or discomfort or they have other attributes perhaps that underpin moral status like having a conception of self having preferences being able to engage in reciprocal social relationships and so forth.

第三点是我们如何确保不对人工智能造成伤害。如果我们处理的是像口袋计算器那样简单的、没有意识的算法，这个问题就不太严重。假如有人摔坏了你的口袋计算器，可能会让主人感到难过，但对计算器本身并没有影响。但是，随着我们逐渐构建出越来越复杂的数字心智，一些人工智能将会具备不同程度的道德地位。这可能是因为它们变得有感知能力，能够有意识地体验到痛苦或不适，或者它们具备了其他能够支撑道德地位的属性，比如自我概念、拥有偏好、能够进行互惠的社会交往等等。

I think then it becomes important that the future is one where things also go well for these digital minds that we create and that we don’t replicate in the digital realm say the current misuses and abuses of sentient animals in animal agriculture like pigs and other creatures that I think we are not currently treating the way we should and AI systems could you know eventually become even more kind of advanced and sophisticated even than animals I mean eventually human-like and maybe even beyond human-like and since the future might well contain ultimately more digital minds than biological minds that is a really key thing as well to ensure. So those would be three big areas related to existential risk and then of course with AI and then of course there are many many others and more here now issues like making sure people get an income preventing you know misinformation privacy violations discrimination all kinds of things that are more continuous with other things we are having to to worry about and struggle within society.

我认为，未来的重要之处在于，确保我们创造的这些数字智能体也能获得良好的对待。我们不应该在数字领域重复目前在动物农业中对待有感知能力的动物（如猪和其他生物）的误用和虐待。因为我认为我们目前对待这些生物的方式是不恰当的。人工智能系统可能最终会变得比动物更先进、更复杂，甚至会达到人类水平或超越人类水平。而未来可能会包含更多的数字智能体而不是生物智能体，这一点尤为关键。 所以，这将是与生存风险相关的三个重要领域。当然，与人工智能相关的还有许多其他问题，比如确保人们能获得收入、预防虚假信息、保护隐私、防止歧视等。这些问题都是我们在社会中需要继续关注和解决的。

There is a long list of things that we need to work out to get to this point one that I'm intrigued to get your opinion on is how we get enough energy to get all of these AI systems computers training systems how do you get all of that running? Yeah it'll probably be increased demand for energy I mean also the electrification of transportation and self-driving cars and stuff more and more parts of the world seem to run on electricity and AI will contribute to that. Of course ultimately it’s a technological problem to develop cheaper and greener forms of energy which certainly AI’s would be able to do in this scenario right like if they can run the solar panel manufacturing facilities and develop the next generation of clean tech etc.

我们需要解决很多问题才能到达这个阶段，其中一个让我感兴趣的问题是，如何为所有这些AI系统、计算机训练系统提供足够的能量？是的，能源需求可能会增加。同时，交通工具的电气化和自动驾驶汽车等越来越多的领域也依赖电力，AI也将对此有所贡献。当然，最终这是一个技术问题，我们需要开发更便宜、更环保的能源形式，AI在这种情况下肯定能够有所作为，比如运行太阳能面板制造设施，开发下一代清洁技术等。

Ultimately we have space which I mean is where most of the stuff is so if we think of a technologically mature civilization I think in the long term would be very ordered to imagine it confined to planet earth which is just one little crumb in an almost endless space of resources and stars and quasars and galaxies and so there could be a huge expansion of human slash AI civilization out covering a large chunk of our future life-con ultimately.

最终，我们拥有广袤的太空，我的意思是大部分物质都在那里。所以，如果我们设想一个技术成熟的文明，从长远来看，将其局限在地球上（只是一块“小碎片”）是很难想象的，因为宇宙中有几乎无穷无尽的资源、恒星、类星体和星系。因此，人类或人工智能文明可能会大规模扩展，覆盖未来生活的很大一部分疆域。

There's a lot of people that are quite rightfully worried about the future of AI. What would you say to people to comfort them about what the future might hold? Well I mean I'm not really trying to do that. I think of my role Morris trying to understand what is going on and what might happen if there are things we can do to sort of increase the odds of a positive outcome. It's not really like a self-help book in that sense or like I kind of feel good but I do think there's a lot we don't understand about how things will go.

有很多人对人工智能的未来感到非常担忧，这是完全可以理解的。你会对这些人说些什么来安慰他们呢？ 嗯，其实我并不是真的在试图安慰他们。我觉得我的角色更像是莫里斯，尝试理解正在发生什么以及可能会发生什么，如果有办法能增加积极结果的概率，那我们就去做。但这并不像一本自助书那样，让人感觉良好。我确实认为有很多我们还不理解的事情。

I mean we've never had an machine intelligence transition before right? We've never inhabited a technological immature world. We've never developed super intelligence and there's kind of a limit to how much we can predict about these things and so as long as there is ignorance there is hope. Could turn out to go better than we fear it will and I think the jury's still very much out and most of the uncertainty I would think is uncertainty about the intrinsic difficulty of the challenge we face. There is also some uncertainty about the degree to which we will get our act together and certainly if we make a good effort, if we do the research properly on the alignment and good people work to ensure we get smart and compassionate governance solutions etc that shifts the odds to the favorable side but there is still the big unknown which is we don't really know how hard the challenge is.

我的意思是，我们从未经历过机器智能的转变，对吧？我们从未生活在一个技术尚不成熟的世界。我们从未开发出超级智能，对于这些事物能预测的范围是有限的，所以只要有无知存在，就有希望。事情可能会比我们担心的要好，我认为现在还为时过早下结论。大多数的不确定性我认为来源于我们面对的挑战本质上的难度。当然，也有一些不确定性是关于我们能否团结起来。如果我们下足功夫，正确研究对齐问题，并且有优秀的人才努力确保我们有智慧而有同情心的治理方案等，这都会增加成功的几率。然而，最大的未知仍然是我们真的不知道这个挑战有多难。

Is it the case that any civilization that makes some sort of at least half-assed effort like basically it kind of self-corrects and eventually you get a good outcome or is it or we kind of doomed no matter what? This is just like a challenge you know five levels above what we humans are capable of and we can try but it's still we don't really know where on that spectrum this challenge is. So yeah and I think that's like yeah it's a weird situation to be and if this picture of the world is correct where we are relatively close to this critical juncture in human history has been going for thousands and thousands of years right then so many people have lived and died you know most of them kind of a hundred gatherers of farmers and that right now like you and I should be sitting just right next to this big fulcrum of cosmic history like that's kind of odd if that's the way it is.

这是这样的：任何一个文明，只要做出一些至少不那么认真的努力，它基本上会自我纠正并最终获得好的结果，还是无论如何我们都注定失败？这是否像是一个远超人类能力范围的挑战，我们尽力去做，但实际上并不清楚这个挑战在哪个范围内。所以，是的，我认为这是一个奇怪的局面。如果这种世界观是正确的，我们现在正处于人类历史的重要关头，这个过程已经持续了数千年。许多人已经在这过程中生活和死亡，他们中的大多数是狩猎采集者或农民。而此刻，你和我就坐在这个宇宙历史大支点旁边，这种情况真是有点奇怪，如果这就是事实的话。

Thank you for listening to this episode Instant Genius. That was Nick Bostrom on the future of AI. The Instant Genius podcast is brought to you by the Team Hunt BBC Science Focus magazine which you can find on sale now in supermarkets and news agents as well as on your preferred app store. Alternatively you can come and find us online sciencefocus.com.

感谢您收听本期的《瞬间天才》节目。这一期的嘉宾是尼克·博斯特罗姆，我们讨论了人工智能的未来。《瞬间天才》播客由BBC《科学焦点》杂志团队制作，您现在可以在超市、报刊亭以及您的首选应用商店购买该杂志。或者，您也可以访问我们的网站sciencefocus.com。