首页  >>  来自播客: Sub Club by RevenueCat 更新   反馈

Why You Should Test Everything and How To Do It — Osman Mansur, Duolingo

发布时间 2023-01-04 09:00:00    来源

摘要

On the podcast we talk with Osman about Duolingo’s culture of experimentation, data and testing as a moat, and why passive aggressive push notifications actually work in the right context.

GPT-4正在为你翻译摘要中......

中英文字稿  

Welcome to the subclub podcast. A show dedicated to the best practices for building and growing app businesses. We sit down with the entrepreneurs, investors and builders behind the most successful apps in the world to learn from their successes and failures. Subclub is brought to you by RevenueCat. Thousands of the world's best apps trust RevenueCat to power in app purchases, manage customers and grow revenue across iOS, Android and the web. You can learn more at RevenueCat.com. Let's get into the show.
欢迎来到Subclub播客，这是一档致力于分享建立和发展应用商务最好实践的节目。我们与全球最成功应用背后的企业家、投资者和开发人员坐下来，从他们的成功和失败中学习。Subclub是由RevenueCat提供支持的。全球数千个最好的应用使用RevenueCat为iOS、Android和Web推动应用内购买、管理客户和增长收入。您可以访问RevenueCat.com了解更多。让我们开始节目吧。

Hello, I'm your host David Barnard, and with me today, RevenueCat CEO Jacob Heide. Our guest today is Osmond Monser, a product manager on the retention team at Duolingo. On the podcast, we talk with Osmond about Duolingo's culture of experimentation, data and testing as a mode, and why passive aggressive push notifications can actually work in the right context.
大家好，我是你们的主持David Barnard，今天和我一起的是RevenueCat的CEO Jacob Heide。我们的嘉宾是Osmond Monser，他是Duolingo留存团队的产品经理。在这期播客中，我们和Osmond聊了Duolingo的实验文化、数据和测试作为一种模式，以及为什么在合适的情境中，被动式的推送通知可以实际发挥作用。

Hey, Osmond, thank you so much for joining us on the podcast today.
嗨，奥斯蒙德，非常感谢您今天加入我们的播客节目。意思是感谢奥斯蒙德参加播客节目。

David, thanks so much for having me. It's great to be here. Super excited to chat with you, Jacob. Good to see you. Happy Friday. I'm here. I'm ready to talk about subscriptions. Let's do it.
大卫，非常感谢您邀请我。很高兴来到这里。非常兴奋和你聊天，雅各布。见到你很高兴。祝你周五愉快。我在这里，准备好谈论订阅问题。让我们开始吧。

Hey, so when it kicked things off, we've got a lot of really cool testing that Duolingo has done. Your blog is amazing, by the way. I hadn't seen it until recently. Tweets, dorms, blogs, there's so many, we'll link to all these in the show notes. But so we'll talk about the individual tests in a minute, because I think there's some really great learnings. But I wanted to kick off just talking about how Duolingo thinks about testing and then kind of what the process is. So I kind of have a few topics I wanted to hit. But let's just start with ideation. Or I don't know if there's even maybe a broader overview you wanted to kick off with.
嘿，所以当测试开始之后，Duolingo所做的很多非常酷的测试，你的博客非常棒，顺便说一句，我最近才看到它。推特、宿舍、博客，还有很多，我们会在节目注释中链接到所有这些东西。但是我们稍后会谈到每个测试的细节，因为我认为有一些非常棒的学习经验。但首先我想谈论的是Duolingo如何考虑测试，以及测试流程。所以我有几个话题想讨论。但是让我们先从创意开始。或者，也许你想先做一个更广泛的概述。

I can start with a deeper cut than that. As I saw that one of the blog posts, it was like, do Duolingo, we test everything? Why? Yeah, that's a great question. I think one thing that we keep in mind as a company is that data and the data that we collect and analyze and use to improve our product is one of our best modes. As a company, we have millions of learners, I think, now we're close to 60 million monthly active users. And that allows us to collect a lot of data that makes us get better at teaching languages, makes us get better at making our product more engaging.
我可以开始比这还要深入的剖析。当我看到博客文章中的这个问题时，它说：“Duolingo，我们测试一切？”为什么？是的，那是一个很好的问题。我认为我们作为一家公司所始终铭记的是，数据以及我们收集、分析和利用来改进产品的数据是我们最好的工具之一。作为一家公司，我们拥有数百万的学习者，现在我们接近6000万的月活跃用户。这使得我们可以收集大量的数据，使我们在教授语言方面变得更好，使我们的产品更有吸引力。

And ever since this this founding of Duolingo, I only joined a couple of years ago. But since founding of Duolingo, AB testing and metrics driven product development has always been in our DNA. And it's worked really, really well. Even like 10 years after the launch, we're still maintaining like 50% year over year, year, year, year growth. So we just have a formula that works and it's something that we're committed to for the long run.
自从Duolingo成立以来，我只加入了几年时间。但是自从Duolingo的成立以来，基于AB测试和度量指向产品开发一直是我们公司基因中的一部分。这种方式非常有效。即使是在推出10年之后，我们的增长率仍然保持每年50％的水平。因此，我们拥有一种行之有效的模式，并且我们承诺将这样做很长时间。

Yeah, and it's, I mean, testing is something that it's not always a silver bullet. Like sometimes the infrastructure you need and the effort it takes and sort of the expertise to not test poorly is also like not easy to come by. And so, yeah, I almost think that by making a bold statement like that, that testing is a part of our culture, you kind of have to do that because otherwise, if you're just like, oh, we're going to try to AB test some things, it's probably not going to go well. You have to make it a big deal.
是的，测试并不总是解决问题的唯一解决方案。有时您需要的基础设施、付出的努力和不测试差的专业知识是不容易获得的。因此，我几乎认为如果你不像测试是我们文化的一部分发表大胆声明，这就要考虑到了，否则，如果你只是说，哦，我们要尝试AB测试一些东西，那么这很可能不会取得好的效果。你必须要把它当成是一件大事。

And it sounds like it sounds like it sounds like that's what you all do. And the, the mode thing is also very interesting because, you know, for folks that think, you know, like as doling you have to be thinking like, why won't there be a next doling? Like why won't some things move faster than us? Be does because typically a smaller company can move faster in some aspects, right? They can experiment. They don't have anything to lose. Like they can try stuff. And so you think about defensibility in modes, right?
听起来你们就是这么做的。此外，模式的问题也非常有趣，因为对于那些认为像你们这样的大公司，你必须要考虑为什么不会有下一个像你们这样的公司呢？为什么有些事情不会比我们的发展更快呢？这是因为通常情况下，小公司在某些方面可以更快地行动，他们可以实验，他们没有什么损失。他们可以尝试各种各样的东西。因此，你需要考虑模式的防御性，这就是对手不易模仿或跟随你个公司的优势所在。

Exactly. And in consumer, it's, it's brand is a big moat, which dolingo has as well. But data, especially in consumer when you have so many users, is especially when you have that many users, there's something a small startup can't replicate, right? And they can't test right away. So it makes strategic sense, especially at scale.
没错。在消费品领域，品牌是一座大堡垒，就像Duolingo一样。但是，特别是在消费品领域，用户越多，数据就越重要，因为一家小型创业公司无法复制，并且无法立即进行测试。因此，特别是在规模上，这是一种战略上的合理选择。

Exactly. Yeah. Well, that's part of what I wanted to go through this process though, because while you can't test at scale as a small company, I think there's so much that every size company can learn from dolingo about the process and kind of the thinking behind it. And that's why I, and a great way to start, Jacob, thanks for the save there on asking the overview. Yeah, start with the why, you know, start with the why.
没错。是的。其实这也是我想通过这个过程来实现的一部分，因为虽然小公司无法在大规模上进行测试，但我认为每个规模的公司都可以从dolingo的过程中学到很多东西，了解背后的思维方式。这就是为什么我认为这是一个很好的开始，Jacob，感谢你提出这个总体概述的建议。是的，从为什么开始，你知道，从为什么开始。

So now let's dive into the details because it, and this is where I do think a lot of ideas get short tripped in tech community. Ideas that are diamond dozen, the yadda yadda, but as part of a testing culture, it is super important to come up with the ideas to test. And whether you're a big company or a small company, whether you're testing just by releasing an update for a week, and then changing it a week later and looking at the results without any kind of sophisticated infrastructure, you still need to come up with ideas and try ideas and have bold ideas. So let's just start with that.
所以现在让我们深入细节，因为这是我认为技术社区许多想法都会受到短视的地方。这些想法就像是钻石一样随处可见，但作为测试文化的一部分，提出要测试的想法是非常重要的。无论您是一家大公司还是一家小公司，无论您是通过发布更新一个星期来进行测试，然后在一周后进行更改并查看结果而不需要任何复杂的基础设施，您仍需要提出想法并尝试这些想法，有大胆的想法。所以让我们从这里开始。

What's the ideation process like how do you come up with ideas to test and what does that look like internally? Yeah, it's a great question. I would say our ideas and our ideation process comes from a few different places. So much other consumer apps. We do a lot of UX research. We have a great design team. A lot of ideas can come from exploration from those functions, but also a lot of the ideas are driven by product managers from a bottom's up approach.
这是一个很好的问题。我们的创意和创意过程来自几个不同的地方。我们通常从其他消费类应用中获取创意，进行大量的用户体验研究。我们还拥有一支出色的设计团队。很多想法可以从这些功能的探索中获得，但许多想法也是由产品经理从底层向上推进的。

So I joined Dueling go right out of undergrad as an associate product manager. And even someone like myself at the junior level was empowered to sort of drive team roadmaps and own parts of the road map to ideate on features. So there's a lot of that just like pure ideation from the bottom level of the company all the way up to the executive level where the ideas are eventually proposed.
我在本科毕业后加入了决斗社作为一名副产品经理。即使像我这样的初级员工也有权力推动团队路线图，拥有一部分的产品规划来探索功能。因此，公司底层到高层都有很多创意的纯意识形态，最终这些想法都会在高层提出。

But more tactically, I think a lot of our ideation comes from the cycle of iterating and running experiments. So every time we run an A, B test, we will analyze the data and from that data, we'll learn like five or six different things that we can then turn around immediately and test again. So we keep a really tight loop of iterating and learning from experiments.
但更加策略性的是，我认为我们许多想法都源自于不断迭代和进行实验的循环过程。每当我们进行一次 A/B 测试时，我们会对数据进行分析，从数据中学习到五六个不同的事情，然后我们立即可以再次进行测试。所以我们保持紧密的迭代和从实验中学习的循环过程。

Another thing I think that's really unique to Dueling go is that it's an app that every employee can use. So we have a really strong dog food and culture where all of our 600 plus employees are using Dueling go. We have a big channel on our Slack called feedback product where anyone at the company can just dump in feedback on the product about anything related to the app. And as PMs we we watch that channel and whoever it is that's dumping and feedback into that channel. We can sort of use that to clean insights, get ideas. And yeah, I think that's really unique to Dueling go because a lot of other subscription apps, maybe not everyone employed at the company uses it regularly. But Dueling goes an app designed to be used daily. It's a daily, it's a daily app. So you can use it every day and get insights to iterate quickly.
我认为 Dueling go 独特的一点是，它是每位员工都可以使用的应用程序。我们有一种非常强烈的“狗粮文化”，有600多名员工都在使用 Dueling go。我们有一个在Slack上大型的称为“产品反馈”的频道，公司中的任何人都可以在其中倾泻与应用程序有关的任何反馈。作为产品经理，我们会关注这个频道，并收集任何有用的信息和想法。所以我认为这是 Dueling go 所独有的，因为其他订阅应用程序，可能不是公司中的每个员工都经常使用它。但 Dueling go 是一款被设计用于日常使用的应用程序。可以每天使用它并获得清晰的见解以快速迭代。

When you have an idea, you always go like we have to design a test around it. And I asked because some ideas, well, there's two cases there were as the visionary founder, I'm like, I don't need to test this. I know it's going to be great. So that's one. And then the second one being like sometimes these are hard to test, right? Like sometimes they don't have like, I mean, I guess you have your KPIs and things like this.
当你有一个想法时，你总是会想我们需要围绕它设计一个测试。我问这个问题是因为有些想法，有两种情况，作为一个有远见的创始人，我不需要测试，我知道它将会很棒。这是第一种情况。第二种情况有时候很难测试，对吧？有时候它们没有明确的关键绩效指标之类的东西。

But do you do you have to be judicious and and then also you can't run infinite tests. Right. There's only so much bandwidth in a product and and new experience. Right. There's there's limitations to what you can test and synchronously or like together and things like this. So like how do you do always test everything before you hit go or or can you just use your judgment on what's an important test or not or how to set how do you set the bar?
但你必须谨慎，而且不能无限制地进行测试。是的，产品和新体验中只有有限的带宽。你只能测试一定数量的东西，有同步或一起测试的限制。那么，你是总是要在组合测试之前测试所有东西，还是可以根据判断仅测试重要的部分，以及如何设定标杆？

Yeah, it's a great question. As crazy as it sounds, we basically do test everything. Only the like the time everything goes into a feature flag. So anything you're building gets flagged and then it just even if you don't expect like some groundbreaking result, you're going to run it on and off just to like at least measure baselines and stuff. Correct. Even if it's just like a backend change that doesn't change the UX experience at all. We'll just run it as an A B test just to make sure there's no regressions where it impacts something else in the app.
是的，这是一个很好的问题。虽然听起来很疯狂，但我们基本上会测试所有东西。不管你在构建什么，都会被标记，并且在特性开关中进行测试。因此，即使你没有期望像一些开创性的结果，你也会对其进行开关测试，以至少衡量基线和其他方面。即使它只是像一个后端更改，根本不会改变UX体验，我们也会将其运行为A / B测试，以确保没有回归，以影响应用程序中的其他东西。

So literally every tiny change will go through as an A B test. That's just sort of how we do things. Interesting. And that has to take a lot of infrastructure. Right. Like not just computers, but the people to maintain those computers and like the people to maintain the testing stuff. And how does it as a PM. I assume you have like a team with like a mission thing. It's like how do you is that all clients and like are you guys basically clients to like all that infrastructure is easy to use. And what does that look like.
从字面上看，每个微小的变化都会进行 A B 测试。这就是我们做事情的方式。有趣。这需要很多基础设施，不仅是计算机，还有维护这些计算机的人员以及维护测试工具的人员。作为产品经理，你是如何处理这些任务的呢？我猜你有一个拥有任务的团队。这看起来像所有客户都是这样，你们基本上是客户，使用这些基础设施很容易。那看起来是什么样子的呢？

Yeah. So we all of our like A B testing infrastructure is built in house. We have a really strong data infrastructure and experimentation team that builds these tools that allows engineers to set up A B tests once they once they build a feature, for example. And within that same tool every morning, every day we generate reports on the new data that comes in from experiments and as a PM. That's like my first thing I do when I went up is check the experience dashboard see all the like right now my team. I think is running like probably 20 experiments and on different parts in the app.
我们的A/B测试基础设施全部是内部建立的。我们有一个非常强大的数据基础设施和实验团队，开发了这些工具，允许工程师在构建功能后设置A/B测试。在同一工具中，每天早上我们会生成关于实验中新数据的报告。作为一个产品经理，我的第一件事就是检查体验仪表板，看看我的团队正在进行的大约20个实验，测试应用程序不同部分的效果。

So just monitoring those experiments is done with this in in house built tool. And yeah, generally in terms of the limitation of not running too many A B tests. The one principle we try to buy by is not running multiple tests on the same real estate at once. And if we do then we will look at the intersection of those two experiments to make sure that there's no weird effects with those experiences being combined.
因此，仅通过内部构建的工具来监控这些实验。是的，在不运行太多A/B测试方案的限制下，我们试图遵守的原则之一是不要同时在同一实物上运行多个测试。如果我们这样做了，我们将会查看这两个实验的交集，以确保这些经验被合并时没有奇怪的影响。

The other thing I'll mention is that because we're both on iOS and Android oftentimes we will run different experiments on the same real estate on the different platforms with the eventual goal of aligning for platform parity depending on what wins. Interesting. How do you think about the funnel dynamics of confounding variables. So if 1 PM is testing an onboarding change that kind of shifts the focus this direction and your way deep in the retention funnel, you know, testing something that is a different direction.
我要提到的另一件事情是，因为我们都在iOS和Android上，所以经常会在不同的平台上运行相同房地产上的不同实验，最终的目标是根据胜者平台来对齐。很有趣。你如何考虑混淆变量的漏斗动力学？所以，如果1个PM正在测试一个改变引导的变化，这个方向就会转移而你深入留存漏斗，你知道，在测试不同方向的东西。

How do you think about those early or middle or late tests kind of being confounding variables against each other. Yeah, because they're not same real estate necessarily right but they're the same real state in the sense that they're all in the app. You know, they're all part of the experience. Yeah, it's a great question. I think from a collaboration perspective, our product or the product team is very collaborative in terms of sharing what each team is working on what different PMs are working on in different real states.
你如何看待早期、中期或晚期测试之间相互混淆的变量？因为它们并不一定是相同的房地产，但从某种意义上来说，它们都在应用程序中。你知道，它们都是体验的一部分。是的，这是一个很好的问题。我认为从协作的角度来看，我们的产品或产品团队非常具有合作性，可以共享每个团队正在处理的不同的房地产的工作以及不同项目经理所处理的不同房地产的工作。

So whenever a product change is approved by our product review council and like the CEO and others, the PM will send an email out to the product org and others can also be on that list of two. Just explaining where about to test this. This is what was approved. This was the conversation and the product review. So then there's at least awareness of oh, this is what's going on. And if a team is especially affected by a certain change, then there will be conversations even before that to make sure that the PMs and the teams are aligned. And then the same process sort of happens at the end of the experiment when you're about to launch it.
所以每当产品审查委员会批准产品更改，并且像首席执行官和其他人一样，产品经理将向产品组发送电子邮件，其他人也可以在名单上。只是说明我们要测试的地方。这是经过批准的。这是谈话和产品审查。因此，至少有人知道发生了什么。如果一个团队受到某个特定更改的影响，那么甚至在那之前就会进行谈话，以确保产品经理和团队保持一致。然后，在试验即将启动时，同样的过程也会发生。

They send out an email with the data and the results saying, hey, this is what we learned. And that allows PMs on other teams working on different areas of the app to sort of look at, oh, hey, this might impact my team's features or this might impact my metrics. So there's sort of that collaborative style from the practical level. And then in terms of the data level, I think we also do a lot of cross analysis of experiments. So if we know another team is running an experiment that might impact one of our experiments, again, we'll look at the intersection of those experiments to see if there's any differences in the data. So yeah, we do make sure that we're not stepping on each other's toes or bumping elbows. But overall, we're able to do it pretty seamlessly, I'd say.
他们发邮件附带数据和结果，告诉大家，这是我们学到的东西。这样，其他团队的产品经理就能够查看，这可能会影响我的团队的功能或指标。因此，从实际层面上来说，存在协作风格。在数据层面上，我们还做了很多实验的交叉分析。所以，如果我们知道另一个团队正在进行实验，这可能会影响我们的实验，我们会查看这些实验的交集，以了解数据是否有任何差异。因此，我们确保彼此不会干扰或冲突。总的来说，我认为我们能够无缝地完成这项工作。

I guess like, and I think this is kind of the natural next question is like results and understanding that stuff like how, like how do you always make follow the data. Do you ever say like, oh, this hurt the data, but the visionary CEO wants this. So we're going to do it anyway. How does that so you get you get the results in there's a big critique out there. And I think you know, if we're trying to apply this to like the broad nut, but there's very few companies in the world like revenue cat certainly I've never worked to come is had an A B testing system so dialed that it sounds like as on rails and powerful is what you all have. It's a big investment to get there. But you know, even on the smaller scale, you're like constantly kind of trying to decide like, oh, like what can I test. I have like kind of resources. And then like sometimes yeah, you're just you don't necessarily want to. You don't want to know the results right. And then sometimes if you get a result, if it's not the result you want, you might just ignore it anyway. So I'm curious like, how do you allow. How do you I mean, there's a big critique of test driven design is like a failed, you know, there's a lot of like people that say like, oh, you're just going to optimize into some local minima or maximum right. How do you do you balance that with like PM intuition and feedback and quantum, quantum to take qualitative stuff as well or how does that. Yeah, how does that go.
我想翻译的是：“我猜，这也是下一个自然问题，就是关于结果和理解的问题，比如，你们怎样才能始终按照数据来处理业务？你们是否曾经说过这个会损害数据，但是有远见的CEO却坚持要做？这会怎样呢？你们如何获得结果，并且在这个过程中会有一个很大的批评，如果我们试着应用于广泛的领域，但是像revenue cat这样的公司在世界上很少，我从来没有为一个公司工作过，他们有一个A/B测试系统这个系统听起来是像轨道上的，强大的工具库一样适用。要达到这个目标需要进行大量的前期投资。但是，即使在更小的规模上，你仍然不断地试图确定，例如，我可以测试什么。我有一些资源，但有时候，当你知道结果的时候，你可能不想知道结果。有时，即使你得到了结果，如果它不是你想要的结果，你也可能忽略它。所以我很好奇，你们是怎么做到的？你们如何平衡PM直觉和反馈和量子，量子应该也取一些定性的东西，还是怎样的？“

Yeah, totally. And we're like totally aware as a company of like the shortfalls. There is with a very A B testing driven framework. So there is a lot of like qualitative input into decisions on calling an experiment. So even if an experiment is a winner for our top line metrics, we will still have a discussion around whether in the long term, this fits into our road map or whether it makes sense for the UX. So a very common situation for a test that wins at Duolingo is that if the feature or the change complicates the UX or makes the app just feel or look busier, then there's always a conversation of, oh, is the metrics win here worth making the app more complicated. And the word the words we hear the most from our CEO, Luis is only launch this if it's a big win. Whenever we propose something that complicates the like the UX, there's like a threshold that isn't always clear, but it comes about when we have a discussion of, oh, is this metrics one worth it. So there is that qualitative discussion that happens with leadership that happens within the product and design orgs.
对，完全正确。作为一家公司，我们完全意识到了A/B测试框架的不足之处。所以在决策试验时，我们会考虑很多定性的因素。即使某个试验能够提高我们的总体指标，我们仍会讨论它是否符合我们的路线图，或者是否对用户体验有意义。在Duolingo，一个常见的情况是，如果一个功能或者变化会让用户体验更加复杂，或者让应用程序感觉更加拥挤，那么我们就会讨论一下，这个指标胜利是否值得让应用程序变得更加复杂。我们的CEO Luis最常使用的词就是只有大胜利才值得推出。每当我们提出一些复杂化用户体验的建议时，就会有一个阈值出现，虽然并不总是很清晰，但就是发生了讨论，是否值得这样做。因此，在产品和设计组织中以及领导层之间，都要进行这样的定性讨论。

Yeah. In general, I think we do a good job of making sure that we're not only following the numbers, but we're also data informed, right? Yeah, data. Data decided itself. How do you how do you all think about like significance, right? Because like, you know, you might you can run any test and even on, there's going to be noise, right? You might end up with, I mean, I guess maybe not at that scale right 60 million, MAU or whatever, DAU or whatever. So that's an amazing number you said that you get a lot of significance out of that, but I'm sure you still get to the point where our lot of results know like our lot of results like no no difference. Yeah, it's a great question. So and you're right that at our scale and the user base that we have a majority of the changes we can make. So significant, which is a really big advantage that we have, especially on teams like my team, the user retention team that works on features that impact basically the entire user base. We can roll out an experiment today and tomorrow morning will have statistically significant results on our top line metrics and maybe like day one retention or something or something. Yeah, exactly. Basically, we're always looking at DAU D1 retention things like that.
嗯，总体来说，我认为我们做得很好，不仅仅是遵循数字，而且也是数据驱动的，对不对？对，数据。数据本身就决定了一切。你们怎么看待显著性这个问题呢？因为你知道，你可以运行任何测试，即使在其中还会有噪音，对不对？你可能会得出，我想可能不是在那个规模上60百万月活跃用户，或其他的DAU或什么的。所以你们得到这样令人惊喜的数字，得到了很多显著性，但我相信你们仍然会遇到许多结果是无差异的情况。是的，这是个很好的问题。而你是对的，在我们的规模和用户基数上，我们大多数可以进行的改变，都是显著的，这是我们所拥有的一项非常大的优势，尤其是对像我团队这样的用户保留团队来说。我们的工作是处理那些影响到整个用户群的功能。我们可以在今天推出一个实验，在明天早上，我们将对我们的最重要的指标和一天的留存率等东西，获得有统计显著性的结果。

Generally, we make sure we run experiments for at least three or four weeks on average, just so we're rolling out novelty effects. So we're making sure that, yeah, there is like a strong significance there. Other teams at the company that are working on like more specific parts or perhaps even new initiatives. For example, we have a product called dualingo for schools, which is basically a dashboard for teachers to allow their students to use dualingo. And since they have a smaller user base of teachers, they need to do more calculations around, okay, how long do we need to run it? Can we actually achieve it within the before the heat death of the universe kind of exactly. Yeah, exactly. Yeah, but for most teams that are shipping features to millions of users and these users are using the product every day, it's really easy to get significant son or data.
通常情况下，我们确保对于实验我们至少进行3到4周的时间，以确保我们不会误认为新颖因素会持续下去。因此，我们确保确实存在强烈的统计学意义。我们公司的其他团队致力于更具体的部分或者全新倡议。例如，我们有一个叫做Dualingo for Schools的产品，它是一种供教师使用的仪表板，允许他们的学生使用Dualingo。由于教师这一用户群体较小，他们需要做更多的计算以确定我们需要多长时间才能够实现目标，以免超过宇宙热寿命限制。但对于大部分向数百万用户发布功能并且这些用户每天都在使用产品的团队来说，很容易得到有效的数据或证据。

How do you all think about look back? So you have a win, you roll it out. It's got statistical significance. It seems like it's going to be a great win. Do you revisit some of these results six months, two years or not two, I mean, two years is a really long time, but six months, nine months from the original test to see if long term retention was sacrifice for short term retention or, you know, monestization was sacrificed over the long haul for, you know, DAU or things like that. How do you think about those long term impacts? Yeah, that's a great question. I think we do two things.
大家怎么看待回顾？你赢了一次，你就推出了它。它具有统计学意义。似乎这将是一个伟大的胜利。你会在六个月，两年或不是两年而是久远的一年后，重新审视其中的一些结果，以查看长期保留是否为了短期保留而牺牲，或者说为了DAU或其他方面的利益而放弃了长期的货币化。你怎么考虑这些长期影响？是啊，这是一个很好的问题。我认为我们会做两件事情。

One thing is for, especially for our bigger features will run holdout experiments. So basically after we launch an experiment will keep maybe one or two percent of users in the control group. And leave that running for like three or four months, something like that that allows us to get more insight on the long term impact of an experiment. This is especially useful for teams that want to get like assess the impact of changes that might not have impact on D1 retention immediately. For example, we have a team that works on social features, basically connecting learners with other learners on the platform. And sometimes adding those features doesn't immediately impact top line metrics like DAU. But in the long term, having a more social experience through like network effects that can impact our top line metrics. So I know that team they run a ton of holdout experiments on a lot of their features. And we do as well.
我们的一个重要做法是针对我们重要的功能进行保留实验。基本上，在我们推出一个实验后，我们会将控制组中的用户保留1-2％，并将其保留运行三到四个月左右，以便我们更了解实验的长期影响。这对于想要评估对D1保留没有立即影响的变化影响的团队尤其有用。例如，我们有一个团队致力于社交功能，基本上是将学习者与平台上的其他学习者联系起来。有时，添加这些功能不会立即影响上线指标，比如DAU。但从长远来看，通过网络效应拥有更社交的体验会对我们的上线指标产生影响。所以我知道那个团队会对很多他们的功能进行保留实验，我们也是如此。

The other thing we do to make sure that we're also tracking the long term impact is that we will track the feature level metrics for a feature that we launch over the long term. So for every feature that we build will have a feature dashboard that tracks some of the more ad hoc metrics feature engagement level things that we might be interested in. Over time, we can see, oh, when we initially launched this feature engagement was really high, but it seems like it's dropping a little bit that sort of gives us a send indication that there might be some novelty effect at play. And yeah, that's sort of the two approaches we take to making sure that we're looking at data in the long term as well.
我们为确保追踪长期影响还采取了另一种方法，就是在长期内跟踪我们推出的功能的指标。因此，对于我们所构建的每一个功能，都将有一个特性的仪表板，用于跟踪一些我们可能感兴趣的更临时的特征参与度等指标。随着时间的推移，我们可以看到：当我们最初推出这个功能时，参与度非常高，但似乎有一些新奇效应在起作用，因此参与度有些下降。这是我们确保我们在长期内看待数据的两种方法。

So I do want to move on to the kind of specific test that run, but one more shot at anything else you want to kind of summarize about the way you'll think about testing or maybe anything we left out that you think is particularly important about about how do you think about and actually does the testing. Yeah, I think the one last thing I'll mention is that because we collect so much data, it allows us to get really granular in the analysis that we do. So I'm sure we're about to talk about notifications a bit more, but one example I'll give is that notification copy obviously performs differently in different UI languages and geographies.
我想开始讨论我们运行的具体测试，但是您是否还有其他想要总结的有关测试方式的内容，或者我们忽略了您认为特别重要的内容，您认为如何考虑和实际进行测试？是的，我想最后提到的一件事是，因为我们收集了如此多的数据，这使得我们在分析上可以变得非常详细。因此，我相信我们即将更多地谈论通知，但是我想举一个例子，即通知的文案在不同的界面语言和地理位置上的表现不同。

But because we collect so much data, we can sort of drill down and look at the click through rates on our notifications by language and that allows us to say, oh, in Romanian this notification isn't doing as well, we should talk to our Romanian localization team and see if there's anything there that we can optimize for. So just by the sheer amount of data that we collect were really able to drill down and optimize a lot of things on the app and it keeps us busy as a product team because there's so much stuff that we know we can improve. I have one more question on general strategy on choosing ideas and stuff, but how does like knowing the test ability of a feature influence you like decision to build it like if there ever.
但因为我们收集了很多数据，我们可以分类并查看我们的通知在不同语言下的点击率，这让我们能够发现，在罗马尼亚，某个通知的效果不是很好，因此我们需要与我们的罗马尼亚本地化团队沟通，看看是否有什么可以进行优化的地方。因此，仅凭我们收集的大量数据，我们真的能够挖掘很多应用程序的优化点，这让我们作为一个产品团队非常忙碌，因为我们知道有很多东西可以改进。我还有一个有关选择想法和方案的一般策略的问题，但是，了解一个特性的可测试性如何影响您决定是否构建它呢？

I guess like you're saying, since you're looking at top lines, you can always have something to measure, right, but do you, does it you find it. I mean, my again, you can maybe just validate my assumption, my assumption is that because you're thinking, well, this actually has to produce a measure of result, does do you think it pushes your product teams to like only focus on things that are going to actually show up in the data.
我猜你的意思是，因为你在关注顶线，所以你总是有东西可以衡量，这样做对你有用吗？我的意思是，你或许可以验证我的假设，我的假设是因为你认为这实际上必须产生一种结果的度量，你是否认为这促使你的产品团队只关注那些实际上会在数据中显示出来的事情。

You know what I mean? Yeah, it's a good question. I think it varies by team at dual and go, for example, we have product teams that work explicitly on improving how the app teaches. So this is the learning R&D area that has multiple teams building features to teach better and for them, their products, their product changes don't really impact top line metrics because. The tragedy of all education apps, right, exactly. Yeah, you got it. And oftentimes when we want to teach better, we have to make content harder and content being harder often is bad. The same in same in fitness apps to right, like you have to like maintain that, you know, that sweet spot.
你知道我的意思吗？是的，这是一个好问题。我认为在Dual和Go的不同团队中有所不同，例如，我们有专门致力于改进应用程序教学的产品团队。因此，这是学习研发领域，有多个团队在构建更好的教学功能，对于他们来说，产品变更并不会对业绩指标产生很大影响，这是所有教育应用程序的不幸，对吗，一点也没错。而且通常情况下，当我们想要更好地教学时，我们不得不使内容更加难以理解，而内容更加困难往往会产生负面影响。体育健身应用程序也是如此，你必须保持那个恰到好处的位置。

Yeah, exactly. So for those teams, obviously, they haven't they're not really incentivized by a top line metric that they're optimizing for. So a lot of their work is more like stakeholder alignment with learning scientists with leadership with designers, things like that. So I think that makes it more so that they're not just looking at the data.
是的，完全正确。因此，对于那些团队来说，很明显，他们并不通过优化顶线指标来获得奖励。因此，他们的许多工作更多地是与学习科学家、领导者、设计师等利益相关者进行调整。因此，我认为这使得他们不仅仅是看数据。

And then yeah, more generally, we make sure that we're not only doing things that are metrics driven, we always set OKRs for reducing tech that things like that improving product quality stuff like that. So sorry, last question.
然后，更一般地说，我们确保不仅仅是以指标为驱动的事情，我们总是设定OKR来减少技术，提高产品质量等等。所以，抱歉，最后一个问题了。 意思是，他们不只是关注数据指标，而且会设定目标来降低技术成本，提高产品质量等等，同时是最后一个问题。

Do you like A B test bug fixes and things like if something's just flat out broken or like maybe not even a bug fix, you're like, oh, this is just wrong. I mean, I guess it's another version of my original question of like, do you test everything? But like, yeah, say would you would you A B test a bug fix if it was like a living bug that's not like ruin, like just crashing the app or something right.
你喜欢A/B测试错误修复和像一个完全错误或可能甚至不是一个错误修复的东西吗？我的意思是，我猜这是我的原始问题的另一个版本，就像你会测试每个东西一样。但是，如果像一个活动性的bug没有破坏应用程序，你会A/B测试错误修复吗？

Yeah, if it's a huge bug fix, then we will like patch it and then maybe run a holdout experiment, like giving the bug back to like one percent of users just just to assess the effect. Yeah, it's crazy. But that's great. I mean, this is why it's a cultural thing, right? And this is I think. And I think as our listeners are thinking through like how they're building their apps and how they want to do product development.
是的，如果有一个很大的错误需要修复，我们会进行修补，然后可能会进行滞留实验，例如将错误返回给百分之一的用户，以评估其效果。是的，这很疯狂。但这很好。我的意思是，这就是文化所在，对吧？我认为这是有意义的，同时我们的听众也应该思考如何构建他们的应用程序以及如何进行产品开发。

Like I imagine this came from the founders right like or from very, very early and push right. This is something that the founders company felt was the way they wanted to do product and they instilled it in the culture and they made it a thing. And I think if you're listening to this and you're going like, oh God, I don't want to set up all the data and destruction all that's up.
我想象这是来自创始人，或是非常早期推动的理念。这是创始人公司觉得是产品开发的方式，并将其注入公司文化，让它成为一种事情。如果你正在听这段话并且感到困惑，不知道该如何建立数据和销毁所有的数据，那么我认为...

You don't have to right there's lots of ways to do this right. But I do think you should be opinionated in how you want to set that up. If you want to be a their company that said we don't run any tests like everything is just like user comment that's fine. Like I think there's a lot of good companies that are built that way. Right. You have to think about what you who you are as a founder and as a app builder and as a and then and then apply the the system that you think is going to be you know be best fit for you to get the best results not just necessarily doing I don't know of another company.
你不必非得写在那里，有很多种方法可以做到这一点。但我认为你应该有自己的想法来设置它。如果你想成为一个这样说的公司，“我们不运行任何测试，所有的一切都像用户评论一样”，那是可以的。我认为有很多这样的好公司。你必须考虑作为创始人和应用程序构建者的身份，并应用你认为最适合你获得最佳结果的系统，而不仅仅是模仿另一个公司。

At least in the our space that's doing quite to the extreme of that. Yeah, but it's pretty crazy. But yeah, we're very data driven or data informed as he said. Yeah, but probably no coincidence that it's one of the larger subscription apps and publicly traded and doing.
至少在我们的领域里，这个做得非常极致。是的，但这相当疯狂。但是，我们非常注重数据，正如他所说。是的，但这可能不是巧合，它是较大的订阅应用程序之一，且已上市表现良好。

I think as well. Like I think it's one of the one of the great success stories of mobile. I would say in general like it's crazy. So I did want to move on to specific results because you built all this infrastructure and you have all this alignment internally you do all these tests but what matters is like what comes out of it.
我也是这样想的。我认为这是移动领域最成功的故事之一。总的来说，这太不可思议了。我想转向具体的结果，因为你建立了所有这些基础设施、内部协调和测试，但最重要的是，它会带来什么成果。

And then I think for our listeners, you know, as we've talked about very few of them are going to be able to operate at this scale, but I think there's learnings that do Linga has been sharing again on the blog and Twitter and other places that are worth testing or for the product are to are just implementing whole hog based on do Lingua success with them.
我认为对于我们的听众而言，正如我们所说，很少有人能够以这种规模运营，但我认为Do Linga一直在博客、Twitter和其他地方分享的经验是值得测试或针对产品进行整体实施的，这取决于Do Lingua对它们的成功。

So the first experiment I wanted to talk about was with notifications. And so you've done a ton of testing around notifications, but there is a specific tweet thread that we'll link to in the show notes. Just talking about how you test notifications and some of the top learning.
因此，我想先谈谈有关通知的第一个实验。你们已经进行了大量关于通知的测试，但在节目注释中，我们会链接到一个特定的推文线程。它主要讨论了如何测试通知以及一些重要的学习经验。

So I'd love for you to share maybe a little background on it, you know, that the tweet that I didn't go super into depth about the testing setup and stuff like that, but maybe just a little background and then let's talk through the results of that and what folks can learn.
我希望你能简单介绍一下它的背景，你知道，我没有深入探讨测试设备等细节，但也许简单介绍一下它的背景，然后让我们讨论一下结果，以及人们可以从中学到些什么。

Yeah, totally. So notifications are a big engine for growth for do a lingo so much so that notifications from do a lingo are sort of a meme. I'm sure maybe you see. Yeah, notifications from the green owl. Yeah, I see the type of green owls becoming a like like the slender band now it's like a harassing mythical character. Exactly.
是的，完全正确。通知是Do a Lingo增长的重要推动力，以至于来自Do a Lingo的通知成为了一种玩笑。我想你也可能见过。是的，来自绿色猫头鹰的通知。是的，我发现绿色猫头鹰变成了一种像斯林德班一样的神话角色，令人感到困扰。没错。

But yeah, notifications are a big driver for our retention and our user growth and it has been for many years. And we're now at a very sophisticated point where we send a lot of notifications. The main one being the daily practice reminder. So we send you a reminder to practice every day. And if you've been inactive for seven days, then we'll stop sending it to you. But within that daily practice reminder, we have call it 250 different copy templates that are eligible to be sent to a user. Wow. And we have a machine learning algorithm that optimizes the best ones to send to a user.
是的，通知对于我们的用户保留和增长是一个巨大的推动因素，已经有很多年了。现在我们已经到了一个非常复杂的点，我们发送了很多通知。主要的通知是每日练习提醒。所以我们会给你发送提醒，每天练习。如果你连续七天没有活动，那么我们将停止向你发送提醒。但在这个每日练习提醒中，我们有250个不同的副本模板，可以发送给用户。哇。我们有一个机器学习算法，可以优化发送给用户的最佳模板。

What we do as PMs is iterate on those 250 copy templates adding more changing existing copy templates and just testing different messaging tone. And content within those notifications to basically squeeze out more retention gains. And we've learned a lot about what works and what doesn't work with notification copy and the messaging there. So yeah, I'll go ahead and dive into a few of the learnings we've had from that.
作为项目经理，我们的工作是对这250个文案模板进行迭代，不断添加、修改文案，还要测试不同的信息语气和内容，以实现更多的用户留存增长。我们已经通过通知文案和信息传达方式，学到了很多关于什么有效，什么无效的内容。因此，接下来我会介绍我们在这方面的一些经验和教训。

So this is a case where these are not like this seems like another system that you built. This is not just another A B test. You guys have a separate like multi armed bandit model that like will throw these in and optimize on an individual basis.
这是一个情况，其中这些并不像它看起来是你们所建立的另一个系统。这不仅是另一种A/B测试。你们有一个不同的多臂赌博模型，它会投入这些数据并在个体基础上进行优化。

Which my question my first question. I mean, maybe I'm going to jump ahead too far but it's like what goes into that individual choice like what vectors are like what inputs into that. You say like, oh, like we're going to try that what do we know about this person we're going to try this. That would be really interesting to hear like how do you actually like. What do you optimize on, you know, yeah, and let's dive into the AI behind it.
这是我的第一个问题，也许我会提前得太远了，但我想知道的是，个人的选择受到哪些因素的影响？就是说，有哪些因素会影响这个选择？例如，当你决定要尝试某种东西时，你会关注这个人的哪些方面？听到这个会很有趣，那么你们实际上是如何优化这些因素的呢？让我们深入了解背后的人工智能技术。

I mean, it's it's a few points ahead in our show notes. But yeah, it's a pretty incredible system you built. Yeah, the first thing I'll mention and this might also sound crazy, but even within this multi armed bandit algorithm, we still do A B tests.
我的意思是，在我们的节目笔记中，它领先几个点。但是，没错，你构建的系统非常惊人。是的，我想先提一下这个，这可能听起来也很疯狂，但即使在这个多臂赌博算法中，我们仍然进行A/B测试。

Why not? So yeah, why not exactly. So how it works is that we have the multi armed bandit that has all these copy templates for notifications. And then as the experiment arm in an A B test, we will have that same bandit except with the additional new copy templates that were added.
为什么不呢？是啊，为什么不呢。这是如何工作的，我们有一个多臂赌博机，里面包含了所有通知的副本模板。作为 A/B 测试的实验组，我们将使用同样的赌博机，只不过加上了一些新增的副本模板。

I suppose like you could even do like a variation changes or like tuning changes to the algorithm and things like that. Maybe that as a whole separate thing. Yeah, exactly. So whether it's a copy template where we add more copy templates to the bandit to optimize on. So you can optimize globally as well. Right.
我认为你甚至可以通过对算法进行变种或调整来进行优化。这可能是一个完全不同的事情。是的，完全正确。因此，不管是添加更多的复制模板到多臂赌博机进行优化，还是在全局范围内进行优化，都是可行的。

So you can say like adding these not just like seeing which is more performant but looking overall when we add these 10 new messages like how does that change the whole performance. Yeah, the A B test would be in control is the multi armed bandit with 250 templates. And then the experiment is the multi armed bandit with 260 adding the 10 new ones.
所以，你可以说，我们不仅要观察哪个比较有效，还要全面考虑当我们添加这10个新的信息后，整体性能会如何变化。是的，A B测试将有一个控制组，使用250个模板的多臂赌博机。而实验组则是使用260个模板的多臂赌博机，其中还添加了10个新的消息。

Exactly. And then we can measure on a local basis of when we send those 10 new ones, what that is doing to click through, but also generally what the overall impact is to top line metrics. And this is all driven by AI machine learning, right? So you have a pretty big data science team building out these models that make those decisions and analyze results.
没错，然后我们可以在本地基础上衡量发送这10个新的邮件对点击率的影响，同时也可以衡量对总体指标的影响。这一切都是由人工智能和机器学习驱动的，对吧？所以你拥有一个相当庞大的数据科学团队，构建出这些模型来做出决策和分析结果。

It is AI and ML driven, but to be honest, it's simpler than it seems. I mean, basically, yeah. I'll give the you can create me maybe if I'm wrong on this, but a multi armed bandit we kind of jumped into the like the technical term. It's basically a computer or a computer science or commonator's problem or question where you have like end choices, maybe have 10 different levers you can pull.
它是由人工智能和机器学习驱动的，但说实话，它比看起来简单。我的意思是，基本上就是这样。如果我说错了，你可以告诉我，但我们现在先跳进一个技术术语中——多臂赌博机。它基本上是一个计算机科学或组合问题或问答，你有像10个不同手柄可以拉动的选择。

And the rewards you don't necessarily know what you're going to get from each lever, right? And so multi armed bandit problem, there's optimizations to say like which lever should I pull when an algorithm that optimizes that is basically what you're saying in this case. And it correct me by this wrong, but instead of levers, they're actually like showing a notification that's your lever and you're trying to optimize which notification show when is is optimal.
奖励并不一定知道从每个杠杆手柄上会得到什么，对吧？因此，多臂赌博机问题要优化的是，例如当优化这个算法时，我应该拉哪个杠杆。请纠正我如果我错了，但是这里不是杠杆，而是显示一条通知，你正在尝试优化什么时候显示最佳通知。

Is that correct? Exactly. So yeah, the levers in this case would be the copy templates, the 250 different ones that are eligible to be shown.
这句话的意思是：「正确的。所以，这种情况下的杠杆作用是文本模板，我们有250个不同的文本模板可以被显示出来。」 建议翻译为：没错。在这种情况下，我们有250个不同的文本模板可以被显示，在这些模板中，可以选择合适的进行应用。

One of the things I just interject a little ethics point when I think I love talking to you about this is that in some ways I hear all this and I think, you know, Facebook convincing me to buy some other stupid thing I don't need or candy crush convincing millions of people to buy more gems and spend more money. And there's a lot of ways these tools can be used to manipulate and drive more views of TikTok and keep you know keep me on Twitter for hours or Twitter's a bad example because they don't do anything like this.
我想在这里插入一个伦理观点，当我觉得我爱和你谈论这个问题时。从某种意义上说，我听到了所有这些，我想，你知道，Facebook让我购买一些我不需要的东西，或者糖果碾压让成千上万的人购买更多的宝石和花更多的钱。这些工具可以用很多方式来操纵和推动更多的TikTok视图，并让我在Twitter上花费数小时，当然Twitter是个不好的例子，因为他们不会做任何像这样的事情。

Oh, no, they do that. They do. Twitter's like retention notifications are crazy. If you don't engage with Twitter for very long, they really start to like spam you. Maybe not maybe not in this exact case, but yeah, but I think you know, TikTok and Reels and things like that are a better example of kind of using this kind of testing and learning to to manipulate versus with doling go.
哦不，他们真的这样做了，Twitter的留存通知让人发狂。如果你长时间不与Twitter互动，他们就会开始像垃圾邮件一样轰炸你。也许不是在这种情况下，但是我认为，TikTok和Reels等平台更好地使用了这种测试和学习的方式来操纵用户，相对于Twitter而言。

You're helping people learn. I mean, you think you're doing all the. It's so manipulate and influence. Are they any different? Right. It's really just the connotation of what you're asking for at the end, right? So the notifications thing. I think this is especially for the intersection of our audience right where yeah the app are tours. The second one you've heard term David. We're no longer we're no longer catering to indie developers, we're turning to app are tours.
你正在帮助人们学习。我的意思是，你认为自己在做所有的事情。它是如此的操纵和影响。它们有什么不同吗？没错。实际上，你在最后询问的是你所要求的内涵，对吗？因此，关于通知的事情。我认为这特别适用于我们受众的交叉点，就是应用程序的游客。你听过 David 这个术语的第二个用法。我们不再关注独立开发者，而是转向应用程序游客。

But I think there is this there's this idea that specifically talk about notifications and we talk about like optimization but they like notifications are bad. They're spammy their noise. They're ads. They're whatever. But I don't really I guess two sides to it like one like yes, Apple has gotten much better giving you controls now like you can easily turn. Do you remember I'm this is this dates me but so I don't know I'm sure your notifications might be push now but like in the early days. There was a different thing local notifications and and push notifications and push notifications you could disable an iOS but local notifications you could not.
我认为有这样一个想法，特别是关于通知的，我们谈论着优化，但是通知被认为是不好的。它们是垃圾邮件，是噪音，是广告，是任何东西。但我不太明白，我想有两个方面：第一，是的，苹果现在已经变得更好了，给了你更多的控制，你现在很容易关闭通知。就我个人而言，我不知道你的通知现在可能是推送通知，但在早期的时候，推送和本地通知是不同的，虽然你可以在 iOS 上禁用推送通知，但却不能禁用本地通知。

So like everybody we were doing this elevate and I know do a link was back in like 2015 you're using local notifications because like you couldn't turn them off. Right and yeah, that's scummy I guess but like it was also cheaper to do than having push. So we had to build our own eventually Apple unified this. They made it so that notifications are all control the same place right but I guess my point being is that like you know in terms of retention I'm sure. Retention is bringing somebody back into the app right what could work better than like. Staking yourself on their home screen like when they're not using the app right is this like this got to be like your biggest one of your biggest surface areas on the retention team I have to imagine right.
就像每个人一样，我们也在进行推广，并且我知道2015年使用本地通知进行链接，因为你无法关闭它们。没错，这可能有些不道德，但是这样做比使用推送便宜。所以我们最终不得不构建自己的通知系统，最终，苹果公司将其统一管理。他们使得通知都可以在同一个地方进行控制，但我的观点是，以留存性能为例，我一定会坚信，在用户没有使用应用程序的时候，在他们的主屏幕上留下自己的印记会比其他任何留存策略效果更好。这一点无疑是你所在的留存团队中最重要的一个点，我想象中的。

Oh yeah I think the retention team at Dueling go has worked on a lot of different features of the years so we have the daily streak mechanic we have leader boards in Dueling go but then notifications is always a huge lover for driving retention and I joined the retention team as an APM but I think I was the first full time PM to focus full time on notifications but that just is like a sign of how valuable it is for our for our growth to opt for.
哦，是的，我认为Dueling go的留存团队在很多不同的功能上努力工作多年了，所以我们有每日连胜机制，在Dueling go中有排行榜，但是通知总是推动留存的一个很大的爱好者，我作为一个APM加入了留存团队，但我认为我是第一个全职PM专注于通知推送，这只是为我们的增长选择其价值的标志。

So we're 30 minutes is. Tell us what you've learned like what can others learn from all these tests that Dueling goes have done about creating impactful. Retentive notifications. Yes so some super basic learnings that we've had and I'm sure you'll link the Twitter thread and the show notes but having a very distinct tone and CTA in your notifications is really important especially when you're sending multiple notifications perhaps over even the same day so.
我们还有30分钟。告诉我们你从Dueling goes所做的所有测试中学到了什么，其他人可以从中学到什么有关创建有影响力的保留通知。是的，我们从中得到了一些超级基础的经验，我相信你会在推特串中看到，并在节目注释中列出，具有非常明确的语气和CTA在你的通知中非常重要，特别是当你在同一天甚至多次发送通知时。

Dueling go we have a few different notifications that we might send one of them is the practice reminder which I mentioned we have a really clear CTA of just hey it's time for your daily practice get back into the app to study Spanish for five minutes something like that and then. For users who are on a daily streak we send another notification if they haven't practiced at the end of the day like 10 PM which we call the streak saber notification. And it has like a more alarming tone that says last reminder or hey it's getting late things like that when you're sending multiple notifications it's really important to have a distinct purpose for each one just so from the user perspective it's more like. Duo is having a conversation with me he told me that I have to practice at 4 PM but I ignored him but now at 10 PM he's getting kind of scared because I might lose my streak so having that sort of narrative that you can talk to the user with with your notifications. It's really important so yeah you mentioned you mentioned tone there so like.
比赛模式下，我们会发送几种不同的通知。其中一种是练习提醒，如我之前提到的，我们会使用清晰的CTA，提示“现在是您日常练习的时间，请回到应用中学习西班牙语五分钟”之类的内容。 对于已经连续练习的用户，我们会在当天结束时（比如晚上10点）发送另一种通知，也就是我们称之为连胜通知。这种通知会有一种比较紧迫的语气，比如“最后提醒，已经晚了”，或者“时间不多了”，因为我们担心用户会失去他们的连胜纪录。 在发送多个通知时，每个通知都必须有明确的目的，这对于用户来说非常重要。这样用户会觉得Duo是在与他们对话，他们会听从第一个提醒，但如果忽视了第一个提醒，会先收到温和的提醒，如果再不理会，就会紧急通知，这样的过程非常简单明了。 因此，在通知中维持良好的口气非常重要，这样才能与用户建立好的互动关系。

Paragraph 1: What's an example of like I guess like there's alarmist versus not but you actually in the notification speak from the voice of the bird right duo which I just learned his name do you. Like who when you guys are writing copy for that like are you trying to be cute with it or is it like Friday basic.
第一段： 有没有一个例子，比如说一些人会过度警惕，但你们实际上在通知中是从小鸟的角度来表达，我刚刚才知道他的名字，你们在写这些通知的时候，是在试图让它变得可爱一些，还是基础一点呢？

Paragraph 2: Yeah we try to vary the tone because we've learned that like variations in tone is good as a user is getting multiple notifications over several days so some of our notifications are really cute and encouraging and positive like amazing job you're on a 10 day streak come back and do Spanish other times the duo might be more passive aggressive or. Yeah like you can practice Spanish if you want so we try to have that very easy so really funny with it like you really like term and me met up yeah I mean yeah make sense to.
我们会尝试变化语调，因为我们已经学到，不同语调的变化对用户很有好处。他们在接收多个通知时可以感受到好玩、鼓励和积极的语气，比如“太棒了，你已经坚持学习10天了，回来学习西班牙语”，另外有时候，小绿鸟会更加消极，比如“如果你想的话，可以继续学习西班牙语”。我们会尽量让语调变化得有趣幽默，比如“你确实喜欢和我见面”，这些都是有意识的调整，而不是胡乱猜测。

Paragraph 3: Are you writing those personally you guys have copy writers are yes so a lot of it so far has been done by PM's like myself but we also have content designers that help us review and write some copy but yeah for a while my job was just writing from the voice of the bird and. You're ready to use for do
你们是自己写这些还是有文案撰写员来做呢？我们有文案设计师来帮助我们审查和撰写一些内容。到目前为止，很多工作都是由我这样的项目经理来完成的。我的工作很长一段时间只是按照小鸟的语气进行写作。你们准备好使用了吗？

Paragraph 4: I mean I think it's worth pointing out a meta point here on I think marketing and something I believe in just in general is that like a opinionated voice that sounds distinct even if it's bad like even if it's annoying to some people is always I think going to perform better then like a bleached and very like. That's not the word sanitize just like a very like sterile a very sterile voice right that you're trying to appeal to everybody I don't think I personally and I think that sounds like with you as a data bears this out does not outperform right you'd rather have hits and in peaks and like a little bit like odd because again like even on you know somebody I I you know curate my notifications very heavily right and I still get a lot of them.
我觉得有必要指出一个元话题，即在营销方面，我认为有个人性化的声音，即使有些人觉得这种声音很烦人，但却比一个很极端、很洁癖、很不具辨识度的声音效果要好。试图迎合所有人的非常“无味”的声音不会比个性化的声音更成功，就像你的数据表现出来的那样。你更愿意看到一些独特的点和高峰，这也有点奇怪，因为即使你对通知进行了严格筛选，你仍会收到很多通知。

Paragraph 5: And so you know one of them makes me smile like I can imagine that's going to that's got to increase the you know likely I'm going to click in yeah exactly especially since we're sending these notifications up to a seven day schedule maybe the first three the user will just ignore but if there's a really wacky or fun or creative one they get on day four maybe that will bring them back so that's why we try to have variation and we obviously have the algorithm also optimizing which ones are most effective over there. Over that schedule
我想告诉你们，其中一款产品让我感到非常高兴，我想象它一定会增加用户的点击率，特别是因为我们将这些通知推送到七天的计划中。可能在前三天，用户会忽略它们，但如果有一个非常古怪、有趣或创意十足的通知在第四天出现，也许会吸引他们回来。这就是我们为什么要尽量保持变化，并且我们也有算法来优化哪些通知在计划中最有效。

Paragraph 6: Next up the next learning was keep it simple what what is what did that mean yeah basically what it sounds like essentially just keeping the messaging or whatever theme of the notification very simple this is especially true with newer users are users that aren't as engaged with the app. People for do a lot of work on maybe shorter streaks keeping the copy really short and concise this was especially true I think I forget whether it was iOS 14 or 15 where Apple started cutting off chopping off more text in the notification header so a lot of our existing notifications starting anymore yeah yeah they started getting chopped off near like the fourth or fifth word so then we we've been doing a lot of tests that just have notifications that three words like got three minutes study Spanish now and though things like that worked really well.
接下来，我们所学的是要保持简单，什么是保持简单？这是什么意思呢？基本上就是像它听起来那样，即使通知的消息或主题非常简单。这对于新用户或那些不太参与应用的用户尤为重要。由于有些人可能只会在较短的时间段内完成许多工作，因此将内容保持简短明了。这在iOS 14或15中尤其重要，因为苹果开始在通知标题中更多地裁剪文本。因此，我们的许多现有通知在第四或第五个单词附近就开始被裁剪了。因此，我们进行了很多测试，只把通知简化为三个单词，如“Got three minutes study Spanish now”等等，这些方法非常有效。

Paragraph 7: Because people as you said people get a lot of notifications from a lot of different apps so if you can like quick grab the users attention with like two or three words that has often reduced this cognitive load I mean it's just a product management general like a good you know anything that makes your users have to think hard is is probably going to have adverse effects to their affinity for the product or willingness to engage because you're just tiring them out you know what is interesting is for our more engaged learners for example if you're on a 50 day streak more complicated messaging can often work well to so within the notifications will reference specific mechanics like on dual and go you can use a streak freeze to save your streak if you don't practice for a day so in some of our notifications will say hey you're out of streak freezes don't lose your 50 days streak and obviously that messaging is a little bit more complicated than something we send to less engaged users but we do have this like variation in messaging that we send for less or more engaged users and so I would understand that that's an input to the algorithm as well as like what if there are 50 return or like first return or something like that that will help you like you'll you'll choose different copy or different notification based on that.
正如你所说，人们会从许多不同的应用程序获得大量通知，因此如果你能用两三个词来快速吸引用户的注意力，这通常能减轻这种认知负荷。这只是一个产品管理的基本原则，任何让你的用户需要思考的东西都可能对他们的产品亲和力或参与意愿产生不良影响，因为这会使他们非常疲倦。有趣的是，对于我们更加投入的学习者，例如如果你坚持了50天的连续学习，更复杂的信息通知通常也会有好的效果。因此，在通知中我们会引用特定的机制，例如在Dual and Go上，你可以使用“冻结连续天数”功能来保存你的连续学习天数，如果你一天没练习的话。因此，在一些通知中我们会说：“嘿，你的‘冻结连续天数’用完了，不要失去你的50天连续学习天数”，显然这种信息传递比我们发送给不太投入的用户的信息复杂一些，但我们确实有发送给不同投入程度的用户不同的信息。因此，我认为这也是算法的一个输入，例如如果有50个用户返回或者首次返回，你可以选择基于它们选择不同的文本或不同的通知。

Paragraph 8: Yeah so it's actually not as complicated we don't have as many parameters within the algorithm itself but what we do is we set specific criteria on those copy templates so for something like okay so this is a certain groups can't get our notification. Exactly.
实际上，这并不像我们想象的那么复杂，我们的算法本身没有太多参数，但是我们会在副本模板中设置具体的标准，以便于某些群体无法收到我们的通知。就像这样。没错。

Paragraph 1: So the band it will just optimize for the eligible templates that a given user is eligible for understand got it that's so cool and so the next learning was keep titles under 30 characters which I think you can already explain it and they keep the simple thing but the next ones are interesting emojis work best in the title not the body that just seems so goofy by the data proved it out like how do you all use emoji and name what were you doing in the body that just didn't work.
那么，这支乐队将仅优化给定用户有资格使用的适用模板，明白了，很酷。接下来的经验是标题保持在30个字符以下，我认为你已经可以解释了，而且要保持简单。但接下来的一些内容很有趣，即表情符号在标题中更有效，而不是在正文中，这似乎很幼稚，但数据证明了它的有效性，你们如何使用表情符号呢？在正文中使用它们的效果不佳。

Paragraph 2: Yeah so emojis are another like vector for how we optimize notifications people generally respond well to different emojis what we have noticed is that generally emojis work better in the title of the notification rather than the body my hypothesis is that in the title emojis work really well as an attention grabber whereas if you have an emoji in the body it makes the notification look a little bit noisy which at that point is tuning it out. So yeah we've tested with this and it seems like it works better if you have the emoji in the title and also the emoji as the leading character rather than at the end of the icon right you're having like a customized like aside from your app icon which gets shared in the notification like you can add even like a topical extra icon that like kind of draws in I'm the same way like when I went when I read text with like emoji embedded within I boomer brain like I just can't like like I this too much like I can't I can't parse this right becomes very noisy so that that that's intuitive see this is thing I had all this vision to begin with we didn't need to run all these tests I could have told you just to be getting after once I've seen the data then I'm certain that I was right.
意思是：表情符号是我们优化通知的另一种方式，人们通常会对不同的表情符号有良好的反应。我们注意到，通常表情符号在通知标题中比在正文中更有效。我的猜测是，在标题中，表情符号可以很好地起到吸引注意力的作用，而如果在正文中使用表情符号，则会使通知看起来有些嘈杂，这时人们可能会忽略它。我们已经测试了这个假设，似乎如果标题中有表情符号，并且表情符号作为首个字符而不是在图标末尾，则情况会更好。您可以添加一个主题额外图标，用来吸引人们的注意力，就像您的应用程序图标一样，在通知中被共享。当我阅读嵌入表情符号的文本时，由于我的“老年痴呆脑”，我无法解析，这使得阅读变得很嘈杂，所以这很直观。我本来就有这个想法，没必要做这些测试，一旦我看到数据，我就相信这是正确的。

Paragraph 3: On the subject of notifications before we move on this wasn't actually shared in the tweet storm or tweet thread that that will link to but as a failed experiment you talked about how using a marketing tone and notifications just doesn't work to tell me the thinking behind that and and what experience you've experienced you've run that just fell flat. Yeah it's a it's a really interesting experiment that we ran recently basically we want learners to tap on our notifications because when you tap directly on a notification it links you directly into a lesson and that increases the completion rate of doing a lesson. We have this incentive to get users to tap directly on the notification rather than just navigating to to the app on their own so we wanted to test copy templates that explicitly instructed users to do that things that said tap here to start a quick lesson as it so happens that sort of tone doesn't work really well.
在我们继续之前，关于通知的问题，这不是在推特风暴或推特线程中分享的，但你谈到了一次失败的实验，你谈谈背后的思考和你所经历的经验，为什么使用营销口吻和通知不起作用，你的实验失败了。是的，这是一个非常有趣的实验，我们最近进行了这个实验，基本上我们希望学习者点击我们的通知，因为当你直接点击通知时，它会直接链接到一个课程，这会增加完成课程的完成率。我们有这样的激励，希望用户直接点击通知，而不是自己导航到应用程序中，所以我们想测试明确指示用户执行这些操作的复制模板，例如“点击此处开始快速课程”的语言，但实际上，这种语气并不起作用。

Paragraph 4: In our analysis we sort of were thinking that this is because it sounds more as like an ad or like you're getting a discount from sort of app duo is like tap here you be like what you doing man yeah it sounds more like a brand friends did that right. It's going to have here to respond to my text yeah it just doesn't work. Again it ties back to the idea that like an organic tone of having a character talk to you works better than having like a brand talk to you from a marketing standpoint. Yeah that's just a great lesson I think in in app marketing sometimes it is we feel like we need to be grown up marketers and like adopts all the like marketing best practices and you know. It doesn't really work in a lot of these cases and I think it's a really important thing to keep in mind.
在我们的分析中，我们认为这可能是因为它听起来更像一个广告或者你从“app duo”获得了折扣，就像是“点击这里”，你会觉得“嘿，你在干嘛呢？”它听起来更像是一个品牌的朋友。回应我的短信有点奇怪，它根本行不通。再次回到有一个角色与你对话的有机语调比品牌从营销角度与您对话更有效的想法。是的，在应用程序营销方面，这只是一个很好的教训，有时候我们感觉自己需要成为成熟的营销人员，拥抱所有的营销最佳实践，但在许多情况下这并不起作用，我认为这是一个非常重要的事情要记住。

Paragraph 5: The next thing I want to talk about I think this is something you actually blogged on was streak flexibility. So yeah tell me tell me about those experiments and then and what you'll learn. Yeah so as I mentioned earlier on Dueling go you have the daily streak so every day you practice your streak goes up and that was a huge mechanic for our attention in the early days just because it gives you the reason to come back every day it's it's and then every other social app copied it like in the world right exactly yeah it is the best retention mechanic you can think of what we learned over time is that this sort of streak mechanic is very punitive in the sense that.
下一件我想谈论的是，我认为你实际上在博客上谈到过的连胜灵活性。所以，告诉我关于那些实验以及你学到了什么。正如我之前提到的，在《决斗围棋》中，你有每日连胜，所以每天练习你的连胜会增加。在早期，这是我们注意力的一个巨大机制，因为它给您一个每天回来的理由，而且几乎每个社交应用程序都复制了它，是世界上最好的保持机制。随着时间的推移，我们学到的是，这种连胜机制在某种意义上非常惩罚性。

Paragraph 6: If you miss one day by a chance then you lose your streak and that is extremely demotivating there's a huge churn point where if you lose the streak even if you're very committed beforehand just the experience of losing a streak is super discouraging and uses will just fall off the platform so we wanted to add more flexibility with how you can go about your streak so we have this in app item that you can purchase or we also give it out sometimes as a reward for doing certain things.
如果你偶然错过了一天，那么你就会失去你的连续记录，这会非常消极。在这个关键时刻，即使你之前非常坚定，失去连续记录的经历仍然是极其令人沮丧的，用户会放弃使用这个平台。因此，我们想增加一些连续性的灵活性，使您可以更灵活地记录。因此，我们在应用中提供了一个项目，您可以购买它，我们有时也会作为某些事情的奖励而赠送。

Paragraph 7: Which is the streak freeze basically you can miss a day and your streak will stay the same you can come back the next day and continue learning and increasing your streak so adding the streak freeze was amazing for attention just giving us a little bit of slack just makes the process of keeping a streak a lot easier and it it's less punitive and punishing for a user to not have to come back every single day and then we've done so many optimizations around the streak freeze.
streak freeze 是一项重要的功能，即使错过一天，你的连续学习天数仍然保持不变。你可以在第二天回来继续学习并增加你的学习天数。这项功能极大地提高了用户对Duolingo的关注度，因为它给了我们一点宽松的时间，在保持学习连续性的过程中更加轻松。不会每天都来学习也不会受到惩罚和责罚。此外，我们对 streak freeze 进行了多处优化。

So you can buy streak freezes with our in app currency called gems or as I mentioned you can get them as a reward for completing your daily quest for example we've optimized the drop rates of how often they drop as a reward we've also increased the number of streak freezes you can have at a certain point so now you can equip up to two at a time essentially you can miss two days and still keep your streak.
因此，您可以使用我们的应用货币宝石购买漏刷冻结，或者如我所提到的，您可以通过完成每日任务获得它们作为奖励。例如，我们已经调整了漏刷冻结掉落的频率作为奖励，同时增加了您可以拥有的漏刷冻结数量。现在，您可以同时装备最多两个漏刷冻结，基本上您可以漏掉两天，仍然保持您的打卡纪录。

Obviously like as we're doing all these optimizations we're trying to keep in mind what we call the sanctity of the streak to make sure it's still make it meaningless right exactly and generally as we've iterated we've seen retention gains and people are still ascribing value to the streak even though there's a chance that you missed a few days in between our all the mechanics only are proactive so like if I if I just miss a streak day and I'm like I missed it can I recover it or is it is a do I still have a chance to do it.
很明显，我们在进行所有这些优化时，一直记得要保持连续性的尊严，以确保它仍然没有失去意义，确切地说，随着我们的不断迭代，我们已经看到了留存率的提高，人们仍然将价值归结于连续性，即使有可能错过几天，所有的机制都是主动的，所以如果我错过连续性记录的一天，我可以恢复它吗，或者我仍然有机会完成吗？

Is it is a do I still have to like put this freeze in place before. Yeah so you have to have the streak freeze equipped we have a separate mechanic also called the streak repair and this is like example on dropping $1.99 on that yeah so that has historically been like an in app purchase or it's a it's a perk that you get in our subscription super where you get a streak repair where in the case that you used all your streak freezes miss today and actually lost your streak you can still get it back.
在我使用冰冻保护前，我是否仍需设置它？是的，你必须装备当前连胜冰冻保护，我们还有一种名为连胜修复的机制，比如说你可以花费1.99美元购买它。这在历史上一直是应用内购买或我们的订阅超级奖励中的特权。如果你今天用完了所有的连胜保护措施，导致失去了连胜，你仍然可以使用连胜修复来恢复它。

But over time we're also trying to make sure that all these features are aligned and not yet doesn't get out of control I mean but it does feel like. I mean as long as there's some activity right like as long as users are thinking about their streak it's kind of what you want to like okay do I care. But then you would go back I guess the sanctity thing it's like at some point if you over optimize for streaks it's own a streak is only important in so much as people are doing language learning right if they're not doing language learning then the streak is just an addiction mechanic and exactly you're not you're not making the world a better place anymore right which is a constant risk of being a product mixture right optimizing create something terrible for the world right.
随着时间的推移，我们也在努力确保所有这些功能能够协调一致，而不会失控。我的意思是，感觉有点像。只要有些活动，用户正在考虑他们的连胜，这就是你想要的，你会问自己，我在意吗？但是，我猜你会回到它的神圣性，有些时候，如果你过于优化连胜，那么连胜只有当人们学习语言时才显得重要。如果他们不学习语言，那么连胜就只是一种成瘾机制，你不再使世界变得更好了，这是产品混合的一种常见风险，优化会为世界创造出一些可怕的东西。

So it's an interesting balance but the streak the streak as I've always wondered about that like the how do the how do the downsides of a failure to maintain that thing. A fact and I'm I'm sure like it doesn't become a dominating effect right like that the people aren't losing streak so often that it it makes a difference but I'm sure at this stage like when you're optimizing everything you can.
这是一个有趣的平衡点，我一直对连胜连败的影响很好奇。如果不能保持这种状态，会带来什么负面影响呢？虽然可能并不会成为主导因素，人们也不会太在意连败，但在优化所有可能的因素时，连胜连败肯定会受到关注。

Like there's more to be you know had there and then also just like you know from a branding perspective makes the conversation like do is a little more chill you know what I mean like just going to give you a break right like letting that weekend there's something I gotta go back like I'm trying to be all over me. I think it's great and actually I've thought a lot about apples fitness streaks and there's times when you're sick where you can't work out or the standard locations where you're a flight.
这段话的意思是，就像品牌的角度来说，它让交谈更轻松一些，同时表现出你的多面性。这让人们感到轻松，不必觉得自己必须全身心地投入工作。我认为这很好，我也经常思考苹果健身中断的问题，例如当你生病或旅行时，你无法进行锻炼。

Oh my gosh get out of here and it's like you might actually do more harm than good. I turn it all this off I like when I got my apple watch I was all about it and then I was for a while it's like come on like you're not you're not adapting to me right like I'm done with this.
天哪，赶紧离开这里，你可能会造成更多的伤害，这样说吧，当我得到我的苹果手表时，我真的很喜欢它。但是后来有一段时间，那个手表好像没有适应我的需求，所以我不想用了。

So while it's breaking the rules it's like breaking the rules in it in a healthy way I mean I think it's pretty cool we actually talk to the former CMO of Tinder. Oh yeah and Tinder has a few things like that we talked about how you know some of their monetization is around breaking the rules and it's interesting how you can break the rules in good ways and you can break the rules in bad ways and the balance is is finding ways to break the rules in ways that are actually positive and beneficial and that's what the users will respond to not just by your way out of something it's like okay it was like you know right yeah yeah Yeah, there's no Pitch that one he's your guys to test that one.
在违反规则的同时，以一种健康的方式来做，就像以一种健康的方式来违反规则。我觉得和前 Tinder 的 CMO 聊天很酷。对了，Tinder 有一些像这样的事情，我们谈到了他们一些盈利方式是围绕着违反规则的，有趣的是你可以用良好的方式违反规则，也可以用不好的方式违反规则。平衡是找到一些能够用积极和有益的方式违反规则的方法，这才是用户真正会回应的。不仅仅是通过一些方式来逃避责任。嗯，这个的测试是你们的任务。

Just tell me about some ideas. Yeah, I'm so glad guys they gave me some great ideas I'm honestly. Oh man.
只要分享给我一些想法就行了。是的，你们给了我一些真棒的思路，我感到非常高兴。天哪。

Okay, we do need to get to wrapping up but there was one more question I had was the widget so widgets have been huge the last few years and tapal you know redesigned the whole home screen.
好的，我们需要开始总结了，但我还有一个问题，就是关于小部件的。小部件在过去几年中非常流行，tapal重新设计了整个主屏幕。

What kind of results have y'all seen from having the the street widget and and I imagine you keep stats on you know the number of users who actually have it on their home screen the number of people who tap on it has it been a pretty big driver of retention and kind of visibility for the app.
你们使用街道插件后，看到了什么样的结果呢？我想你们统计了一下使用该插件的用户数量，以及有多少人点击了它，这对于应用的留存率和可见度来说是否是一个很大的推动因素？

Yeah, it's a great question so on iOS we have a streak widget it's basically a home screen widget that shows the current length of your streak and it has a cute picture of duo over the course of the day as you don't practice duo gets like angry and angry or like more worried so it's kind of like almost like a time got you like thing that's living on your home screen to get you to practice that's so cool.
是的，这是一个很棒的问题。 在iOS上，我们有一个连胜小部件，它基本上是一个主屏幕小部件，显示您当前的连胜长度，并且随着您不练习而变得越来越生气或担心，具有可爱的Duo图片。因此，它几乎像是一个在您的主屏幕上运行的“时间提示器”，以便您练习。太酷了。

Yes, so we launched it this year and in fact it exceeded our expectations that how good of a retention lever it is and there's two reasons for one is that for users that are active that are honest streak it is like a constant notification that lives on your home screen reminding you about your streak but it's also cuter than a notification it's more adorable and it's something that you opted into to adding to your home screen.
是的，我们今年推出了它，事实上它超出了我们的预期，它是一个非常好的留存杠杆，原因有两个：一是对于活跃用户，它是一个不断在你的主屏幕上提醒你坚持的通知，但它比通知更可爱、更可爱，这是你添加到主屏幕上的东西。

So it feels a lot less spammy than a real notification but it's a constant reminder like hey this is your streak do you actually do some like educational content in there like ombre is man does it have any kind of like actually reinforce learning or is it mostly about the streak that is on a road map we don't have that yet but there's a lot of like optimizations and like.
这种通知并不像真正的垃圾邮件那样令人讨厌，但它会不断提醒你保持记录，就像是"嘿，这是你的连续记录，你确实会在里面添加一些教育内容吗？" Ombre这个人在里面做了一些什么事情吗？他是否真正加强了学习，或者这只是一个记录的计划而已？我们目前还没有这样的计划，但是我们正在进行很多的优化和改进。

I think just get open the app you could just turn up a number said your streak is three right like a very phone didn't widget and like I've built these features for Apple before and if I've learned like if you don't make it on its own something that provides value it's nobody's going to be really excited about it but I give you're putting like some character into it you're like making it something and a little bit of an experience in itself and like a vocation emotion just with it then you have a chance I guess my question when we're going to be able to do that.
我认为只要打开这个应用程序，你就可以看到一个数字，它显示你的连续记录是三，就像一个没有窗口小工具的手机一样。我之前为苹果开发了这些功能，如果我学到了什么，那就是如果你不让它成为自己提供价值的东西，那么没有人会对它感到兴奋，但是如果你为它注入一些个性，让它成为一些小体验，带有一些情感，那么你就有机会了。我想问的是，我们什么时候能做到这一点呢？

I guess my question would be like doesn't move the needle for you all unless you get people to actually add it right like are you trying to like actively push that are you just saying it's successful for the people that already have it installed.
我的问题可能是，除非您确实让人们添加它，否则它对您所有人都没有太大影响，就像您是否尝试积极推广它，还是只是说对于已经安装它的人而言它是成功的。

Yeah, that's a great question so there's basically two things that we have to work on with the widget one is getting users to install it so we have like in app promotions when you finish this a session or a lesson will say hey we have a widget you should install it and when we tested showing that card it actually like let us surprisingly high number of users to install the widget so our widget DAU graph has kind of been going hockey stick recently which has been awesome but yeah after we get the user to install it there's also the optimizations.
这是个很好的问题。对于小部件，我们基本上需要做两件事情。首先是让用户安装它，因此在用户完成一次课程或会话后，我们会推出应用内推广，告诉他们可以安装小部件。当我们测试显示该卡片时，我们发现有意外高数量的用户安装了这个小部件，所以我们的小部件DAU图最近一直呈现出高斯曲线，非常棒。但是，当我们让用户安装了小部件后，还需要进行一些优化。

What the widget actually looks like and both of those levers getting you just to install and making it valuable are both levers for retention for us yeah it's interesting like on a product that's a decade plus now I'm not sure if something like that right that there's still still more to be.
小部件的实际外观以及这两个杠杆让您仅安装并使其有价值，对我们来说都是保留杠杆。是的，很有趣，就像一个十年以上的产品，我不确定是否存在这样的东西，还有更多可以做的。

And like you said like you know it's still still growing 50% DAU growth or whatever it was over the last year is pretty crazy right I would have never have in 2013 when the app store was nice and there was this app do a lingo which we're like well they're giving it away free like we didn't really understand like I would never have guessed they would have gotten to the point today but then again I didn't know I didn't know what you guys were doing and now it kind of makes sense so it's pretty amazing.
就像你所说的那样，它仍在继续快速增长，去年增长了50％的DAU，这真是令人惊讶。在2013年应用商店还很友好，还有一款应用叫做do a lingo，我们会说：他们免费提供这个应用，但我们并不真正理解它，我从未想过他们会发展到今天这个地步。但另一方面，我不知道你们在做什么，现在我明白了，所以这真是太神奇了。

Yeah so I think it's a great place to wrap up such an incredible app what an incredible journey to from from early in the app store to publicly curated company and it's so cool all the different ways that they do all incorporate testing and data to to help influence that and to continue to grow and continue to make it a great experience.
嗯，我认为这是一个很好的地方，可以总结这么一个令人难以置信的应用程序旅程，从早期的应用商店到现在成为公开策划的公司，真是太棒了！他们采用各种不同的方式来进行测试和数据分析，以帮助影响应用的发展，不断增长，并保持优秀的用户体验。

Anything you want to share in closing I know doing is still hiring like crazy you know the rest of tech there's a lot of people looking for jobs so anything you want to share there yeah now we're very fortunate that we're still in good standing and we're still hiring so look up our careers page if there's any positions that interest you and yeah download do a lingo if you haven't we love we love our users that we love hacking yeah always be real hacking exactly.
在结束时，如果你有什么想分享的，我知道我们还在不断招聘，科技界其他地方也有很多人在找工作，所以如果你想分享什么，欢迎。现在我们很幸运，我们仍然处于良好状态，仍在招聘，所以如果有任何你感兴趣的职位，请查看我们的职业页面，并下载 Do a lingo，如果你还没有的话。我们爱我们的用户，我们热爱黑客精神，永远保持真实的黑客精神。

Yeah all right thanks so much this is a fascinating conversation and again I appreciate so much that how open do lingo is to share this stuff so again we'll share links in the blog in the show notes for for a lot of really great blog posts and tweet threads and things like that so thanks again thank you so much David and Jacob.
好的，非常感谢。这是一次非常有趣的交谈，并且我非常感激Do Lingo非常开放地分享这些东西。我们将在博客和节目注释中分享许多非常棒的博客文章和推文等链接，因此再次感谢David和Jacob。

Thanks so much for listening if you have a minute please leave a review in your favorite podcast player you can also stop by chat dot sub club dot com to join our private community.
非常感谢您的聆听。如果您有一分钟的时间，请在您喜欢的播客播放器上留下评论。您也可以访问 chat dot sub club dot com，加入我们的私人社区。