Future Academy: Evaluation of our first program

The first round of Future Academy has come to an end

The aim of this post is to provide a brief evaluation and reflection on Future Academy - our first program (and experiment). Future Academy aimed to equip university students and early-career professionals worldwide with the thinking, skills, and resources they need to pursue ambitious and impactful careers. For the full version, please read this document.

It was a free six-month program consisting of four in-person weekends with workshops, presentations, socials, and monthly digital events. Furthermore, the 21 fellows worked on an impact project with an experienced mentor and received professional coaching to empower them to increase their impact and become their best selves. Upon completion of the program, all participants went to a global impact conference (EAGx Nordics) where four fellows presented their projects. We awarded stipends of a total of $20,000 to the best projects. The projects included a sentiment analysis of public perception of AI risk, a philosophy AI alignment paper, and an organization idea for improving research talent in Tanzania. Our faculty included entrepreneurs and professors from Oxford University and UC Berkeley.

Note that this document attempts to assess to what extent we’ve served the world. This involves an assessment of the wonderful fellows who participated in Future Academy, and our ability to help them. This is not meant as an evaluation of peoples’ worth nor a definite score of general abilities, but an evaluation of our ability to help. We hope we do not offend anyone and have tried our best not to do so, but if you think we write anything inappropriate, please let us know by reaching out to sebastian [at] impactacademy.org.

Main results and successes

• We confirmed a key hypothesis underlying Future Academy - namely that we can attract promising and talented people who i) have no to moderate knowledge of Effective Altruism and longtermism, ii) are coming from underserved regions (e.g., 40% came from Southern and Eastern Europe), and are more diverse (e.g., 56% were female).
• We created a bespoke primary metric called counterfactual expected career contribution (CECC) inspired by 80,000 hours’ IASPC metric and Centre for Effective Altruism’s HEA metric. We think the total score was 22.8, and ~ four fellows made up the majority of that score.
• We had a control group of the top 30 rejectees (only 14/30 responded to our surveys). Thus, this was a quasi-experiment - not a randomized control trial. 70% (10 people) of the control group didn’t do anything similar to Future Academy, and of those who did, we think it’s likely that two of them would’ve been better off by doing Future Academy, whereas one of them was probably better off not doing Future Academy and one where we are unsure. The control group had a final ECC that was 40% lower than the fellows, which might support our assumption of being able to create counterfactual impact via programs like Future Academy. This supports our assumption of being able to counterfactually engage people in EA.
• The fellows were very satisfied, with an average satisfaction score of 9.6/10 (with an 86% response rate). We asked the control group how satisfied they were with what they did instead of Future Academy. The average satisfaction was 6.7.
• Future Academy seems to have increased the fellows' number of connections relevant for doing good substantially. The average number (before) was 3. The average number (after) was 17. This corresponds to an increase by a factor of 4.5 (compared to the control group, which was increased by a factor of 2).
• 92% of the fellows reported that they changed their career in ways they think will make it more impactful (compared to 71% of the control group).
• Our focus on well-being and general human development via coaching appears to have been a success. For example, two out of the four well-being metrics had a statistically significant increase with a moderate-strong effect size. The two other metrics didn’t change in statistically significant ways. Relatedly, it increased the overall engagement with the program (fellows reported an average engagement increase of 30.5% due to coaching) while providing career-related outputs (e.g., being more ambitious and optimistic about the impact they can have).
• We were excited about awarding $20,000 to the best projects.
• The total cost (including staff salary) of Future Academy was $229,726, which corresponds to $11,000/fellow and ~ $10,000/Counterfactual Expected Career Contributions (CEEC). The total FTE of the core staff was ~ 2.

We’ve decided to run another version of Future Academy where we will continue to primarily target people who have no or moderate knowledge of and engagement with EA/longtermism and are from underserved regions and groups.

Main challenges and mistakes

• Impact evaluation presents multiple challenges. We take all of the results with substantial uncertainty and worry that our conclusions could easily change within six months. Relatedly, the assessment of our control group could be much more rigorous.
• Three fellows reported having had difficult experiences with one of the organizers - e.g., feeling that the organizer had been too pushy with some of the EA ideas. This made them feel uncomfortable and less prone to engage with the effective altruism community. None of the fellows are hurting in any way - they shared the information to help Future Academy thrive. We took this feedback very seriously, solicited advice from around ten people, and considered a wide range of options for moving forward with the organizer (including whether the person should continue with us). We eventually decided to continue as we trust the person’s ability to grow and saw that the person created a convincing development plan for improvement.
• We needed to spend more time and resources on marketing and attracting applications to the program. We weren’t particularly systematic in the recruitment process, as multiple individuals were involved, and we want to set aside more time and resources for both marketing and application processing. With a simple budget of $2,000 at least three participants found the program through paid advertising (“eyeballs can be bought”).
• The fellows didn’t get as much out of the Impact Projects and mentoring as we would’ve liked. For example, 3/18 didn’t hand in a project, only four fellows worked in groups, and some mentor-mentee-pairings seemed not to have taken off. However, compared to other programs (e.g., SERI), it seems as if this was a high rate of project completion.
• We need fewer normal lectures and more time for active learning and a flipped classroom approach. Relatedly, the exercises prepared for the workshop elements were of varying quality - e.g., they hadn’t been test-run.

Conclusions for Impact Academy

Overall, we think Future Academy was a successful experiment as we were satisfied with how we ran the program and the outcomes of the program. However, there was significant room for improvement, and we don’t want to run Future Academy in its exact form again. We’ve decided to run another version of Future Academy where we will continue to primarily target people who

• Have no or moderate knowledge of and engagement with EA/longtermism and
• Are from underserved regions and groups.

We’ll also update the program to reflect best practices within education, the science of learning, and other programs we think highly of. Finally, we’ll also explore the feasibility of targeting early-mid career professionals as the wider community seems to be very interested in individuals with 3+ years of experience.

You can learn more about the other project (an AI governance fellowship) we will be running here.

Recommendations for others who share our aims

Based on our experience with Future Academy, we think these recommendations might provide value to others trying to help build future changemakers, e.g. the Effective Altrusim community:

• More focus on recruiting from underserved communities (like Eastern and Southern Europe and specific regions in Africa and Asia).
• Consider adding a component around well-being and general human development via coaching. For example, two out of the four well-being metrics had a statistically significant increase with a moderate-strong effect size. The two other metrics didn’t change in statistically significant ways. Relatedly, it increased the overall engagement with the program (fellows reported an average engagement increase of 30.5% due to coaching) while providing career-related outputs (e.g., being more ambitious and optimistic about the impact they can have).
• Learn from educational best practices. Many things related to field-building can be modeled as education. From the design of programs (e.g., by backward chaining from where you want fellows to end up) to the impact evaluation (it can be similar to grading hard things like essays). See this excellent blog post by Michael Noetel for more.
• Be more rigorous and systematic about impact evaluation. E.g., creating a baseline estimate or a simple control group by sending a survey with a $20 Amazon gift card to top rejectees led to a surprisingly high response rate. That said, impact evaluation is hard, and we’d be interested in exploring some form of external evaluation. For instance, multiple organizations could hire someone to do it collectively for them.
• Run more ambitious experiments, including experiments that are not EA-branded.

Acknowledgments

We are incredibly grateful for having been able to deliver Future Academy to everyone who supported us. Our funders who believed in the idea. Our speakers who were eager to give some of their time and travel. Our fellows who took a chance on us by being open-minded (and crazy enough) to join a completely new program. Our mentors who guided the fellows while they were finishing their project. Our own mentors (very much including Michael Noetel) for providing us with ongoing feedback and guidance and helping us set a high bar. To the people who provided feedback on our evaluation (Anine Andresen, Henri Thunberg, Eirik Mofoss, Emil Wasteson, Vaidehi Agarwalla, Cian Mullarkey, Jamie Harris, Varun Agrawal, Cillian Crosson, Raphaëlle Cohen, and Toby Tremlett). Finally, to the wider community of do-gooders who collaboratively provided input on everything from the program design to the impact evaluation.

Thank you!