That program worked. So we’re making it bigger.

The majority of our grants have gone toward compute and API credits for people running evals, training small models, and replicating safety research, typically for a few hundred dollars. Our Technical AI Safety Projects Sprint has been great at surfacing these.

But some of our most impactful grants didn’t look like that at all:

• AI Safety Poland received $4,800 to organize meetups across the country, covering venue and tooling costs, and building a national community from scratch. They’ve had hundreds of attendees, and dozens of people they’ve referred have ended up taking a BlueDot course.

• Justin Dollman received $3,000 to lead our weekly evals reading group, coordinating a growing community of people learning about and working on model evaluations. We’ve since funded more reading groups on AI governance and AI x cyber and are excited about scaling these up from ~120 to 300 attendees per week.

• Sally Gao received $1,000 to run AI Safety meetups in New York, hosting guests such as Alex Bores. Her next event on the State of AI Safety is coming up on April 23rd!

• Eitán Sprejer received $4,200 in facilitator stipends to run our Technical AI Safety Projects Sprint at BAISH in Buenos Aires, accessing a new and underleveraged talent pool for AI safety work.

• Aaron Maiwald received $5,000 to attend a biosecurity conference in DC, travel to SF, and connect with senior folks in the field to accelerate his journey.

• Zac Saber received $8,000 to drop out of EF and validate an AI safety-focused startup instead.

None of these grants fit neatly into “compute for a project.” But they were some of our highest-impact bets. As the program picked up speed and more of these came in, we started treating rapid grants as small, focused bets on talented people doing or experimenting with high-impact projects - research, fieldbuilding, talent acceleration, you name it. In some cases, we approached people directly and pitched them on applying for work they were already doing.

Going forward, we’re expanding the official scope to match our internal understanding: Rapid Grants now fund much more than compute - we’re excited to back work on events, talent acceleration, BlueDot community building, and more. The bar for funding hasn’t changed - if anything we’ve raised it as we became more calibrated - we still look for concrete work in progress, a specific cost that’s the bottleneck, and a reason for us to believe the work matters for making AI go well.

Grant sizes now go up to $10,000 to allow us to make bigger bets for more impact. Decisions will still be fast - we’re targeting around five working days. For grants above $5,000, we may hop on a quick call.

Who should apply

If you’re in the BlueDot community - a course participant, alum, facilitator, or wider community member - and you’re doing something high-impact in one of our focus areas that a grant would accelerate, apply.

If you’re on the fence, the default, as always, is simple: apply.

Apply and see the full list of public grantees and program details at bluedot.org/grants/rapid.

This short blog post is intended to help you figure out if you should spend time on an application.

FAIGC is at its core about the governance of frontier AI systems - AGI, ASI - the most capable models being built by a handful of labs, the decisions governments and other actors need to make as those systems become more powerful, and what happens as general capabilities rapidly approach and exceed human-level.

If your guiding question is “how should we govern AI systems that might be smarter than humans within the next few years” you’re in the right place!

What FAIGC is not

It is not a corporate AI governance course

• If your goal is ISO 42001 compliance, responsible AI frameworks for your product team, or helping your organization navigate the EU AI Act - that’s real work and we respect it, but it’s not what we focus on. You should look at something like IAPP’s AIGP certification or ISO 42001 programs instead.

It is not an AI ethics course

• We don’t cover algorithmic bias in hiring tools, fairness metrics, or the full range of social impacts of current AI systems. If that’s your focus, you’re better served by something like this or this.

It is not introductory

• We require the AGI Strategy course or some equivalent background. If you’re still building your understanding of why this matters, apply to the AGISC first. It too is free and runs every month!

What gets you in

It’s not (just) your CV - we’ve rejected plenty of people from prestigious institutions with impressive resumes who wrote three-word applications, and we’ve also accepted sharp people with more modest titles who showed they were already sprinting ahead.

What we’re looking for: you’ve already started engaging with frontier AI governance specifically. You can name a concrete gap between where you are and what the course provides. And your post-course plan better be more specific than “apply for fellowships” - a line we see in dozens of applications every round.

Your application itself is evidence for us too. We use it to infer how you’ll show up in a live discussion session of seven to nine people working through hard material together. Effort, agency and clarity go a long way.

A note for AGI Strategy course graduates

We view completing our AGI Strategy course as a prerequisite and not as a ticket! We may still reject AGI Strategy grads who apply. If you’ve completed the AGISC and then started building something, writing something, or working on something in frontier AI governance, tell us about that. But if you completed it and your main next step is taking this course, you may need more time.

As a general heuristic, before you apply, ask yourself these three things

• Am I focused on frontier AI, or AI in general?

• Do I have a specific reason to take this course now?

• Will I act on this fully within six months?

We’re building a pipeline into the institutions that govern frontier AI and are looking for people who have a good chance to be in those institutions - not people adding a line to their CV.

If you’ve read this and you’re thinking yes, this is me - apply here.

How it works

At all times, only assume we’ll cover costs we’ve confirmed in writing for your specific project. If you’re ever uncertain, contact us before spending money you expect us to reimburse.

1. Submit a proposal here (under 15 minutes for most applications). We typically respond within 5 working days with one of three outcomes:

   • Accepted: We’ll confirm exactly what spending we’ve approved.

   • Clarification needed: We’ll ask follow-up questions to evaluate your request.

   • Not approved: We don’t think this meets our criteria, or we’re unable to approve for another reason.

2. Do your project, confident you can spend on what we’ve approved.

3. Claim reimbursement. We process most claims within 5 working days. If you didn’t end up spending the money, no problem - just don’t submit a claim.

What we fund

The following examples are illustrative; all decisions are ultimately at our discretion:

• Compute for a technical AI safety project (API costs, cloud GPU, training runs).

• Access to paywalled resources like articles, research papers, datasets, or textbooks.

• Hosting costs for your application or tool.

We expect most grants to fall between $50-100 for initial experiments to develop your proof of concept, then up to $500 once you have evidence of initial traction (a strong proof of concept or promising preliminary results). We encourage you to submit a separate application for each of these stages.

What we don’t fund

• Compensation for your time on the project.

• Equipment you’d reasonably already have (laptops, phones, external drives, etc.)

• General productivity subscriptions (ChatGPT Plus, Claude Pro, Cursor, Grammarly, etc. unless this is highly leveraged).

• Personal expenses.

Other funders may cover some of these - see this resources page for AI safety funding opportunities.

When should you apply?

Before applying, you should have a project idea. It doesn’t need to be a slam dunk, but you should:

• Be able to briefly articulate how it relates to AI safety.

• Have ideas for initial experiments you want to run.

• Have a rough estimate for how much those experiments will cost (I recommend using an LLM to help you calculate this).

If you’re unsure whether your application is strong enough, submit an application anyway! We’re not looking for perfection. Your facilitator can also help you stress test your project, so have a low bar for reaching out to them!

While you wait for a grant decision, you should continue to develop your project idea, do a deeper literature review, or start to run smaller experiments that fit on the free Google Colab GPU.

Eligibility

• You must be a current or past participant of BlueDot Impact’s Technical AI Safety Project Sprint.

• We reimburse via bank transfer (Wise) or PayPal (UK), so we cannot send payments to sanctioned countries.

Questions or feedback? Contact us.

Apply

Submit your proposal here - it takes under 15 minutes for most applications. We aim to get back to you within 5 working days.

When I first asked an LLM to code me that annoying function or, worse, that whole experiment pipeline, I felt guilty and anxious as if I was cheating and getting away with it. I’m sure you’ve felt this way too.

It can be easy (especially if you are new to AI safety) to feel like you should avoid this, to feel like you should write all of the code in your project yourself, understanding each line and all the libraries that you use.

There is a time and a place for this, but a 30hr research sprint is not it.

If you are doing a research sprint, you should use LLMs / coding agents for all of your coding. Below I give you three reasons why.

You don’t have time to learn both engineering and research

The skill you should build in a research sprint is research taste: choosing what experiments to run, forming hypotheses worth testing, and updating your intuitions when the results surprise you. This only develops through many iterations and 30hrs is not a lot of time, especially if you spend hours coding experiments that could take minutes with AI. You should be very explicit about which skill you are trying to train, and optimise for that skill alone.

You might be thinking “but I need to be sure my code is actually implementing the experiments I intended” and you’re absolutely correct. However, this is entirely compatible with vibe coding your whole experiment pipeline. Just get the same model to provide you with a summary of the code and keep asking it questions until you are confident it is doing it right.

You’ll do more research and do more writing

If you spend less time coding, you will cover more ground and do better research. You can go deeper, ask that extra question, run that extra experiment, discover that extra awesome result. You’ll develop better intuitions about how AI models behave and which research directions are worth pushing.

You’ll also have more time for writing. Sharing your project is what lands you that next opportunity and where all the best feedback comes from. Nobody will be impressed by how “human written” your code is. They will be impressed by your clarity of thought and the depth of your research.

Coding with AI is a skill worth practicing

AI models are amazing at coding. With Claude Code, you can now build an app in a weekend and a whole machine learning experiment setup in minutes. And this won’t change. For the rest of your life, LLMs will be faster at coding than you, no matter how much you practice. The most productive AI safety researchers use AI for coding, and so should you.

If you want to learn how to use AI for coding, ask your favourite LLM! They’re very good at giving you tips and tricks on how to use them better.

Conclusion

Don’t feel guilty about using AI for your coding. Learn to operate fast while staying in control. Be explicit in the skills you want to develop, and be ruthless in pursuing them. So, which skills do you want to develop in this sprint?

• Startups can become integrated in the AI supply chain, giving them good information about valuable safety interventions. Safety becomes a feature to be shipped directly to users by virtue of this market position.

• Better access to capital, talent, and ecosystem-building is available to for-profits than non-profits. VC funding dwarfs philanthropic funding, and there is little reason to believe that profitable safety-focused businesses aren’t possible.

• Joining a frontier lab is a clear alternative, but most AI deployment happens outside labs. Your marginal impact inside a large organisation is often smaller than your impact when founding something new. Equally, profitable businesses aren’t an inevitability. You should seriously consider working for or founding an AI safety startup.

Introduction

Markets are terrible at pricing safety. In the absence of regulation, companies cut corners and externalise risks to society. And yet, for-profits may be the most effective vehicle we have for deploying safety at scale. Not because the incentives of capitalism align by chance with broader human values, but because the alternatives lack the resources, feedback loops, and distribution channels to turn safety insights into safer outcomes. For-profits are far from perfect, but have many advantages and a latent potential we should not ignore.

Information, Integration, and Safety as a Product

For advanced AI, the attack surface is phenomenally broad. It makes existing code easier to crack. Propaganda becomes cheaper to produce and distribution of it becomes more effective. As jailbreaking AI recruiters becomes possible, so does the data-poisoning of entire companies.

Information about new threats and evolving issues isn’t broadcast to the world. Understanding where risk is most severe and how it can be mitigated is an empirical question. We need entities embedded all across the stack, from model development to deployment to evaluation. We need visibility over how this technology is used and misused, and enough presence to intervene when needed. ‘AI safety central command’ cannot provide all these insights. Researchers acting without direct and constant experience with AI deployment cannot identify the relevant details.

Revenue is a reality check. If your product is being bought, people want it. If it isn’t, they either don’t know about it, don’t think it’s worth it, or don’t want it at all. For-profits learn what matters in an industry directly from the people they serve, giving the best insights money could buy.

This is not to say that AI safety non-profits aren’t valuable. Many do critical work which is difficult to support commercially. But by focusing entirely on research or advocacy and ignoring the commercial potential of their work, organisations cut themselves off from a powerful source of feedback. Research directions, careers, and even whole organisations can be sustained for years by persuading grantmakers and fellow researchers of a thesis, rather than proving value to people who would actually use the work. Without this corrective pressure, even well-intentioned research may drift from what the field actually needs. Commercialisation should not be seen as a distraction or a response to limited funding, but as a tool for staying at the bleeding edge of what is useful for the world.

Productification

Turning research into a product people can buy is extremely powerful for distribution. You are no longer hoping that executives, engineers, and politicians see value in work they do not understand tackling risks they may not believe in. It becomes a purchase. A budget decision. A risk-reward tradeoff that large organisations are very well suited to engage with.

There are clear gaps in securing AI infrastructure which can be filled today. If you’re wondering what an AI safety startup might actually do, here are some suggestions for commercial interventions targeting different parts of the stack.

• Frontier Models: Interpretability tooling, evaluations infrastructure, and formal verification environments. Tools which might be implemented by labs and companies with direct access to frontier models to understand and control them better.

• Applications: Content screening, red-teaming as a service, and monitoring for misuse. Helping startups building on frontier models catch accidental or deliberate misuse of their platforms.

• Enterprise Deployments: Observability platforms, run-time guardrails, and hallucination detection. Enterprises and governments using AI to automate critical work should be able to catch issues early and reliably.

• Market Incentives: Model audit and certification, and safety-linked insurance. Creating market incentives which reward safer models when they’re released into the world.

None of these require waiting for frontier labs to solve alignment, or hoping that someone else finds your work and decides to implement it. Instead of writing white papers hoping governments will regulate or frontier labs will dutifully listen, you build safety directly into products that customers come to rely on. One path hopes someone will do the work, whereas the other is the work.

Safety Across The Stack

When you tap your card at a shop to make a purchase, a network of financial institutions plays a role in processing your transaction. The point of sale system reads your card information, sends it on to a payment processor, who forwards the request to the appropriate card network. The issuing bank for your card authorises the transaction, the money is sent to cash clearing systems, and cash settlement is performed often through a central bank settlement system.

There is a sense in which all fraud happens at a bank. They have to release the fraudulent funds, after all. But the declaration that all fraud prevention initiatives should be focused on banks and banks alone comes across as fundamentally confused. Fraud prevention might be easier at other layers, and refusing to take those opportunities simply because it is in principle preventable at some more central stage would not lead to the best allocation of resources.

Similarly, when a user prompts an AI application, they are not simply submitting an instruction directly to a frontier model company. Just as tapping your card does more than instruct your bank, such a message goes through guardrails, model routing, observability layers, and finally frontier model safety measures. Every step of this process is an opportunity for robustness we should not let go to waste.

This becomes even more critical as AI agents begin acting autonomously in the world, doing everything from browsing and transacting to writing and executing sophisticated code. When an agent’s action passes through multiple services before having an effect, every link in that chain is both a potential failure point and an opportunity for a safety check.

Exclusively focusing AI safety interventions on frontier labs would be like securing the entire financial system by regulating only banks. Necessary, but nowhere near the most efficient or robust approach.

Capital, Talent, and Credibility

Successful for-profits are in an inherently better position to acquire resources than non-profits. Their path to funding, talent acquisition, and long-term influence is far stronger than that of their charitable counterparts.

There is an immense amount of venture capital washing around the AI space, estimated at ~$190 billion in 2025. Flapping Airplanes raised $180 million in one round, comparable to what some of the largest AI safety grantmakers deploy annually, raised in a fraction of the time. VC allows you to raise at speed, try many approaches, and pivot more freely than would be possible in academia or when reliant on slower charitable funders.

In AI safety, non-profits are less likely compared to other sectors to be trapped in an economic struggle for survival. However, even in the AI safety ecosystem, philanthropy is much more limited than venture capital and more tightly concentrated among fewer funders. Non-profits are vulnerable, not just to the total capital available, but to the shifting attitudes of the specific grant makers they rely on. VC-backed companies, by contrast, are much more resilient to the ideological priorities of funders. If one loses interest, many others remain available as long as you have a strong business case.

Yes, there is a large amount of philanthropic capital in AI Safety compared with typical non-profits. Safety products can also be difficult to sell. But the question of whether safety-focused products sell well, as they do in other industries, is a hypothesis you can go out and test. If it turns out that they do, there could be an immense amount of capital available which should be used to make our world safer.

For-profits attract talented people not just through hefty pay packages, but also through their institutional prestige and the social capital they confer. You can offer equity to early employees, which is extremely useful for attracting top technical talent and is entirely unavailable to non-profits. Your employees can point to growing valuations, exciting products with sometimes millions or even billions of users, and influential integration of their technology with pride. For many talented and competent people, this is far more gratifying than publishing research reports or ever so slightly nudging at the Overton window.

All of this, increased access to talent, capital, and credibility, makes for-profits far easier to scale. And safety needs to scale. The amount of time we have until transformative AI arrives differs wildly between forecasts, though it seems frighteningly plausible that we have less than a decade to prepare. If we are to scale up the workforce, research capacity, and integration into the economy of safety-focused products, we cannot afford anything other than the fastest approach to building capacity.

Success compounds. Founders, early employees, and investors in a successful for-profit acquire capital, credibility, and influence that they can reinvest in safety, whether by starting new ventures, funding others, or shaping policy. This virtuous cycle is largely unavailable to the non-profit founders, unless they later endow a foundation with, as it happens, money from for-profits.

In addition to tangible resources, a mature ecosystem of advisors and support networks exists to help startups succeed. VC funds, often staffed by ex-founders, provide strategic guidance and industry connections that are crucial for closing sales. There are many talented people who understand what startups offer and actively seek them out. An equivalent ecosystem just doesn’t exist for non-profits.

Shaping The Industry From Within

Being inside an industry is fundamentally different from being adjacent to it.

Embedding an organisation inside AI ecosystems enables both better information gathering and opportunities for intervention. If you can build safe products appropriate to the problems in an industry, you allow companies to easily purchase safety. If companies can purchase safety, then governments can mandate safety. But to get there, it is not enough to make this technology exist; the technology must be something you can buy.

Cloudflare started as a CDN. By becoming technically integrated, they slowly transformed into part of the critical infrastructure of the internet. Now, they make security decisions which shape the entire internet and impact billions of users every day. A safety-focused company embedded in AI infrastructure could do the same.

Will Markets Corrupt Safety?

Market incentives are not purely aligned with safety. The drive to improve capabilities, maximise revenue, and keep research proprietary will harm a profit-seeking organisation’s ability to make AI safer.

However, every institution has its pathologies. The incentives steering research-driven non-profits and academics are not necessarily better.

Pure Research Also Has Misaligned Incentives.

The incentives of safety and capitalism rarely align. The pressure to drive revenue and ship fast pushes towards recklessly cutting corners, and building what your customers demand in the short term rather than investing in long term safety.

However, research organisations have similar harmful incentives driving them away from research which is productive in the long term. The need to seek high-profile conference publications, pleasing grant makers, and building empires. Incentives of research organisations and individual researchers are notoriously misaligned with funders goals’ in academia and industry alike. Pursuing a pure goal with limited feedback signals is extremely difficult as an organisation, regardless of structure.

Ideally, we would have both. For-profits which can use revenue as feedback and learn from market realities, alongside non-profits which can take longer-term bets on work needed for safety. The question is how to build a working ecosystem, not whatever is more purely focused on safety.

Proprietary Knowledge Is Not Always Hoarded.

For-profits have an incentive to keep information hidden to retain a competitive advantage. This could block broader adoption of safety techniques, and restrain researchers from making optimal progress.

Assuming that for-profits add resources and people to the AI safety ecosystem, rather than simply moving employees from non-profits, this is still advantageous. We are not choosing between having this research out in the open or hidden inside organisations. We are choosing between having this research hidden or having it not exist at all. In many sectors, the price of innovation is that incumbents conceal and extract rents from their IP for years.

Despite this, for-profits do have agency over what they choose to publish. Volvo famously gave away the patent to their three-point seatbelt at the cost of their own market share, saving an estimated 1 million lives. Tesla gave away all of their electric vehicle patents to help drive adoption of the technology, with Toyota following suit a few years later. Some of this additional knowledge created by expanding the resources in AI safety may still wind up in public hands.

Markets Force Discovery Of Real Problems.

The constant drive to raise money and make a profit is frequently counter to the best long-term interests of the customer. Investment which should be put into making a product safer today instead goes into sales teams, salaries, and metrics designed to reel in investors. It is true that many startups which begin with a strong safety thesis will drift into pure capabilities work or adjacent markets which show higher short-term growth prospects.

However, many initiatives operating without revenue pressure, such as researchers on grants or philanthropically-funded non-profits, can work for years on the wrong problem. For-profits will be able to see that they are working on the wrong thing, and are driven by the pressure to raise revenue to work on something else.

This is not to say that researchers are doing valueless work simply because they are not receiving revenue in the short term. Plenty of work should be done to secure a prosperous future for humanity which businesses will not currently pay for. Rather, mission drift is often a feature rather than a bug when your initial mission was ill-conceived. The discipline markets provide, forcing you to find problems people will pay to solve, is valuable.

Failure Is A Strong Signal.

The institutional failure modes of non-profits and grant-funded research are mostly benign. The research done is not impactful, and time is wasted. On the other hand, for-profits can truly fail in the sense that they fail to drive revenue and go bankrupt, or they can fail in more spectacular ways where they acquire vast resources which are misallocated. The difference is not that for-profits are inherently more likely to steer from their initial goals.

Uncertainty about impact is common across approaches. Whereas research that goes unadopted fails silently, and advocacy which fails to grab attention disappears without effect, for-profits are granted the opportunity to visibly and transparently fail. The AI safety ecosystem already funds work which fails silently, and is effectively taking larger risks with spending than we realise. Startups aren’t any more likely to fail to achieve their goals; they are in the pleasant position of knowing when they have failed.

Visible failure generates information the ecosystem can learn from. Silent failure vanishes unnoticed.

Your Counterfactual Is Larger Than You Think.

Markets are not efficient. The economy is filled with billion-dollar holes, which are uncovered not only by shifts in the technological and financial landscape but by the tireless work of individuals determined to find them. Just because there is money to be made by providing safety does not mean that it will happen by default without you.

Stripe was founded in 2010. Online payments had existed since the 1990s, and credit-card processing APIs were available for years. Yet it took until 2010 for someone to build a genuinely developer-friendly API, simply because nobody had worked on the problem as hard and as effectively as the Collison brothers.

Despite online messaging being widely available since the 1980s, Slack wasn’t founded until 2013. The focus, grit, and attention of competent people being applied to a problem can solve issues where the technology has existed for decades.

Markets are terrible at pricing in products which don’t exist yet. Innovation can come in the form of technical breakthroughs, superior product design, or a unique go-to-market strategy. In the case of products and services relevant to improving AI safety, there is an immense amount of opportunity which has appeared in a short amount of time. You cannot assume that all necessary gaps will be filled simply because there is money to be made there.

If your timelines are short, then the imperative to build necessary products sooner rather than later grows even greater. Even if a company is inevitably going to be built in a space, ensuring that it is built 6 months sooner could be the difference between safety being on the market and unsafe AI deployment being the norm.

For many, the alternative to founding a safety company is joining a frontier lab. However, most AI deployment happens outside labs in enterprises, government systems, and consumer-facing applications. If you want to impact how AI meets the world, you may have to go outside of the lab to do it. Your marginal impact inside a large organisation is often, counterintuitively, smaller than your marginal impact on the entire world.

Historical Precedents

History is littered with examples of companies using their expertise and market position to ship safety without first waiting around for permission.

Sometimes this means investing significant resources and domain expertise to develop something new.

• Three point seatbelt: Volvo developed the three point seatbelt and gave away the patent. Their combination of in-house technical expertise and industry credibility enabled a safety innovation that transformed the global automotive industry.

• Toyota’s hybrid vehicle patents: Toyota gave away many hybrid vehicle patents in an attempt to accelerate the energy transition.

• Meta’s release of Llama3: At a time when only a small number of organisations had the resources to train LLMs from scratch, Meta open-sourced Llama3 making it available to safety researchers at a time when little else was in public hands.

Or perhaps the technology already exists, and what matters is having the market position to distribute it or the credibility to change an industry’s standards.

• Levi-Strauss supply chain audit: At the peak of their market influence, Levi-Strauss audited their supply chain insisting on certain minimum worker standards to continue dealing with suppliers. They enforced workers rights’ in jurisdictions where mistreatment of employees was either legal or poorly monitored, doing what governments couldn’t or weren’t prepared to do.

• Cloudflare’s Project Galileo: Cloudflare provides security for small, sensitive websites at no cost. This helps journalists and activists operating in repressive countries avoid being knocked off the internet, and is entirely enabled by Cloudflare’s technology.

• WhatsApp end-to-end encryption: The technology existed, and the cryptography research was mature by this point. WhatsApp just built it into their product, delivering privacy protection to billions of users worldwide.

• Security for fingerprint and face recognition: Apple stores face and fingerprint data in a separate chip, making it impossible to steal or legally demand. This did not require regulation; this decision actually led to clashes with the US government. Because of their market position, Apple was able to push this security feature and protect hundreds of millions of users unilaterally.

All of these required a large company’s resources, expertise, credibility, and market integration to create and distribute valuable technology to the world.

Building a for-profit which customers depend on, be it for observability, routing, or safety-tooling, lets you ship safety improvements directly into the ecosystem. When the research exists and the technology is straightforward, a market leader choosing to build it may be the only path to real-world implementation.

It’s Up To You.

For-profits are in a fundamentally strong position to access capital, talent, and information. By selling to other businesses and becoming integrated in AI development, they can not only identify the most pressing issues but directly intervene in them. They build the technological and social environment that makes unsafe products unacceptable and security a commodity to be purchased and relied upon.

Non-profits have done, and will continue to do, critical work in AI safety. But the ecosystem is lopsided. We have researchers and advocates, but not enough builders turning their insights into products that companies buy and depend on. The feedback loops, distribution channels, and ability to rapidly scale that for-profits provide are a necessity if safety is to keep pace with capabilities.

The research exists. The techniques are maturing. Historical precedents show us that companies embedded in an industry can ship safety in ways that outsiders cannot. What’s missing are the people willing to found, join, and build companies that close the gap between safety as a research topic and safety as a market expectation. We cannot assume that markets will bridge this divide on their own in the time we have left. If you have the skills and the conviction, this is a gap you can fill!

If you’re thinking about founding something, joining an early-stage AI safety company, or want to pressure-test an idea - reach out at team@bluedot.org. We’re always happy to talk.

BlueDot’s AGI Strategy Course is also a great starting point - at least 4 startups have come out of it so far, and many participants are working on exciting ideas. Apply here.

Thanks to Ben Norman, Daniel Reti, Maham Saleem, and Aniket Chakravorty for their comments.

Lysander Mawby is a graduate of BlueDot’s first Incubator Week, which he went on to help run in v2 and v3. He is now building an AI safety company and taking part in FR8. Josh Landes is Head of Community and Events at BlueDot and, with Aniket Chakravorty, the initiator of Incubator Week.

The key challenge that surfaced was sourcing talent who can own and lead work independently, and care deeply about AI risk.

These are people with a strong track record, whether through years of professional experience or a few impressive projects, who can hit the ground running.

Here are some examples of what this experience looks like (these are more illustrative than specific!):

Engineering (Lead / IC)

• You have the technical depth to make good architectural calls on complex systems — databases, ML pipelines, large codebases.

• You’ve shipped high-quality code in demanding environments (e.g. big tech, high-velocity startup)

• You know how to get a junior colleague unstuck on a tricky bug, suggest better tooling, or propose a new process when the current one isn’t working.

• (Lead) You’ve led and grown engineering teams, and can set a high bar for your team.

• (IC) You can take vague requirements for a system and turn them into working systems, even in unfamiliar contexts.

Research lead

• You know what it takes to publish high-quality research end-to-end — scoping questions, choosing methodology, organising the work, and communicating findings to diverse stakeholders.

• You’re a recognised expert in your domain, with the publications and collaborations to show for it.

• You ask good questions that open up impactful research directions, and you lean on a strong professional network to get things done.

Communications lead

• You have a track record of producing high-quality content tailored to specific audiences, and your public profile reflects that.

• You’ve managed large project budgets and timelines, making evidence-based decisions to improve outcomes.

• You’ve engaged seriously with AI safety — perhaps even producing content on it already.

Policy lead

• You’ve worked directly with government and understand how to navigate political institutions.

• You have a strong network in policy environments and know how to get things in front of the right people.

• You have a track record of translating complex technical research into clear, actionable policy memos and briefings.

Note: these profiles are composites from the organisations we spoke to and are meant to be illustrative rather than specific.

You don’t need years of AI safety experience but you do need to care

Given that relatively few people can claim 5+ years of direct AI safety experience, orgs are looking for the next best thing — people with a strong track record of excellence on similar projects outside the field.

Mission alignment helps bridge the gap.

Orgs are looking for people who’ve engaged seriously enough with AI risk to articulate why this work matters from their own perspective, and ideally made career choices that reflect that conviction.

These are small (<50), mission-driven orgs with salaries that can’t compete with big tech companies. As a result, they are looking for those whose motivation goes beyond compensation.

How to get started

The biggest barrier for capable talent entering the field is context.

Our AGI strategy course is designed to give you that context. You’ll develop your understanding of the risks AI poses and explore how you might contribute your expertise.

It’s a free, 30-hour course that AI safety orgs view as a strong signal of motivation. It is the first step most take in their AI safety careers.

AI safety needs your expertise to help ensure a positive future with AI. Apply here to get started.

Our Courses

We currently run four courses: the AGI Strategy Course, the Technical AI Safety Course, the (Frontier) AI Governance Course, and the Biosecurity Course.

Using Our Curricula (Permissionless)

Running an independent version of our courses is largely permissionless. You can pick up our curricula and start running sessions with your group right away. This works well for friends, workplace groups, student societies, and other local organizations who want to engage with the material together.

You can share and adapt our course materials including resource lists, exercises, sample answers, and discussion docs, provided that you:

Brand it distinctly: Market the course in a way that makes clear who’s running it. Don’t use our name in a way that could confuse people as to who is offering the course.

Give appropriate credit: When presenting these materials, please credit us as the source as “BlueDot Impact”.

Don’t hold us liable for errors: While we try our best to ensure the accuracy and relevancy of our course materials, they’re provided solely on an “as is” basis, without warranty of any kind.

Note that we don’t own many of the linked-to resources themselves, so you might need permission from their original owners if you want to copy or translate them.

Facilitator resources: We also provide access to our facilitator templates and documents to help you run effective sessions. You can find these linked here.

Tell us how it went: After you’ve run your course, we’d love to hear from you. Email us at team@bluedot.org with how many people completed the course, what challenges you ran into, what worked well, and whether there were any standout participants you think we should know about. This helps us improve our materials and potentially connect promising people with future opportunities.

Running an Official BlueDot Cohort

If you’re interested in running an official BlueDot cohort in your location, reach out to us first at team@bluedot.org so we can discuss whether this makes sense and what it would involve.

Requirements: You must either have completed our facilitator training program, or be someone with deep existing context in the relevant field.

How it works: We would pay you our normal facilitator rate. However, all participants would need to apply through our standard application review process. If not enough applicants from your location clear our bar, the cohort cannot proceed. We’ll discuss expected numbers and logistics with you before you commit to anything.

What accepted participants receive: Access to our Slack community, a certificate upon completion, and additional career acceleration support.

Who this is for: This option is most relevant for local hubs in places with sufficient talent density. We maintain a high bar for our facilitated cohorts, so this works best in locations where you’re confident there’s a strong pool of potential applicants.

Running Courses at Companies or Institutions

If you want to run our courses internally at a company or institution - or if you’d like us to run them for you - please email us at team@bluedot.org to discuss what this may look like. We typically don't do this but may make exceptions for particularly high-impact opportunities.

Example Marketing (Independent Version)

Here’s an example of how you might market an independent version of our course, following the guidelines above:

Calling all students interested in the future of AI safety and alignment!

🤖 The Greendale Community College AI Society is excited to be hosting an AI safety course this semester, based on the popular AGI Strategy Course developed by BlueDot Impact.

We’ll be following their curriculum. You should do the readings and exercises beforehand, and then each week we’ll host a facilitated group discussion adapted for our university setting.

The discussions will run every Wednesday from 6-8pm in Group Study Room F starting on 17 September. To join the seminar series, RSVP at this link.

If you have any other questions, email ai.soc@greendale.edu. Looking forward to some fascinating discussions! 🙂

What’s Not Okay

Examples of things that would not be okay:

• “GCC AI Society is launching a round of the BlueDot Impact AGI Strategy course”

• “We’re running the BlueDot AGI Strategy Course in Greendale, Colorado”

• “Apply to the AGI Strategy Course”, linking to your version of the course

• “Run in collaboration with BlueDot Impact”, unless we’ve explicitly agreed this

• Issuing certificates “for completing the BlueDot AGI Strategy course” (but it’s fine to issue certificates “for completing GCC AI Society’s AI Alignment course, based on the BlueDot Impact curriculum”)

FAQs

Can local groups get copies of the discussion docs?

Yes! These are linked here.

Can local groups use BlueDot’s infrastructure for running courses?

Yes, almost all our software and corresponding documentation is available on GitHub. You can raise issues there if you get stuck. We aren’t currently able to provide hosted versions of our software, or technical support beyond this. Local groups are also welcome to direct users to our course hub to help learners track their own reading completions and exercises.

Can we use your facilitator training program?

Yes! The resources, exercises and session plans for our facilitator training course are available online. Though, we can’t run facilitator training for you.

I’m planning to run a local group / previously participated in a local group version. Could I join the BlueDot facilitated course?

Yes, please apply in the normal way and mention this in your application. Note that this does not guarantee you a place on our course.

What’s the difference between running an independent version and an official BlueDot cohort?

An independent version uses our curricula but is entirely run by you, for your community. An official BlueDot cohort means participants apply through our process, get access to our full resources (Slack, certificates, career support), and you’re compensated as a facilitator. The first is permissionless, the bar for the latter is higher.

If you have any other questions, feel free to contact us at team@bluedot.org.

Dangerous capability evals make sense for safety. We need to know how worried we should be about models causing harm.

But we’re missing benchmarks for specific positive safety things we want AI to do. Capabilities that could actively keep us safe or steer us toward better futures.

What beneficial evals would measure

Imagine if model cards included sections on how well AI systems perform at:

• Safety research: Assisting with interpretability work, alignment research, or red-teaming other AI systems

• Cyber defense: Not just penetration testing, but actually defending networks, detecting intrusions, and patching vulnerabilities

• Pandemic preparedness: Accelerating vaccine development, optimising PPE distribution, or improving disease surveillance

• Information integrity: Helping people find truth in a world full of mis/disinformation: fact-checking claims, detecting coordinated inauthentic behaviour, or surfacing reliable sources on contested topic

Models have graduated from ‘confidently wrong intern’ to ‘occasionally helpful new grad’, at least for some coding tasks. These evals and reinforcement learning (RL) environments could extend these into beneficial capabilities. If we’re heading toward a capabilities explosion, we need to ensure that beneficial capabilities outpace dangerous ones by a wide margin.

From evals to RL environments

Evals and RL environments are two sides of the same coin. An eval asks “can the model do X?” and produces a score. An RL environment asks the same question, but feeds that score back as a training signal.

In practice, this means a well-designed eval is most of the work to building an RL environment. If you can measure whether a model successfully identified the right papers, traced activations correctly, or generated a viable therapeutic drug candidate, you can turn that measurement into a reward signal. The eval’s scoring rubric becomes the RL environment’s reward function. The eval’s test cases become the RL environment’s training distribution.

A word of caution

Not everything that sounds safety-relevant actually is. Be careful that you’re measuring something genuinely useful for safety, not just general capabilities dressed up in safety language.

Getting models to be better at coding in Python is a general capability, not safety-specific. But improving models’ use of TransformerLens for interpretability research unblocks a specific safety-relevant bottleneck.

The goal is to create targets that, if hit, would actually make AI development go better, without inadvertently pushing general capabilities that could be misused.

How to build beneficial capability evals

Approach 1: Reproduce existing safety projects

Take existing safety-relevant projects and try to get AI to reproduce them. Where does the model fail? Those failure points become eval problems you can measure against.

For example, BlueDot Impact runs courses where participants complete projects on AI safety, biosecurity, and other safety-relevant domains. Take a completed project — say, a mechanistic interpretability analysis of a small transformer — and see if a model can replicate it. If the model gets stuck on a specific step (maybe it can’t correctly identify attention head functions), that’s a concrete eval task.

This approach grounds you in real work that humans have already validated as useful.

Approach 2: Talk to safety researchers

Go directly to people doing safety-relevant work. Ask them to keep a log for a couple of weeks: every time AI tools frustrate them, slow them down, or just can’t do something they need, write it down. Then collect these logs and look for patterns.

You might find things like: “Claude keeps hallucinating paper citations when I ask for related work,” or “I can’t get any model to correctly trace activations through residual stream connections,” or “It takes 10 back-and-forths to get the model to understand my codebase structure.”

These frustrations point directly to capability gaps. Turn them into evals.

Breaking down the work

Whatever approach you take, you’ll need to:

Map the process. Break down the safety-relevant task into all its component steps. For example, with vaccine development: literature review → identifying which parts the immune system can target → choosing an approach (mRNA, viral vector, etc.) → designing candidates → scaling up manufacturing → running clinical trials.

Prioritise. Figure out which parts of the pipeline are most important to work on first. Maybe running clinical trials is the biggest bottleneck or the literature review step that takes researchers months.

Define tasks. For each component, specify what success looks like in a way that’s measurable. For the literature review step, this might be: “Given a novel pathogen genome sequence, identify the 10 most relevant prior papers on similar pathogens, ranked by relevance to vaccine development”. Include a rubric for scoring relevance and completeness.

Build evaluation infrastructure. Many modern evals are “agent-evaluated”—you get one model to attempt a task, then use another model as a judge to verify whether specific conditions have been met. Inspect is a good framework for getting started with building evals. Consider submitting your work to inspect-evals to make it discoverable.

Iterating and improving. A key is desirable difficulty: we need evaluations that remain challenging and informative as capabilities improve. If in doubt, err on the side of too difficult.

Getting Your Work Used

Building good evals isn’t enough. You need companies to actually find and use them.

• Make it discoverable: Publish on your blog, get it circulating on Twitter/X, or submit to relevant publications

• Make it easy to run: Companies are more likely to adopt your eval if they can spin it up quickly

• Work directly with companies: Offer to help them run it, troubleshoot issues, and interpret results

The business opportunity

Building good evals is a serious undertaking. No single person or team is going to create comprehensive positive capability benchmarks across all safety-relevant domains.

But there’s a business opportunity here. Companies will pay for RL environments: the Docker containers, code infrastructure, and carefully written rubrics for each task. A path here might be:

• Start with a small batch or proof of concept

• Pitch it to a company

• If they’re interested, they might pay for a set of ~100 problems

• They’ll QA the work, and if it’s good, they’ll ask for more

Two people working full-time could potentially (with a lot of coffee) get an initial batch done in a month.

Start building

Pick a safety-relevant domain you know, map the steps, find the bottlenecks, and build something. You can also:

• Apply to our incubator weeks.

• Use or apply to the Technical AI Safety Project sprint

• Learn more about plans to make AI go well on the AGI strategy course.

• Tag @BlueDotImpact on Twitter or LinkedIn. We’d love to see what you build!

But luckily there are a few common prefixes and suffixes that crop up again and again. Once you’ve learned this lingo, you’ll have a far easier time fighting through biology papers and wikipedia articles.

Cell structure and location

• Cyto- → cell (cytoplasm, cytoskeleton, cytokine)

• -some → discrete body/structure (ribosome, lysosome, chromosome, endosome)

• Endo- → within/inside (endocytosis, endoplasmic reticulum, endotoxin)

• Exo- → outside/outward (exocytosis, exotoxin, exonuclease)

• Peri- → around (periplasm, perinuclear)

• Trans- → across (transmembrane, transcription, translation)

• Intra- → within (intracellular, intranuclear)

• Extra- → outside (extracellular, extrachromosomal)

Enzymes and processes

• -ase → enzyme (protease, kinase, polymerase, lipase)

• -sis → process/action (mitosis, apoptosis, lysis, synthesis)

• -lysis → breaking down/splitting (hydrolysis, glycolysis, proteolysis)

• -genesis → creation/origin (biogenesis, pathogenesis, oncogenesis)

• Kinase → adds phosphate groups (protein kinase, tyrosine kinase)

• Phosphatase → removes phosphate groups

• -merase → polymerising enzyme (polymerase, telomerase)

• Enzyme names usually end in -ase with the substrate as prefix (lactase acts on lactose, protease on proteins)

Eating, destroying, killing

• Phago-/-phage → eat/engulf (phagocyte, macrophage, phagocytosis)

• Bacteriophage/phage → virus that infects bacteria (you’ll see “phage” used standalone constantly in molecular biology)

• -cidal/-cide → killing (bactericidal, virucidal, fungicidal)

• -static → inhibiting growth without killing (bacteriostatic)

• -lytic → causing lysis/destruction (cytolytic, haemolytic)

Molecules and macromolecules

• Glyco-/-glycan → sugar/carbohydrate (glycoprotein, glycolysis, proteoglycan)

• Lipo-/Lip- → fat/lipid (lipoprotein, lipase, lipid bilayer)

• Proteo-/Prot- → protein (proteome, protease, proteolysis)

• Nucleo- → nucleus or nucleic acid (nucleotide, nucleosome, endonuclease)

• Poly- → many (polymer, polypeptide, polysaccharide)

• Oligo- → few (oligonucleotide, oligosaccharide)

• Mono- → one (monomer, monosaccharide)

• -ome1 → complete set (genome = complete set of genes, proteome = complete set of proteins)

Useful general prefixes

• Bio- → life/living (biosynthesis, biomarker)

• Hetero- → different (heterozygous, heterogeneous)

• Homo- → same (homozygous, homologous)

• Iso- → equal/same (isomer, isotonic)

• Hyper- → excess/above (hyperactivation, hypertonic)

• Hypo- → deficient/below (hypotonic, hypoxia)

• Anti- → against (antibody, antigen, antibiotic)

• Pro- → before/precursor (promoter, prophage, proinflammatory)

Test yourself!

Go through the abstract from the start of this post word-by-word and try to rewrite it in simple language with the help of the terminology above. You might not get everything, but I bet you’ll get further than you expect.

Use an LLM to check your work and help you with any parts you’re still confused about.

The need for field-level strategies

A field is a community of individuals and organisations working to address a shared problem. In our case, the fields we operate in are trying to mitigate risks from AI and synthetic biology.

These fields are under-resourced and confused. Without a strategy for what we’re trying to achieve and how we’re going to achieve it, we risk producing elegant answers to pointless questions, focusing our scarce resources on unimportant problems, and failing to protect humanity from harm.

It’s nobody’s job to produce field-level strategy — most actors in the community focus on their own organization or their own role.

Many “strategies” that people do produce are shallow, one-off wishlists that don’t get distributed to relevant stakeholders. They set goals before deeply understanding the problem and analysing what matters most for whether or not we’ll succeed. Strategy isn’t a goal, it’s a form of problem-solving, and you can’t solve a problem you don’t understand and haven’t defined.

Field strategy is still challenging to do well. Defining the “problem-to-be-solved” is contentious and a moving target. Information is privately held or not written down. Experts disagree because they have different values and world-models. You might just aggregate people’s opinions and produce a vague mush. Few people have done this well before. You’ve not been appointed to do this, so you need to earn legitimacy and trust by doing the hard work well.

Here’s how you can overcome these challenges.

How to produce a field strategy

These steps should be done in parallel.

Read and take notes

You’re not the first person to think about this problem. Start by reading the field’s seminal literature, and important pieces from adjacent fields. You can use AI to search for the best papers and blog posts, and they’re also great thought partners to ask you questions, give you feedback, and prod your mental models for how things work.

Take lots of scrappy notes. Don’t write prose; stick to one-sentence bullet points. Synthesise what you’re learning into the most critical insights, and group your notes into sensible-seeming categories to help you build a stronger mental model of the field. Keep track of your confusions and uncertainties, as these can steer further research and interview questions.

Ask and listen

You’re unlikely to find the most important insights on the public web. They live inside people’s heads or in private google docs. To acquire this information, you need to talk to lots of the right people. Connect with them via cold emails or via warm introductions. As a heuristic, try to have 50 calls in your first 50 days of working full-time on this.

The usefulness of conversations tends to follow a power-law: most conversations will be fine, a few will be extraordinary, some you won’t realize were useful until weeks later. So you need to constantly search for the most insightful people.

To gather the most juicy insights during your calls, you need to build a connection quickly. At the start of every conversation, share your backstory, ask them for theirs, and find common ground. Demonstrate you’re competent, be vulnerable, and mirror their energy.

Avoid biasing them with your ideas early on. Ask open questions that give them a lot of space to explore and share things you don’t expect. For example, “If we’ve solved X problem in 10 years, what’s the most important thing that needs to happen next year?”. Read The Mom Test for question-asking advice.

After you’ve asked a broad open question, listen intently and ask a follow-up question that zooms in on a specific thing they said that seems important. You know it’s a good question if their eyes light up or if they have to think super hard and then say something really valuable.

You should also come prepared with sharp, open questions that enable them to build on what you already know. For example, “In X paper, I noticed Y, but this conflicts with my intuition Z. What’s your intuition about Z, and what drives that?”.

If you’re confused, express that confusion and give them the opportunity to help you. If you hear them say something that doesn’t make sense to you or doesn’t align with your model for how things work, chase it. Ask follow-up questions. This is how you discover unknown unknowns.

Move fast through the network

Every call is an opportunity to bounce into the next conversation. In the final 5 minutes, prioritize generating names for potential introductions. If you’ve asked great questions, demonstrated that you’re determined to solve this problem, and they’ve had fun, they’ll feel excited to introduce you to people they admire.

To help them generate names, prompt them with:

• Who’s done the best writing on this problem?

• Whose work are you most impressed with?

• Who do you turn to for help when you’re stuck?

• Who would I learn the most from?

Explicitly ask for introductions before the call ends. Then you need to make it as easy as possible for them to make those introductions. Immediately after the call ends, send them a complete draft email they can use to make the introductions. This should include a subject, a bio of yourself and a description of this project. Aim for <100 words.

Once you’ve been introduced, you need to move fast. If you send them a scheduling link, it might take them 1-2 weeks to click on it, and they might schedule a meeting for 1-2 weeks after that. But you need to move as deep into the “intros chain” as quickly as possible.

Immediately after you’ve been introduced to someone, send them a calendar invite for the next day at a reasonable time for their timezone. Use this format for the calendar event title: “[TBC] YourName<>TheirName”. Reply to the introduction asking if that time works, and if not, when works best for them. This tactic runs a higher risk of annoying people and shouldn’t be used with everyone, but I think it’s worth the speed benefits most of the time. Use it thoughtfully.

Write and rewrite

After your first few calls, start writing your v1 scrappy strategy doc. In separate sections, answer these questions:

• What is the precise problem/threat you’re trying to solve?

• If this field succeeds in 5-10 years, what’s different about the world?

• Where are things at today? Who’s working on this, what’s been tried, what’s the funding and policy landscape?

• What’s the biggest obstacle between here and success?

• What approach could overcome this obstacle, and why would it work?

• What needs to happen in the next 6-12 months?

Here are some work-in-progress examples: UV air disinfection, DNA synthesis screening, pandemic early warning systems.

This structure is adapted from Richard Rumelt’s “Good Strategy, Bad Strategy,” which is well worth the read. The most common failure mode when doing strategy is jumping from “what success looks like” to “what should we do”, without diagnosing why we’re not already succeeding.

Remember that your first call with an expert is only the beginning of your relationship with them. Give them commenting access on your google doc, thank them for their input, and ask for their feedback. You know this is working when people spend a lot of time in your doc, when experts debate each other in the comments and generate new insights, and when people ask you if they can share the doc with their peers.

This isn’t a one-off, linear process. It’s messy, and you’ll jump around a lot. You’ll discover big new questions and conflicting evidence that undermine your confidence and makes you feel confused. You’ll get lost down rabbit holes. And you should be prepared to rewrite everything from scratch a few times.

But after a few weeks or months, you’ll have a clear sense for where this field needs to be, a sharp diagnosis for the biggest obstacle blocking progress, a plan of action for overcoming that obstacle, and widespread buy-in from the most powerful and influential stakeholders in the field.

Taking action

This process doesn’t just produce a document. It gives you strong relationships with influential players throughout the field. It gives you situational awareness about what everyone’s doing and why, and what’s blocking them. It makes you a person people turn to when they want to know what’s happening, what matters most, and what to do next.

Then the next phase begins: obsessively communicating the strategy, overseeing implementation, removing obstacles to progress, and updating and refining the strategy as the field evolves and we learn from reality. More details on this in a future blog post.

If you’re working on an important global problem and you don’t know what to do (and seemingly neither does anyone else), this is how you can make yourself useful to a field that needs direction.

Further reading

These blog posts will help you understand what we mean by strategy. I believe it’s worth spending 5+ hours engaging with these.

• Strategy as Problem-Solving – US Army War College

• Good Strategy, Bad Strategy – Richard Rumelt

• What makes a strategy great – Jason Cohen

• Strategic Command – David Petraeus

• Field Building for Population-Level Change – Bridgespan Group

• The Case for a Partnership Between Field Strategists And Philanthropists – Tom Kalil

The long answer depends on what you’re trying to do. Are you trying to figure out how you might contribute your technical skills to AI safety, or get opportunities to do it?

This blog post is written for graduates of the Technical AI Safety course who have spent 30 hours learning about current safety techniques, and the gaps to building safer AI.

What counts as ‘can code’?

I don’t mean you have to have professional software engineering experience. I mean that you at least feel comfortable independently completing a simple coding project.

This means things you should be able to:

• Read and review code

• Debug errors and figure things out when you get stuck

• Write basic functions and loops

If that’s not you yet and you want to do a research / engineering project, you’ll want to build those skills first.

You should probably do a project if…

You’re applying for research and engineering roles, but not getting the roles you want. We’ve spoken to hiring managers from a variety of AI safety organisations including government, non-profits and frontier AI companies, who all tell us the same thing. An excellent project is one of the strongest hiring signals.

Most AI safety orgs are not credentialist, so having a project also puts you on more even footing, even if you don’t have years of professional software engineering or research experience. This is also the reason why there are so many AI safety research fellowships like ARENA, MATS, Pivotal, LASR, PIBBS.

Moreover, doing a project will give you some of the skills for technical interviews when applying.

You’re still figuring out how you want to contribute. While you could (and should!) get others’ takes on what technical work is like, just trying the thing can give you way more information than just reading or talking to people. It can be a cheap test for you to see for yourself what doing this work is like.

Before you devote time to upskilling, see for yourself what that work entails. You’ll surprise yourself by how much you can just jump in and learn as you go, rather than upskill generally.

You’re earlier in building your technical skills. Doing the real thing is one of the most effective ways to learn because it forces you to focus on the actual skills you need. Even if you don’t land a role immediately, you’ll be building your portfolio along the way.

You might not need this if…

You already have a strong portfolio. When I speak to hiring managers, and AI safety researchers and engineers, they recommend just applying because people often underestimate how experienced they are or overestimate how experienced they need to be for the role.

They’ve also said that you’re not penalised for reapplying. Just make sure to highlight what’s changed since your last application. In fact, this is often a strong signal of how high agency you’ve been in upskilling since then.

If you already have a strong portfolio or technical skills, apply first. Do a project later.

This probably isn’t for you if…

You’re doing this because you think it’s the only way to contribute. It’s not.

Especially if you’re several years into your career, you’ve racked up expertise that others are spending months trying to acquire. Leverage that!

Look for areas where you can contribute your unique skills and experience. You can read 80,000 Hours career review or check out our AGI Strategy course to learn about other pathways. There are many high-impact paths that don’t require touching code.

So you’ve decided to do a project

You can start right now by following our Technical AI Safety Project sprint.

You can get a richer experience by applying, where we’ll provide:

• Mentorship: Having someone familiar with AI safety can guide your project direction with how to pick and scope your project, what tools to use, and who to talk to.

• Accountability: Setting goals is hard, sticking to them is even harder. Working on this with someone else will be a major boost!

• Rapid feedback: A point person to review your work as you go, so you can iterate faster.

Whether you follow the guide independently or apply to join a cohort, we’re excited to see what you come up with! Tag @BlueDotImpact on LinkedIn or Twitter with your project.

I’d thought about running an event like this before, but felt daunted. It seemed like a huge lift. I wasn’t even sure women would want to attend.

When I went to one that Monika Jotautaitė hosted, I realised that (1) there’s a LOT of demand for this event and (2) it’s pretty straightforward to organise.

Here’s how you can plan this event in less than 2 hours.

This blog post is written for graduates of BlueDot’s courses.

What you’ll create

When I look around, I don’t see many women in AI safety. I’m not sure why, but what I do know is that these events help create:

• A space where women feel comfortable showing up

• Connections with people navigating similar paths

• Role models whose journeys they can learn from

My hope is that by concentrating women in one place, we make it easier to show up, stay in the field, and maybe even bring friends along.

Two event formats

Here are two that worked well and are simple to organise:

Speed friending

Have people pair up for 7-minute conversations, give people a minute to exchange contacts at the end, then swap over to a new pair. After 6-8 rounds, have an open social.

I recommend placing this timer where everyone can see it.

Talk + socials

Invite someone to give a 20-minute talk with a 20-minute Q&A about something that other women in AI safety might find useful.

The talk encourages attendance and gives people something to discuss during the social.

Who to invite

A rule of thumb is to organise an event that YOU would be excited to attend. That includes deciding who it’s for.

“Women in AI safety” is still a broad category. Maybe you want women already working in the field. Maybe you want women curious about entering. Maybe you want a mix. All are valid — just be intentional.

It can feel painful to tell people no, but thoughtful, considered exclusion is vital to any gathering. (Take it from Priya Parker, an expert in event planning!)

4 steps to bring your event to life

1. Pick a date and time: I find that weekday evenings (6/7pm) tend to work well.

2. Find a space (or go online): If you don’t have access to an events space, reach out to other AI safety orgs to see if you can use their office space. Otherwise, you can run it online using our Zoom account.

3. Submit our events form: We’ll get you set up on our events calendar, which handles invites, feedback forms, reminder emails and guest list. We can also provide a budget for snacks.

4. Start sending out invites: You’ll surprise yourself by how many people are excited by this type of event. Having 10-20 people show up is plenty for great conversations.

It’s that simple

We’d love to see more women in AI safety, and I believe that events like these can help make AI safety feel more approachable. This isn’t the only thing we can do to encourage diversity in the field, but it’s one way to start.

We’re excited to see your event come to life! Fill out the form and let’s make it happen.

• Scale AI safety research

• Build tools for AI safety researchers

• Contribute directly to AI safety research

This guide walks through a project you can complete in <1 week to make your first contribution.

This blog post was written for graduates of BlueDot’s Technical AI Safety course who want to contribute their software engineering skills.

Why do a project?

Projects help you figure out where to apply your skills. Does your experience programming GPUs apply to training infrastructure? Does your agent scaffolding experience translate to evals? You’ll learn more from trying than from months of deliberation.

If you find a gap you can fill, it could lead you to orgs doing that work — or inspire you to start something yourself.

When applying to AI safety roles, this project could also serve as a helpful signal for your skills and motivation. A well-executed project can help demonstrate clear reasoning, good communication, and high agency. I’ve spoken to several hiring managers who made offers or fast-tracked candidates because of excellent projects.

While you could complete a project through programs and fellowships which provide more structure, mentorship and stipends, you could also do it yourself!

You already have what it takes to start. Here’s how.

Getting started

• Block out 20-40 hours in your calendar.

  • If you want to keep going after that, schedule more time later. Right now, focus on finishing something by the end of the week.

• Schedule focused blocks.

  • Aim for 2-4 hour sessions where you can get “stuck in”. The more fragmented your time, the more time you’ll burn context switching.

  • Rope someone in to work on this with you or be accountable to.

• Protect this time.

  • It’s easy to let social events or work meetings eat up your project hours. If this matters to you, treat these blocks like any other important commitment.

  • Make a calendar event for yourself!

• Build a routine.

  • Work at the same time each day if possible.

  • Set a clear intention and put it somewhere you can see it. E.g. “Every day for the next 5 days, I’ll spend 4 hours working on my project at my desk.”

Choose your path

Option 1: Fix open issues in AI safety tools

Contribute to tools that AI safety researchers use every day.

Pick 2-5 good first issues to solve from an open source AI safety repo, like:

• Inspect Evals

• ControlArena

• Inspect

• TransformerLens

• Neuronpedia

• nnsight

• safety-tooling

• Other volunteer projects

• (email me if you know of more! anglilian@bluedot.org)

You can also message the maintainers, join their Discord / Slack communities or just try out the tools to figure out what needs improving.

For example, Anthony Duong looked at the issues on TransformerLens and spent a few weekends making PRs.

Option 2: Replicate and extend a research finding

Get closer to research by reproducing and extending a published result.

The goal is to reproduce and add a small tweak to ONE interesting finding from a paper.

Some ideas for picking a starting point:

• Reviewing the resources from the Technical AI safety course

• Get inspired by past projects from BlueDot or ARENA alumni

• Replicate and extend Anthropic’s alignment faking demo

• Pick an open problem in evals

• Pick an open problem in mech interp

If you want to spend closer to 20 hours on the project, pick papers that have code (and datasets if applicable) available for you to run. Otherwise, expect to spend a lot more time working out how to implement the code.

Replicating a finding from scratch is more feasible for papers like evals or elicitation techniques that don’t require deep ML expertise.

Don’t get too bogged down with trying to make a novel research contribution. It takes months to develop good research taste. Instead, follow where your curiosity leads you.

Then, find the fastest way to get signal on whether this is an idea worth pursuing before running a high volume of tests. Can you prompt the model and see what happens? Can you use a fine-tuning API like TogetherAI or OpenAI?

Remember, this is meant to be a short project. You can always build on this in your next iteration.

You can get compute for your project from providers like RunPod or Hyperbolic, and access open source models via OpenRouter. If funding becomes a constraint, you can apply for a small grant as a BlueDot course graduate.

Ethan Perez has written useful tips for empirical research here.

Option 3: Make research reproducible

Unblock other researchers by fixing what’s broken in the replication process

Many published papers are hard to replicate because the code is buggy, dependencies are missing, or the workflow is unnecessarily painful.

Your goal is to pick a paper with available code, try to run it, and fix whatever breaks or makes it painful to work with.

This could mean:

• Fixing broken code or missing dependencies

• Writing clearer setup instructions or documentation

• Building micro-tooling for repetitive, manual steps (e.g., a script for batch querying LLMs, a config manager for hyperparameters, or a notebook that visualises results)

• Packaging the reproduction in a way that “just works” for the next person

Focus on making it easy for everyone who comes after you to build on it. By the end, you should have opened a PR or created an issue that ideally gets merged!

Write it up!

The biggest mistake people make is treating the write-up as an afterthought. Your work won’t advance AI safety if no one engages with it.

Plan to spend at least one full day writing this up. If you’re spending less, you’re not spending enough.

This includes the different forms of write-ups, like your longer-form blog and the Twitter/X or LinkedIn post that helps with distribution.

As a rule of thumb, you should allocate your writing time proportional to how much total reading time each format will receive. Writing a viral thread deserves as much effort as writing a detailed post.

Why write it up

Your write-up is how you get feedback and find a home for your work. It’s also how others gauge how well you know your stuff and how much you’ve thought it through.

Many of our past graduates have found their co-founders, collaborators, roles and funding opportunities from posting their projects.

You might be thinking: “I can write it up much faster than that” or “I’d rather spend time working on the project”. But if you want your work to reach people, it’s worth communicating well.

Writing clearly demonstrates your understanding. You can’t write clearly about something you don’t fully get.

Think about explaining your own codebase to a new hire. If you’re fumbling through the explanation, you probably don’t understand the architecture as well as you thought. Clear communication is an indicator of understanding, and it takes work to achieve.

How to write it well

Lead with what you did. Don’t bury your insights in walls of text or unnecessary jargon. State it clearly upfront.

Explain why it matters. Motivate why you did this and why anyone should care. People are more likely to read your work (and remember it) when they understand the why.

Keep it simple. The goal is for people to actually read and understand what you’ve done. This is not easy. Writing something short and clear is much harder than rambling on. But that’s exactly why you need to devote real time to it.

Get feedback constantly. Good writing requires iteration. Explain your project to others as you work on it. See if they understand. See if they’re convinced it’s compelling. If not, workshop your idea or delivery.

Don’t wait until the eleventh hour. Many AI safety researchers like Neel Nanda have highlighted how important this is. Start early and keep refining as you go.

Share your work

I know it feels scary to stake your name publicly on what you’ve done. But here’s the thing: your work is far more likely to get drowned out in the noise than criticised. You have to put in a LOT of work to be seen (there’s a whole industry around marketing!). And being seen is exactly what you want.

If you’re working on research, much of the community is on Twitter/X, so focus on making a thread and posting on LessWrong and the Alignment Forum.

Star this project on your GitHub, feature it on your blog (make one if you don’t have one!), post it on LinkedIn and keep talking to people about it.

So what are you waiting for? Let’s get started!

PS: Use/apply to our Technical AI Safety Project sprint.

Let us know how you’re using our courses here, so we can potentially offer more support in the future.

We’re thrilled you’re excited to run an independent version of our course! This page sets out guidance for branding your course, as well as explains what support we can offer local groups.

It’s great when friends, workplace groups, student societies and other local organizations run versions of our courses. This helps further our mission of accelerating driven individuals to develop the knowledge, skills and connections needed to have a significant positive impact.

To support this, you can share and adapt our course materials including resource lists, exercises, sample answers and discussion docs, provided that you:

• Brand it distinctly: Market the course in a way that it’s clear who’s running the course. Don’t use our names like “AI Safety Fundamentals” or “BlueDot Impact” in a way that could confuse people as to who is offering the course.

• Give appropriate credit: When presenting these materials, please give credit to the source, for example “AI Safety Fundamentals” or “BlueDot Impact”.

• Don’t hold us liable for errors: While we try our best to ensure the accuracy and relevancy of our course materials, they’re provided solely on an “as is” basis, without warranty of any kind.

Note that we don’t own many of linked-to resources themselves, so you might need permission from their original owners if you want to copy or translate them.

Example

Here’s an example of how you might market your course, following the guidelines above:

Calling all students interested in the future of AI safety and alignment!

🤖 The Greendale Community College AI Society is excited to be hosting a AI alignment course this semester, based on the popular AI Safety Fundamentals courses developed by BlueDot Impact.

We’ll be following their curriculum. You should do the readings and exercises beforehand, and then each week we’ll host a facilitated group discussion adapted for our university setting.

The discussions will run every Wednesday from 6-8pm in Group Study Room F starting on 17 September. To join the seminar series, RSVP at this link.

If you have any other questions, email ai.soc@greendale.edu. Looking forward to some fascinating discussions! 🙂

Examples of things that would not be okay:

• “GCC AI Society is launching a round of the AI Safety Fundamentals course”.

• “We’re running the AI Safety Fundamentals course in Greendale, Colorado”.

• “Apply to AI Safety Fundamentals”, linking to your version of the course.

• “Run in collaboration with BlueDot Impact”, unless we’ve explicitly agreed this.

• Issuing certificates “for completing the AI Safety Fundamentals course”. But it’s fine to issue certificates “for completing GCC AI Society’s AI Alignment course, based on the AI Safety Fundamentals curriculum”.

This similarly applies to the names AISF, AGISF, AGI Safety Fundamentals, Biosecurity Fundamentals. We’re fine with people using the generic course names: AI alignment, AI governance, and pandemics.

FAQs

Can local groups get copies of the discussion docs?

Yes! These are linked on the individual course webpages.

Can local groups use BlueDot’s infrastructure for running courses?

Yes, almost all our software and corresponding documentation is available on GitHub. You can raise issues there if you get stuck. We aren’t currently able to provide hosted versions of our software, or technical support beyond this. Local groups are also welcome to direct users to our course hub to help learners track their own reading completions and exercises.

Can we use your facilitator training program?

Yes! The resources and exercises for our facilitator training course are available online for self-study.

I’m planning to run a local group / previously participated in a local group run version. Could I join the BlueDot facilitated course?

Yes, please apply in the normal way and mention this in your application. Note that this does not guarantee you a place on our course.

If you have any other questions, feel free to contact us!

Examples include:

• Building benchmarks to test whether AI models can assist with dangerous biology. These can help AI developers identify and constrain dangerous capabilities before deployment, or inform policymakers about emerging risks. But the benchmark code or collated data could also help malicious actors test which models are most useful for their purposes, or provide a roadmap for extracting harmful outputs.

• Developing AI agents with access to biological tools. Such agents could accelerate defensive research, drug discovery, or outbreak response. But the same capabilities could be repurposed to help bad actors navigate complex biological procedures or identify vulnerabilities in biosecurity infrastructure.

• Red-teaming exercises that probe biosecurity vulnerabilities. These can reveal weaknesses in time to fix them, and build the case for policy action. But detailed findings about what’s broken, if shared too broadly, hand attackers a list of exploitable gaps.

• Threat modelling that produces detailed attack scenarios. This can help defenders prioritise resources and anticipate adversary behaviour. But concrete scenarios, if they escape controlled settings, can serve as instruction manuals.

Whether any particular project does more good than harm depends heavily on context: who’s doing it, where, with what safeguards, and how the outputs are managed. There’s genuine uncertainty here, and reasonable people disagree. The point isn’t that these projects should never happen, but that they require more care than work without dual-use dimensions.

Dual-use is a spectrum, not a binary

Not all dual-use work is equal:

• Some should be done openly. Low-risk contributions to detection, broad-spectrum countermeasures, or policy analysis.

• Some should be done carefully. In controlled settings with experienced collaborators, perhaps with government stakeholder input.

• Some requires serious information security. Maybe even classification, with appropriate institutional backing.

It’s important to remember that your good intentions don’t eliminate the risk. Someone building a benchmark to measure misuse-relevant AI capabilities, even with the goal of demonstrating those capabilities should be restricted, is producing something that could itself be useful to bad actors.

What to do

If you’re considering a project with dual-use dimensions:

1. Get experienced mentors. This is the single most important step. Senior people in biosecurity have developed intuitions about what’s safe and what isn’t. They’ve seen projects go wrong. They have relationships with funders and policymakers who can help navigate tricky situations.

2. Don’t do it alone. The “unilateralist’s curse” applies here: if twenty-four people independently decide a risky project isn’t worth pursuing, but the twenty-fifth goes ahead anyway, the damage is done. Check your reasoning with others before acting.

3. Consider the institutional setting. Some work genuinely belongs in organisations with government relationships, security clearances, and established protocols for handling sensitive material.

4. Think about outputs. Before you create a dataset, tool, or benchmark, ask: if this were publicly released, who would benefit? If the answer includes potential bad actors, you need a plan for controlled access, or you need to reconsider whether this project should exist at all.

And remember: defensive value doesn’t happen automatically. If you’re building something intended to help biosecurity professionals, you need to actively get it into their hands. Don’t assume the right people will find your tool or analysis and use it. Test whether it’s actually useful to them before you start, tailor it to their expressed needs, and build relationships that ensure it reaches them. A dual-use tool that only bad actors end up using is worse than no tool at all.

When in doubt

If you’re unsure whether a project idea has dual-use concerns, that uncertainty is itself informative. Reach out to your course facilitators, or to experienced biosecurity professionals who can help you navigate these questions.

The biosecurity field needs capable, ambitious people. But capability and ambition should be paired with caution and collaboration. The goal isn’t to discourage you from working on hard problems; it’s to ensure that when you do, the work makes things better rather than worse.

Here are three broad ways how:

1. Scale AI safety research

2. Build tools for AI safety researchers

3. Contribute directly to AI safety research

This blog post was written for graduates of BlueDot’s Technical AI Safety course who want to contribute their software engineering skills.

Scale AI safety research

This means taking promising safety experiments that work on 1,000 examples and running them efficiently on millions, or moving from single GPUs to distributed clusters.

This path allows you to leverage your engineering expertise in scaling code without deep ML expertise.

Some examples of what this might look like:

• Optimising compute utilization: Turning single-GPU experiments into distributed training runs, or improving GPU utilisation from 30% to 90% through better batching and memory management.

• Scaling research code: Refactoring Jupyter notebooks that take days to run into production pipelines that finish in hours, or creating frameworks that handle datasets too large to fit in memory.

• Making evaluations reliable: Debugging why evaluation runs partially fail overnight, handling API timeouts gracefully, and implementing automatic retries for failed samples.

• Building observability: Creating dashboards that show which evaluation samples are running, which failed, and why – bringing established SWE practices like distributed tracing to ML workflows.

It’s generally useful to understand the basics of ML, but in most cases, good questions to researchers will get you the context you need. You don’t have to know how to design a loss function or interpret attention patterns. Focus on your value-add – your engineering skills!

Building tools for AI safety researchers

This means creating the tools that multiply researcher productivity.

AI safety researchers spend significant time on infrastructure: running evaluations, analysing model internals, managing compute, and reproducing experiments. Some common examples of research workflows and tools include:

Evaluation frameworks

Researchers run the same evaluations repeatedly to track how models perform over time, and when a paper publishes results, others want to reproduce them or test them on newer models.

This requires three things:

• Infrastructure layer: Managing resource allocation of servers and compute, where large-scale evaluations run.

  • For example: using Kubernetes to manage compute, setting up cloud infrastructure like AWS, GCP or Azure and using job schedulers for running millions of evals.

• Orchestration layer: Building the framework for running evaluations.

  • Tools like Inspect provide the building blocks for prompting models and analysing their responses systematically.

  • Inference optimization tools like vLLM and SGLang handle efficient model serving with batching and memory management at scale.

  • Model serving platforms like Ollama make it easy to run models locally or self-host them for evaluation workflows.

  • API rate limiting infrastructure that centrally manages rate limits across model API providers (OpenAI, Anthropic, etc.).

• Evaluation layer: Making specific evaluations reproducible and portable across models.

  • Libraries like inspect-evals provide ready-to-run evaluations for MMLU or other benchmarks. Instead of manually implementing them each time, researchers can run them with a single line of code.

  • ControlArena uses Inspect’s framework to evaluate different control protocols.

  • Analysis tools like Docent or InspectScout help researchers interpret evaluation results without custom analysis scripts.

  • Visualisation tools like Inspect Viz or dashboards help communicate these results more broadly.

Mechanistic interpretability tools

Understanding what happens inside models requires examining activations at each layer, testing changes, and tracking how information flows through the network.

• Experimentation tools: Tools like TransformerLens let researchers probe models without building infrastructure from scratch. They can use standard functions to run experiments in minutes instead of writing custom code that takes hours.

• Model access: Large models often don’t fit on a single GPU or require significant compute resources. Tools like NNsight provide API access to models, so researchers can run experiments without self-hosting.

Running open-source models

Testing open-source models involves practical challenges like:

• Debugging: Getting models to run as expected, handling API quirks, and troubleshooting configuration issues

• Reproducibility: Setting up chat templates, parameters, and output formats to match published results

• Self-hosting: Configuring models for local deployment to reduce costs or have more control over the setup

AI-assisted research infrastructure

Building better tooling here means creating scaffolds that help AI understand research contexts and produce working, reliable code with good scaffolding, like:

• MCP integration: Connecting AI agents to MCP servers to access evaluation results, model outputs, or experimental data

• Automated workflow design: Using AI to generate evaluation pipelines, data processing scripts, or analysis code based on research requirements

• Structured codebases: Organising projects so AI tools can navigate research code, understand context, and suggest relevant changes

The gap

These tools exist but aren’t perfect. The key here is having a product mindset:

• Treat AI safety researchers as your customer and the tools you’re building as the product.

• Replicate their paper to understand the workflow of running an evaluation or training experiment.

• Talk to researchers to understand their pain points.

• Understand why some researchers opt against using particular tools.

• Ask the maintainers of existing tools where the gaps are.

As a software engineer (or technical product manager), you can leverage your unique value here! You can build tools that handle multiple use cases, have actual documentation, and won’t break when someone updates a dependency.

Some quick ways to start:

• Pick up issues in open source repos, like Inspect, ControlArena or TransformerLens

• Try out using the libraries to get a sense of where the gaps are

• Implement a benchmark for inspect-evals in their open issues

• Run an evaluation on a self-hosted open-source model

Here are more details on research tools and workflows.

Contribute directly to AI safety research

This means developing techniques to train AI systems to be safer, experimenting with approaches, testing what works, and implementing solutions directly on the models.

The depth of ML expertise you need depends entirely on how you want to contribute:

• More ML-heavy contributions involve designing the experiments. They propose hypotheses on safety techniques and guide the research direction. These roles are typically research leads or scientists.

• More engineering-heavy contributions involve turning research ideas into runnable experiments. They implement the training pipeline. These roles are typically called research engineers or contributors.

Note: While the job title might be the same, the split between ML and engineering varies widely by org. So don’t anchor too hard on job titles.

With a basic understanding of how AI systems work, you could:

• Test whether a safety technique that works on GPT-4 also works on Claude or open-source models.

• Take a paper on jailbreak resistance and test it on new prompts or different model sizes

• Replicate a paper and tweak one variable. (example)

• Evaluate how METR’s paper on AI’s doubling of SWE task lengths looks for offensive cybertasks. (example)

Getting these experiments to actually work is where your engineering skills shine! Research code is messy, and reproducing results often requires debugging. You’d be solving problems similar to what AI researchers face.

Then, you can post your findings on LessWrong or the Alignment Forum. These are genuine research contributions! You’re validating results, finding edge cases, and building evidence about what works. Many successful researchers started here, and there’s a lot of low-hanging fruit.

However, you’ll need far more ML expertise if you want to do things like:

• Design novel reinforcement learning approaches for alignment

• Propose radically new mechanistic interpretability techniques

• Lead research directions

If ML expertise isn’t your differential advantage, don’t force it! Your software engineering skills are already incredibly valuable. You don’t have to spend 100s of hours upskilling on ML when there are also other ways to contribute.

This isn’t to say you need to learn everything there is to know about ML or get a PhD to start. You might instead start with a particular area, research paper or question and gain just enough context to achieve your goal. If you do want to upskill, self-studying ARENA is a good place to start.

Other engineers contributing to direct research have also written advice, like Ethan Perez and Andy Jones.

What do I do now?

Start applying. AI safety needs great engineers, and many orgs are looking for engineering talent, even if they haven’t posted roles yet.

Check the BlueDot community Slack for open roles, or directly reach out to AI safety orgs and ask about their engineering bottlenecks.

While you’re applying, you can also:

• Facilitate BlueDot’s Technical AI Safety course

• Talk to engineers or researchers in AI safety to understand where the engineering bottlenecks are.

• Follow our guide to complete an AI safety project in <1 week.

• Replicate a safety paper to understand the workflow. (project examples)

• Resolve issues on or contribute to improving open source AI safety tools. (example)

• Build your ML foundation (not because you need to be an expert, but because understanding what you’re building infrastructure for makes you more effective)

There are almost certainly more ways to contribute your engineering skills to AI safety. Lean on what you do best!

Acknowledgments

As someone outside both engineering and AI safety research, I’ve leaned on the experience of others. Thanks to Adam Jones, Alexander Meinke, Jun Shern Chan, Max McGuinness, Monika Jotautaitė, Oliver Makins and Rusheb Shah for their feedback. Any misrepresentations are my own.

One common barrier is lacking the resources to do the project well. Today we’re launching an updated version of our rapid small grants program for participants and facilitators on our courses. It’s aimed at people who, without funding, couldn’t reasonably afford what they need for their chosen project.

If you’re unsure whether to apply: apply.

How it works

At all times, only assume we’ll cover costs we’ve confirmed in writing for your specific project. If you’re ever uncertain, contact us before spending money you expect us to reimburse.

1. Submit a proposal here (under 15 minutes for most applications). We typically respond within 5 working days with one of three outcomes:

   • Accepted: We’ll confirm exactly what spending we’ve approved.

   • Clarification needed: We’ll ask follow-up questions to evaluate your request.

   • Not approved: We don’t think this meets our criteria, or we’re unable to approve for another reason.

2. Do your project, confident you can spend on what we’ve approved.

3. Claim reimbursement. We process most claims within 5 working days. If you didn’t end up spending the money, no problem - just don’t submit a claim.

What we fund

We expect most grants to fall between $50 and $1,500. Amounts toward the higher end typically require evidence of initial traction - a strong proof of concept or promising preliminary results.

We fund work that is already in motion! If you haven’t started, begin with what you have and come back when you hit a blocker to moving fast or moving at all.

The following examples are illustrative; all decisions are ultimately at our discretion:

• Compute for a technical AI safety project (API costs, cloud GPU, training runs)

• Access to paywalled resources - articles, research papers, datasets, or textbooks

• Conference travel

  • We have a high bar for conference travel funding. We typically fund travel only when:

    • You’re presenting research or leading a session (not just attending)

    • The conference is directly relevant to AI safety or biosecurity

    • You’ve exhausted other funding routes (conference travel grants, institutional support, community funding)

    • You can articulate a specific deliverable or outcome beyond “networking”

    • We typically don’t fund conference attendance for networking, professional development, or career exploration. Many conferences also offer need-based travel scholarships.

• Project-specific software tools that save significant time on your specific project

• Project-specific equipment (e.g., a quality microphone for an AI safety YouTube channel)

• Participant recruitment for empirical experiments

• Hosting costs for your application or tool

What we don’t fund

• Compensation for your time on the project

• Equipment you’d reasonably already have (laptops, phones, external drives, etc.)

• General productivity subscriptions (ChatGPT Plus, Claude Pro, Cursor, Grammarly, etc. unless this is highly leveraged)

• Personal expenses

• Other funders may cover some of these - see this resources page for AI safety funding opportunities. For larger grants ($10-50k), you’ll need to apply to and complete one of our Incubator Weeks.

Before you apply

Ask yourself:

1. Have I already started? What have I built, written, or tested so far?

2. What specifically is blocking me? Can I name the exact resource and why I need it?

Eligibility

• You must be a current or past participant or facilitator on a BlueDot Impact course.

• We reimburse via bank transfer (Wise) or PayPal (UK), so we cannot send payments to sanctioned countries.

Questions or feedback? Contact us.

Apply

Submit your proposal here - it takes under 15 minutes for most applications. We aim to get back to you within 5 working days.

• misunderstood the course’s purpose,

• lacked technical readiness or

• did not sufficiently demonstrate commitment to making AI safer

Based on this, here’s some advice to help future applicants improve their chances of success!

Also, read our analysis of AI Alignment application mistakes. Almost all the advice there applies here too, especially:

• Have a clear path to impact

• Make your application easy to understand

• Strike a balance with response length

• Highlight impressive or relevant experience, even if it’s not a ‘formal’ qualification

Mistake #1: Misunderstood the course’s purpose

The course is focused on technical approaches to preventing catastrophic risks from AI, like power concentration, disempowerment, critical infrastructure collapse and bioengineered pandemics. It helps you to understand current safety techniques, where the gaps are and how you can contribute to plugging them.

Strong applicants demonstrated they understood this by:

• Articulating specific risks they’re concerned about (not just “making AI safer” broadly)

• Recognising how transformative AI could be, for better or for worse

• Showing they want to contribute to pushing the frontier of safety techniques

If you’re new to AI safety, we’d recommend our Future of AI course to start.

If you don’t have a good sense of what it means for “AI to go well”, we’d recommend completing our AGI Strategy course first to have a big picture understanding of how you can contribute.

Mistake #2: Lacking technical readiness

Strong applicants demonstrated a background in ML, either through formal experience, education or personal projects.

They demonstrated that they have sufficient understanding of how LLMs are trained/fine-tuned to keep up with technical discussions that build on this. It’s hard to critique technical proposals for training safer AI if you don’t understand the basics of how they are trained in the first place!

We’re looking for evidence that you’ve engaged deeply with the concepts, not just consumed content about them. Watching videos about AI safety and reading LessWrong is a start, but it doesn’t show us that you can work with these ideas.

Some strong signals from non-technical backgrounds include but are not limited to:

• Writing explainers of relevant technical concepts

• Facilitating discussions that require you to explain technical concepts

• Building and training your own simple neural networks (even if the code is messy!)

This course does not select against you if you don’t have a CS degree or work in tech. We’ve had successful applicants from philosophy, policy, biology, and business backgrounds. What matters is that you’ve put in genuine effort to understand the technical foundations.

Mistake #3: Insufficient evidence of commitment

We need people who’ll act on what they learn, not just learn for learning’s sake.

Some signals from strong applications include, but are not limited to:

• Identifying specific organisations and roles that match your concerns about AI

• Organising discussions, reading groups, or events around AI safety topics

• Setting aside a non-trivial amount of time / resources to transition into AI safety

• Building prototypes or tools related to AI safety

The top 20% of applicants showed they’re ready to take bold action (e.g. founding new initiatives, making significant career pivots, or leveraging unique positions of influence). But these aren’t the only paths. We’re looking for evidence that you’ll act on what you learn, whatever form that takes in your context.

It’s not about having the perfect plan. It’s about showing you’re already moving toward action, even if you’re still figuring out the specifics.

Applying to our course

The last common mistake is not applying at all, or forgetting to do so by the deadline! Now you know how to put your best foot forward, apply to our Technical AI Safety course today.

• BlueDot is recruiting founders to solve critical problems threatening civilization.

• Our approach: deep problem scoping, targeted recruitment, and capital coordination.

• First focus: pandemic PPE stockpiles.

Context

• BlueDot is a non-profit building the workforce that protects humanity.

• We’re a 5-person team. We train 1,000s of people each year in AI safety and biosecurity, and we place the highest-potential people into impactful jobs at AI companies, governments and non-profits.

• Since 2022, we’ve raised $34M.

• Today, ~2,000 people work full-time on AI safety. We believe this needs to increase to 100,000 by 2030 to prevent worst-case scenarios.

• To help scale the workforce, we’re testing a “startup studio” model alongside our courses. We’re attracting leaders, builders and entrepreneurs, and helping them start and scale the most important projects that protect humanity.

• We’re building this in collaboration with Jonah Weinbaum and Hugo Walrand.

Initial approach: AGI Strategy course

• Two months ago, we launched our AGI Strategy course alongside a $1M fund.

• We’ve attracted 15 founders to the course who’ve raised $1M+ each, and ~$50M in total.

  • We’ve invited a handful of them to join our in-person “incubator weeks”, where we’ve helped them improve their ideas and launch new AI safety companies.

  • We’ve deployed $130k in grants so far to 1) build a risk engine for AI insurance and 2) improve the cybersecurity of critical infrastructure.

• This approach works well for founders who already have a particular area of expertise, project ideas that leverage that expertise, and are motivated to work on AI safety.

• To test what it takes to drive founder-energy towards a shortlist of high-priority ideas, we’re running a second experiment in parallel.

Second experiment: Problem scoping and serial entrepreneurs

• Compared to other AI safety incubators, we’re taking a different approach:

  • Doing deep in-house problem scoping before recruiting anyone, and

  • Targeting serial entrepreneurs rather than junior talent.

• We’ve spent the past 3 weeks going deep on pandemic PPE stockpiles.

  • We’ve done 50+ expert interviews, asked for intros from everyone, read many papers and blog posts, and shared a commentable doc with everything we’ve learnt and our understanding of the situation with those experts.

  • We’re writing a “landscape analysis” to understand the current situation, and help us develop a strategy for how this problem will get solved.

• We’re headhunting serial entrepreneurs with relevant experience in the problem area we’re focused on, and making the most compelling pitch that we can for why they should work on solving this problem.

  • Founding and scaling a successful company is exceptionally hard, and the best predictor of future success is past success.

  • We make the pitch personal and concrete:

    • “Here’s the threat and its scale. Your expertise in X is exactly what’s needed to make Y happen. This is why we believe it’s solvable. Here’s the funding situation (validated business model or $100M+ in philanthropy). Here are the experts we can introduce you to. Your work could save millions of lives.”

Why we’re exploring pandemic PPE stockpiles first

• We wanted to start with a high-priority, tractable problem that is well-scoped but is still nowhere near to being solved.

• We believe bioweapons are one of the most tangible and early ways a global catastrophe could occur due to AI - see this blog post.

• In the event of a catastrophic pandemic, PPE stockpiles are essential for protecting critical workers and keeping civilization functioning.

  • Let’s imagine a bad actor releases 10 pandemic-potential pathogens into the population, each with a similar R0 to COVID-19, and a 20% infection fatality rate.

  • Once this is detected, governments lockdown their populations akin to March 2020.

  • The lockdown only holds if people continue to have food, water and energy delivered to their homes.

    • If people are cold, thirsty and hungry, they will leave their homes and get infected en-masse.

  • To maintain the lockdown, workers in the food production, distribution, water and energy sectors need to continue working.

    • If they believe they’ll get infected if they go to work, and that they could then infect and kill their families, they won’t go to work!

    • Critical workers need high-quality respirators that protect them from infection.

• Existing PPE stockpiles are woefully inadequate.

  • In the US, there’s enough N95 masks to protect healthcare workers for 10-20 days.

    • N95 masks do reduce transmission, but they don’t provide adequate protection to prevent infection in this scenario.

    • There are no stockpiles for critical workers in other industries.

    • If the above scenario happened today, critical industries would collapse, the lockdown would fail, and a significant fraction (30%+?) of the population would die.

  • We need a large stockpile of reusable respirators (e.g. elastomeric half-mask respirators, EHMRs).

    • They’re much better at reducing transmission AND they cost less than N95s when accounting for multiple months of usage.

    • The filter in an EHMR only needs to be replaced every 6-12 months.

Who’s working on it

• The field is extremely small — we know of <10 people working on it full-time:

  • Protective Equipment (1-2 FTE) - designing and manufacturing better EHMRs

  • Blueprint Biosecurity (2 FTE on PPE) - researching the best PPE to stockpile and advocating government for it

  • Amodo Design (2-3 FTE on PPE) - engineers testing PPE

  • Centre for Health Security - reusable respirator research and advocacy

• Compare this to the scale of the problem: for North America, it could cost $1-20B to build the pandemic PPE stockpiles.

  • We believe this money could come from some combination of:

    • Government procurement via the Strategic National Stockpile (SNS),

    • Investors investing into a PPE stockpile company offering pandemic insurance for critical industries, and/or

    • Philanthropic grants

  • Recent momentum includes a podcast discussion with Andrew Snyder-Beattie from Open Philanthropy, and the OpenAI Foundation’s $25B commitment to societal resilience.

  • By default, we still believe very little money will be spent in the US on effective pandemic PPE stockpiles.

    • To change this, we’ll need serial entrepreneurs attacking this problem (perhaps using market mechanisms), effective government advocacy campaigns, and sustained outreach to foundations and high net worth individuals.

Meta learnings

• We aim to build a repeatable playbook for building scalable companies that address the greatest civilisational challenges.

• We’ve landed on three questions that are critical to answer for this type of project:

  • 1) What needs to be done? What does success look like? What are the critical challenges and hurdles?

    • PPE: design masks fitting diverse populations, scale manufacturing and reduce unit costs, establish rapid distribution systems for pandemic-time.

  • 2) Who will pay for it? Are there feasible business cases? What would it take to get government funding? Could philanthropists cover it?

    • PPE: Most likely philanthropic grants (e.g. early employees at AI companies, OpenAI Foundation, Open Philanthropy), but pandemic insurance business models might be possible. We’re still testing this.

  • 3) Who will make it happen? Who are the serial entrepreneurs, leaders, motivated generalists and experts who will take responsibility to drive this work?

    • PPE: Serial entrepreneurs with design, engineering and manufacturing experience.

• Our next blog post will share our preliminary landscape analysis, detailing what work is needed, where the gaps are, and how it might get funded.

• We’ve also narrowed our target audience to three main personas:

  • 1) Serial entrepreneurs who build and scale new companies.

  • 2) Smart, motivated generalists who do research, run experiments, co-found and join as early employees.

  • 3) Domain experts who provide guidance to the entrepreneurs and generalists, help to validate what’s possible, and join as employees/executives.

What we’re doing now

• Completing our landscape analysis.

• Trying to recruit serial entrepreneurs to work on biodefense, to test if this target profile is viable.

• Evaluating funding options by doing customer interviews with philanthropists, investors and government lobbying groups.

• Building an expert network throughout the PPE space.

How to get in touch

• We’re looking for:

  • Serial entrepreneurs interested in biodefense

  • Domain experts who are excited to advise us on how to make this happen

  • Philanthropists interested in supporting this work ($1-10B needed!)

  • Smart generalists who want to jump into the deep end

• If you want to contribute, get in touch via email - we’re excited to hear from you!

Today we’re announcing v2 of incubator week.

At BlueDot, we’ve trained over 5,000 people since 2022. We launched our courses to build the workforce needed to make AI go well. But training isn’t enough. There simply aren’t enough organizations with the ambition, scale, and speed to make sure AI goes well. We need new organizations, not just more people flowing into existing ones. Incubator week means we can help the best founders build the companies and organizations the world needs.

How it works:

We’re inviting the strongest, most entrepreneurial participants from our AGI Strategy Course - people who’ve already proven they can think rigorously about AI risks. If selected, we fly you to London and host you for the week, all expenses paid.

Here’s the (preliminary) structure:

• Monday: You’ll develop step-by-step models of how threats to the future might develop. We’ll focus on deep problem space exploration, not surface-level takes. You’ll identify top experts in your area.

• Tuesday-Wednesday: You’ll build and iterate on your intervention ideas. What can actually move the needle on the problems you’ve identified? What can be implemented and scaled? By the end of Tuesday, we expect you to have called top experts working on this problem. On Wednesday we’re hosting a broader community social in the evening.

• Thursday: You’ll create your pitch for a high-impact new organization.

• Friday: You’re pitching to us for funding.

Throughout the week, you’ll work from our office at LISA alongside Apollo Research, Workshop Labs, and other leading organizations and researchers. We’ll also be bringing in founders, funders, and experts to accelerate your work.

How you can still join:

The AGI Strategy Course and Technical AI Safety course are our feeders. So far we’ve received over 2,000 applications. When it comes to incubator week we’re looking for people with strong buildery energy, clear problem thinking and a focus on mitigating AI risk.

Our next incubator week runs November 17-21. v3 will be running from December 1-5.

The future will arrive sooner than most expect, but there’s still time to shape it. We’re putting our money where our mouth is and backing people to build. If you want in - we’re saving 1-2 spots for standout talent - apply to the AGI Strategy Course and show us what you can do.