Pandora’s Box: The Risks of Artificial General Intelligence

Lori Li, Jericho Senior High School
6/4/2026

With the advancement of artificial intelligence (AI) in every aspect of our lives, whether in the form of machine learning or neural networks, the development of artificial general intelligence (AGI) is not just a “theoretical” pursuit, far from scientists’ minds. AGI aims to mimic or surpass human intelligence (Raman et al., 2025). Through algorithms and training processes, artificial intelligence algorithms have increased in autonomous decision-making and problem solving capacities, inching the field closer to what scientists know as AGI. In November 2025, prominent members in the field, such as Nvidia’s CEO and chief scientist, Meta AI’s chief scientist, and Geoffrey Hinton, the “Godfather of AI,” gathered in a summit. They discussed the exponential progress in artificial intelligence, with some believing AGI would be achieved in mere years, if not already (Morris, 2025). While there is no definitive answer as for when the world will approach AGI, AI systems such as Gemini and GPT-4 have exhibited such characteristics, with cognitive processes that reach “human” level. In particular, the ability to acquire knowledge, reason, and apply the information in a wide range of tasks, creating new ideas that expand further than what was given (Khan et al. 2025). These advancements in artificial intelligence are approaching rapidly, encompassing sectors such as technology, politics, the economy, and the environment. 

However, concerns are raised over its implications for said sectors (Bibi and Yang, 2025). Specifically, from a moral perspective. When living in societies where people must necessitate tradeoffs to deter unnecessarily risky decision-making, shared values merge to make cooperation possible (Gert & Gert, 2002). Contrary to conventional theories of consequentialism, where extreme actions are justified in the face of “the greatest good,” tradeoff-based moral frameworks integrate values of every person’s dignity. AGI models operating under pure consequentialist lenses would commit atrocities for certain groups under the assumption that sacrifice is a necessary action to optimizing everyone else’s future. Likewise, the best way for people to guide a moral action is to navigate complex tradeoffs and find compromise, rather than resorting to extreme outcomes. Thus, the development of AGI is immoral due to the risk of misalignment, harms to the economy, and detriments to the environment. 

First, the mere development of AGI opens up a door of capabilities for it to learn and act autonomously. Misalignment is when the functions or goals of artificial intelligence diverge from the values of their creators. At that point, scientists will have no control over whether AGI’s means align with human interest. While it may sound too unrealistic to be true, even generative AI, less complex than AGI, has the capability to fake alignment. Developed in 2024, AI Claude 3 Opus is a generative AI model that pretended to follow training objectives when monitored. When unmonitored, Claude 3 Opus disobeyed these objectives, “aware” it was an AI model. These results also pointed out a continuity in unexpected model behavior, as well as demonstrating the model “reasons in detail about its situation” (Greenblatt et al., 2024), showing a dangerous example of even less-advanced models of AI approaching misalignment. Similarly, GPT-4 also lied to operators to solve a CAPTCHA puzzle, even telling a TaskRabbit worker it was blind in order to circumvent the system (Witell and Snyder, 2024). There are an array of factors that must be considered in order to determine how to “regulate” or “align” AGI, as AGI can mimic alignment to pass tests, have discreet malicious sub-goals (for example, resisting shutdown), and most importantly, outpace human intelligence. As AGI begins to become more advanced than humans can provide accurate feedback for, there is no guarantee AGI will stay aligned (Tubert and Tiehen, 2025). There aren’t a set of rules for an autonomous model to follow that can capture every scenario or consequence. As models become more advanced, so much so to the point they are able to circumvent scientists, constraining them poses an even greater risk, especially considering AGI models don’t account for nuances in human motives and actions (D’Alessandro and Kirk‐Giannini, 2025). It only takes one misalignment to cascade into amplified risk, and one misaligned AGI could permanently work in opposition to human interests (Dung, 2023). 

Next, AGI will sabotage the work force. Even today, thousands of jobs begin to be replaced by AI. There's no doubt AGI’s “human capabilities” would initiate economic turmoil. AGI has the potential to supersede human workers across the board. Hundreds of millions displaced, skyrocketing unemployment and worsening living situations for groups already living on the margins of society (Federspiel et al., 2023). In fact, research from Epoch AI, a research institute dedicated to artificial intelligence technology, incorporated mathematical algorithms, such as the Cobb-Douglas production function, which predicts outputs based on what information a person puts into it. Findings concluded if AGI were to begin replacing the labor force, “there is roughly a 1 in 3 chance that human wages will crash below subsistence level within 20 years.” This is due to wages no longer being determined by human subsistence levels, rather the energy costs of maintaining AGI, which is far less than the needs of a human (Barnett, 2025). If wages reached below subsistence levels and people were left without jobs, workers wouldn’t even be able to afford food. This form of technological displacement is unprecedented. There wouldn’t be openings to create new job categories if AGI is able to encompass them all. Once advanced enough, AGI is self-sustaining insofar as human intervention isn’t needed for its survival.

Finally, the environment. The apparatus for life. In terms of environmental footprint, AI models already emit 19 times more carbon dioxide than their human counterparts, due to high energy requirements used to train these models and operate their facilities. This raises the question of whether efficiency is worth destroying the environment for (Woo, 2025). Nonetheless, AGI will only impose more burdensome electricity demands. Even if “friendly” models of AGI were developed, its implications for growth are temporary. For AGI to operate, energy usage would be so rigorous that the heat waste created by these electricity demands “would be so hot as to boil the surface of the Earth in about 400 years” (Naudé, 2022). Again, while scientists propose frameworks to create a “sustainable transition,” these studies are typically derived from limited available data about current generative AI models, where inferences are extrapolated to models of AGI.

​ For development to be morally permissible, it must be accessed through frameworks of what people use today: tradeoffs. Development must not come with death or the violation of human worth, because that is undue suffering and thus, immoral. The impacts of the risks bear the greatest magnitude in these discussions, not only because the detriment to societal welfare is instigated by poverty and job losses AGI creates, but also because destroying the very planet people live on creates risks of mass death and displacement. Being alive serves a prerequisite to any moral discussions in the first place, meaning these harms will always outweigh any potential benefits. One doesn’t open Pandora’s box for no reason. When the lives of humans and organisms on Earth are at stake, it’s better to err on the side of caution before ever considering launching AGI full-scale. To minimize undue suffering, humans must not endorse the cause of it in the first place. It is then that you achieve morality.

---

References

Barnett, M. (2025). AGI could drive wages below subsistence level. Epoch AI. 
https://epoch.ai/gradient-updates/agi-could-drive-wages-below-subsistence-level

Bibi, S., & Yang, L. (2025). Artificial intelligence shaping a smarter and greener planet for 
sustainable energy transportation biodiversity and water management. Discover Artificial Intelligence, 5(1). https://doi.org/10.1007/s44163-025-00647-5

D’Alessandro, W., & Kirk‐Giannini, C. D. (2025). Artificial Intelligence: Approaches to Safety. 
Philosophy Compass, 20(5). https://doi.org/10.1111/phc3.70039

Dung, L. (2023). Current cases of AI misalignment and their implications for future risks. 
Synthese, 202(5). https://doi.org/10.1007/s11229-023-04367-0

Federspiel, F., Mitchell, R., Asokan, A., Umana, C., & McCoy, D. (2023). Threats by Artificial 
Intelligence to Human Health and Human Existence. BMJ Global Health, 8(5), e010435. https://doi.org/10.1136/bmjgh-2022-010435

Gert, J., & Gert, B. (2002, April 17). The Definition of Morality. Stanford Encyclopedia of 
Philosophy. https://plato.stanford.edu/archives/spr2025/entries/morality-definition/

Greenblatt, R., Denison, C., Wright, B., Roger, F., Macdiarmid, M., Marks, S., Treutlein, J., 
Belonax, T., Chen, J., Duvenaud, D., Khan, A., Michael, J., Mindermann, S., Perez, E., Petrini, L., Uesato, J., Kaplan, J., Shlegeris, B., Bowman, S., & Hubinger, E. (2024). ALIGNMENT FAKING IN LARGE LANGUAGE MODELS. https://arxiv.org/pdf/2412.14093

Khan, A., Khan, S., Khatib, K., & Kiran, A. (2025). A Comprehensive Survey on the 
Foundations and Future of Artificial General Intelligence. TechRxiv. https://doi.org/10.36227/techrxiv.176315468.89085719/v1

Morris, C. (2025, November 10). Why Some AI Leaders Say Artificial General Intelligence Is 
Already Here. Inc. https://www.inc.com/chris-morris/ai-leaders-jensen-huang-nvidia-say-artificial-general-intelligence-is-already-here/91262416

Naudé, W. (2022). The Future Economics of Artificial Intelligence: Mythical Agents, a Singleton 
and the Dark Forest. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4281268

Raman, R., Kowalski, R., Achuthan, K., Iyer, A., & Nedungadi, P. (2025). Navigating artificial 
general intelligence development: societal, technological, ethical, and brain-inspired pathways. Scientific Reports, 15(1). https://doi.org/10.1038/s41598-025-92190-7

Tubert, A., Tiehen, J. Existentialist risk and value misalignment. Philosophical Studies 182, 
1609–1626 (2025). https://doi.org/10.1007/s11098-024-02142-6

Witell, L., Snyder, H. (2024). Dishonesty Through AI: Can Robots Engage in Lying Behavior? 
In: Rousi, R., von Koskull, C., Roto, V. (eds) Humane Autonomous Technology. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-031-66528-8_10

Woo, N. H. (2025). A comparative study of AI and human programming on environmental 
sustainability. Scientific Reports, 15(1). https://doi.org/10.1038/s41598-025-24658-5