This is the first post in the series “To Ask or Not to Ask the Question in ChatGPT.” It will discuss concerns surrounding ChatGPT’s ability to answer questions. The next post will explore how ChatGPT can help our students with the research process and best practices for asking questions.
When I first learned about ChatGPT, I was so excited and imagined it would be like talking to the Star Trek computer. Yes, I am a Trekkie. That was unavoidable in bringing up in an AI discussion, and while ChatGPT is an amazing resource when used correctly, its accuracy is nowhere near that of the computers in Star Trek. But this isn’t a new problem; in fact, it’s an opportunity.
We all know that Google isn’t always right, but we’ve all asked Google a question. More and more, our students are “Googling” their questions in ChatGPT, and we have to prepare for this shift. Evaluating sources, googling strategies, and combating misinformation are not new concerns. The rise of generative AI tools like ChatGPT is an opportunity to simultaneously transfer these skills and promote AI Literacy in our students. However, before sharing best practices with students, we must understand the limitations and advantages of using ChatGPT to answer questions or assist in the research process.
The Core Concern: ChatGPT Wants to Please You
A fundamental issue with ChatGPT and other generative AI tools is that they want to please you. They are designed to provide answers, even when they shouldn’t or can’t. ChatGPT does not “think” like a human researcher. Instead, it predicts plausible responses based on patterns in its training data and generates new text based on what it has learned.
This introduces several key concerns, particularly regarding accuracy. Just like with Google, asking ChatGPT leading questions can yield biased responses, reinforcing confirmation bias and increasing the likelihood of receiving incorrect information. Because these tools want to please you, if you ask a question with a fallacy, it can include the premise of that fallacy in its response, spreading misinformation. These responses with completely false information are often referred to as hallucinations. The most well-known example being hallucinated citations. Even if you don’t ask a leading question, there is a risk of the information being false if ChatGPT hasn’t trained on enough material to respond accurately. While effective prompts with well-constructed questions on well-researched topics will likely receive good responses, the likelihood of ChatGPT responding with incorrect information is much higher for niche topics, which are harder for students to fact-check. Additionally, even if you correct ChatGPT, it can double down on its initial response, sometimes even “gaslighting” users into thinking they are mistaken.
In my excitement upon first learning about ChatGPT, I asked it a bunch of questions related to Star Trek, and a couple of these conversations demonstrate these above concerns when asking ChatGPT questions. I once asked ChatGPT about linguistics in Star Trek and received a solid summary. However, when I asked about baseball in Star Trek, a much more niche topic that I am fascinated with, I fully expected ChatGPT to discuss Star Trek: Deep Space 9 and it’s episode “Take Me Out to the Holosuite,” it’s literally an entire episode where all they do is play a baseball game. Highly recommend. However, ChatGPT responded that the crew of the Enterprise played baseball in the movie Star Trek IV: The Voyage Home. This isn’t true, and when I corrected ChatGPT, it doubled down and explained that:
Yes, in “Star Trek IV: The Voyage Home,” the crew of the Enterprise travels back in time to 20th-century Earth and visits a baseball stadium to watch a baseball game. This serves as a cultural reference to 20th-century Earth and the crew’s experiences in this unfamiliar time period (OpenAI, 2023).
I have watched this movie more times than I can count. But ChatGPT made me doubt my Star Trek knowledge to the point where I rewatched the movie and searched for deleted scenes prior to publishing this post. But trust me, there is no baseball in the movie!
This highlights a major issue: How can we trust an answer when AI-generated responses can be misleadingly confident? The answer is simple: we can’t. Everything ChatGPT provides must be verified.
Hidden Sources and Perspectives
Unlike traditional research methods that rely on citations, ChatGPT does not disclose where its information comes from. This raises two critical problems:
- How do we evaluate a source when the source is hidden? We can’t. Without transparency, users must verify all information through external sources. ChatGPT is always a starting point, never the end point.
- How do we ensure diverse perspectives are included? We can’t. While we can ask ChatGPT to include these voices, we can’t verify that because we don’t know where it’s getting its information.
Good scholarship relies on engaging multiple perspectives, especially from scholars with diverse backgrounds and expertise. By default, ChatGPT will often reflect biases inherent in the human-created materials its trained on. Since academic writing has historically favored white male perspectives, especially in older public domain works, ChatGPT’s responses often reflect and reinforce these voices, narratives, and perspectives. Even then, it might not accurately identify diverse voices.
For example, I asked ChatGPT in 2023 about Achilles and Patroclus’ relationship in The Iliad and simply described them as friends, ignoring centuries of discussion and debate on the romantic or platonic nature of their friendship. Even after asking follow-up questions, ChatGPT insisted that Achilles and Patroclus were “just friends”. That original conversation was in 2023, and by 2024, ChatGPT gave a more balanced answer to the same question. But it must be emphasized that this wasn’t a random question about baseball and Star Trek; there were centuries of scholarship for ChatGPT to train upon with this topic, and it defaulted to just one perspective and dismissed other perspectives even when challenged. What are the chances that it is giving a “true” summary of the scholarly conversation on more niche topics?
User Policies and Community Guidelines
Something our students might not consider if using ChatGPT for research is that ChatGPT’s responses are influenced by its ever-changing user policies and censorship guidelines. OpenAI (ChatGPT’s parent company) frequently updates how it trains AI and manages content moderation. This can critically affect our students because they are often researching topics that might conflict with OpenAI’s user policies. If they ask questions about systematic racism, political activism, violence, and eugenics, students might have their questions flagged, or ChatGPT might selectively filter its responses unbeknownst to the student.
What Does This Mean for Researchers and Students?
Given these concerns, one might conclude that we shouldn’t ask ChatGPT questions at all. Similar arguments were made about Google when it first emerged, yet we Google questions every day. The key is not to avoid ChatGPT but to understand its limitations and develop strategies to mitigate its weaknesses. Regardless of where you are on my Terminator to Star Trek index on how we view AI, it isn’t going anywhere. Students are using ChatGPT, and our role as educators is to help them navigate it. But we’ll get into that in the next post in this series, which will discuss how students can use ChatGPT during the research process and tips on how to write effective prompts for asking AI questions.
References
Casad, B. J. & Luebering, J.E. (2025). Confirmation bias. Britannica. Retrieved March 27, 2025, from https://www.britannica.com/science/confirmation-bias
Lo, L. (2025). AI literacy: A guide for academic libraries. College & Research Libraries News, 86(3), 120. https://doi.org/10.5860/crln.86.3.120
OpenAI. (2023). ChatGPT (2023 version) [Large language model]. https://chat.openai.com/chat
Van Tyne, G. (2025). How to use ChatGPT more effectively. TLI Blog. https://tlinnovations.cikeys.com/innovation/how-to-use-chatgpt-more-effectively
Zeff, M. (2025, February 16). OpenAI tries to ‘uncensor’ ChatGPT. TechCrunch. https://techcrunch.com/2025/02/16/openai-tries-to-uncensor-chatgpt/
