In the whirlwind world of software development, new tools emerge, promise a revolution, and then quietly fade, or, if they’re truly groundbreaking, become an indispensable part of our daily grind. ChatGPT, and generative AI at large, has firmly planted itself in the latter category. For many developers, it’s moved from a curious experiment to an essential sidekick, a silent partner in countless coding battles.
But what exactly are we asking this AI assistant? Are we just having it write boilerplate code, or is there more to the story? A recent study by Huizi Hao and a team of researchers delved into this very question, meticulously analyzing developer interactions with ChatGPT within GitHub issues and pull requests. Their findings offer a fascinating glimpse into the developer psyche, revealing not just *what* we ask, but *how* these AI tools are shaping our workflow. Let’s pull back the curtain on the most common developer queries to ChatGPT.
The Developer’s AI Sidekick: What Are We Really Asking For?
The research, which meticulously categorized 580 initial prompts, paints a clear picture: developers aren’t just dabbling. They’re engaging ChatGPT with real, everyday software engineering challenges. The study identified 16 distinct categories of inquiries, but a dominant “Big Five” truly stands out, accounting for a staggering 72% to 81% of all software engineering-related prompts. This tells us where the immediate, high-frequency value lies for us developers.
The Big Five: Dominant Developer Queries
If you’ve ever used ChatGPT for coding, these probably won’t come as a huge surprise. They mirror the everyday hurdles and tasks we face, from the conceptual to the concrete:
Code Generation: Unsurprisingly, this is the reigning champion, making up 20% of prompts in pull requests (PRs) and 27% in issues. We’re asking ChatGPT to conjure code snippets from descriptions, transform existing code between languages, or even generate test cases. It’s like having a hyper-efficient junior dev who never sleeps, ready to draft that API call or create a basic script.
Conceptual: This one resonates deeply with me. Beyond just writing code, 14-18% of developers turn to ChatGPT for clarity. “Explain WebAssembly memory limits,” or “Is it feasible to build a Redis-like cache with SQLite?” These aren’t just academic questions; they’re the foundational queries that help us understand complex systems, evaluate architectural choices, and make informed decisions before we even type the first line of code. It’s our personal technical consultant.
How-to: We all hit those moments of “I know what I want to do, but not how to start.” A significant 13% of PR prompts and a higher 22% in issues fall into this category. “How do I make an iOS framework M1 compatible?” or “I’m doing X, how can I achieve Y?” ChatGPT becomes our step-by-step guide, offering initial pathways and insights when we’re staring at a blank screen, unsure of the first move. This higher percentage in issues suggests we’re leaning on AI more often in the very early, problem-solving stages.
Issue Resolving: This is where ChatGPT truly shines as a debugging buddy. Around 12-14% of prompts involve developers seeking help to squash bugs or troubleshoot errors. We’re sharing error messages, pasting code snippets with descriptions of unexpected behavior, or even asking for help with setup issues like installing a library. The study found that direct error messages are the most common input, highlighting our desire for immediate, contextual solutions when faced with a cryptic stack trace. Interestingly, some even share external links, hoping ChatGPT can parse the context and offer solutions.
Review: This is a powerful use case, particularly evident in PRs (9% vs. 4% in issues). Developers are asking ChatGPT for suggestions to improve code quality, optimize performance, or even compare different implementations. It’s like having an impartial pair of eyes providing constructive feedback, helping us refine our work before it gets merged. This emphasizes ChatGPT’s potential not just for creation, but for refinement and quality assurance.
Beyond the Basics: Nuances Across Workflows (PRs vs. Issues)
While the “Big Five” dominate, the study also uncovered subtle but important differences in how developers use ChatGPT depending on whether they’re working on a GitHub Pull Request (PR) or an Issue. This contextual distinction is key to understanding the AI’s role in the development lifecycle.
The PR Context: Review, Comprehension, and Human Language
When it comes to Pull Requests, the focus shifts slightly towards refinement and clarity. We saw “Review” being significantly higher in PRs. But it doesn’t stop there:
- Comprehension: In PRs, 7% of prompts ask ChatGPT to explain code snippets or software artifacts (“Explain this code”). This makes perfect sense; before merging, understanding every line, especially in a complex codebase, is paramount.
- Human Language Translation: A unique finding for PRs, 6% of prompts involved translating technical text. Think UI labels, documentation comments, or project descriptions for international teams. It’s a testament to the global nature of development and ChatGPT’s often-underestimated linguistic prowess.
- Documentation: PRs also saw more requests (5% vs. 2% in issues) for creating, reviewing, or enhancing technical documentation. From commit messages to READMEs, clear documentation is vital for collaboration, and ChatGPT is proving to be a valuable assistant here.
These trends suggest that in the PR context, developers are using ChatGPT to enhance collaboration, improve code clarity, and ensure the final output is well-understood and documented.
The Issue Context: Data, Math, and Initial Problem Solving
On the flip side, GitHub Issues often represent the initial stages of problem-solving, feature development, or bug reporting. Here, developers lean on ChatGPT for more foundational tasks:
- Data Generation & Formatting: In issues, 4% of prompts asked for data generation (e.g., test inputs, API specifications) and 3% for data formatting (e.g., transforming data into JSON). This highlights ChatGPT’s role in scaffolding data-dependent tasks early on.
- Mathematical Problem Solving: A small but notable 3% in issues (vs. 1% in PRs) involved mathematical problems. This could be anything from understanding algorithms to generating test cases based on numerical logic. It underscores ChatGPT’s utility as a general problem-solver, not just a code machine.
The higher “How-to” prompts in issues also reinforce this idea: developers are often at the conceptual or early-implementation stage, seeking guidance to kickstart a solution.
Surprising Asks: What Else Developers Throw at ChatGPT
While the major categories give us the bulk of the story, some of the less frequent inquiries reveal the true breadth of developers’ experimentation and trust in AI:
Verifying Capability: A few prompts simply asked, “Are you familiar with TypeDB?” or similar. This is fascinating! It shows developers are actively trying to gauge the AI’s knowledge boundaries, assessing its reliability and building a “trust profile” before diving into complex queries.
Prompt Engineering: One developer even asked ChatGPT to provide a *better prompt* for a given inquiry. This meta-level interaction shows a desire to optimize the AI interaction itself – using AI to get better at talking to AI. A sign of things to come, perhaps?
Execution and Data Analysis: Perhaps the most intriguing findings were requests for ChatGPT to *execute* tasks or *perform data analysis*. “Try running that against this function and show me the result” or “Benchmark that for me and plot a chart” and even “find all the entries that are present in the left and in the right column” from a CSV. These aren’t just about generating text or code; they’re about treating ChatGPT as an environment or a data scientist. This hints at a future where AI isn’t just an assistant but an active participant in testing and analysis workflows.
The Future is Conversational
This research provides invaluable insights into how developers are truly leveraging AI. It’s clear that ChatGPT is far more than a glorified search engine or a simple code generator. It’s a conceptual guide, a debugging partner, a documentation assistant, and even, in some cases, an execution environment. The nuanced differences between how it’s used in PRs versus Issues highlight its adaptability across different stages of the development lifecycle.
As AI-powered tools continue to evolve, understanding these core developer needs will be crucial for building even more effective and integrated solutions. The future of software engineering is undeniably conversational, and developers are leading the charge in defining what that conversation looks like. It’s an exciting time to be building, with an increasingly intelligent sidekick by our side.
