In the rapidly evolving landscape of artificial intelligence (AI) and its integration into various industries, the software development domain remains at the forefront of innovation. Developers today are equipped with an unprecedented set of coding tools and AI agents designed to navigate and simplify the complexities of software engineering. However, as software projects grow intricate, the demand for precise and efficient coding assistance becomes critical. That’s why we conducted a StackSpot AI User Study to learn the Developer’s Viewpoint about our solution.

Curious about what developers have to say about StackSpot AI? Good read.

Limitations of existing coding assistants

Despite their groundbreaking nature, general-purpose AI assistants – that is, tools that, although trained on vast amounts of publicly available data, know nothing about your company or your daily job routine – currently have a fundamental shortcoming: they often provide generic or inaccurate responses, particularly when confronted with contextualized, domain-specific queries. 

This gap is felt by developers who seek guidance, for example, in tasks related to optimizing a database query or deciphering the complexities of a proprietary codebase. 

While advanced in many aspects, conventional AI tools such as ChatGPT and Google Bard often fall short of delivering the depth and specificity required in these scenarios. 

Such limitations hinder productivity and pose a barrier to harnessing the full potential of AI in software development.

The industry is responding to these limitations by developing contextualized coding AI assistants. These tools, underpinned by advanced AI models, are specifically designed to access and utilize proprietary, domain-specific knowledge, which general-purpose assistants typically lack. 

This specialized approach enables them to offer targeted assistance, which is especially useful in complex, domain-specific scenarios. 

Beyond General AI

To illustrate, imagine a scenario where a developer is working on an intricate e-commerce platform and encounters a challenge related to optimizing a multi-tier product recommendation algorithm. 

While a general-purpose AI might offer broad-based guidance or algorithmic solutions, a contextualized coding assistant, familiar with the proprietary nuances of that specific e-commerce platform and its surroundings, might pinpoint exact issues based on historical data or even provide solutions that account for platform-specific constraints, or company-based frameworks. 

This is why every developer should consider using a contextualized coding assistant.

In essence, while a general-purpose AI tool might suggest generic best practices, a contextualized assistant could reference company-specific requirements documents and related projects, offering answers that are not only effective but also tailored to that organization’s unique needs. 

This is where StackSpot AI shines! 

StackSpot AI’s Idea

StackSpot is a highly contextualized coding AI assistant that considers individual developers’ nuanced requirements and the intricacies of specific codebases.

The main idea behind StackSpot AI is that we could generate better responses if we could enrich the prompt with contextualized information, that is, our knowledge sources. 

In summary, RAG enhances LLM-generated content by anchoring it in external knowledge sources. In question-answering systems, RAG accesses up-to-date, reliable information and provides transparency to users regarding the model’s information sources, promoting trust and verifiability. 

In another blog post, we have detailed StackSpot AI’s architecture, describing how we use Facebook’s RAG approach to support this enriching process.

StackSpot AI User Study

To understand to what extent developers take advantage of StackSpot AI, we have been conducting numerous user studies with the developer community. The study aimed to familiarize participants with the StackSpot AI tool, guiding them through its capabilities and functionalities via an online meeting. 

The facilitators’ approach involved progressing from introductory concepts and product demonstrations to interactive and hands-on exploration, followed by discussions. The objective was to ensure participants gained an understanding of the StackSpot AI tool, its features, and its applications.

The session started with a welcoming session to introduce the facilitators and encourage the completion of a pre-study questionnaire. This was followed by a tool demonstration, highlighting key features and interaction methods, equipping participants with the knowledge to perform basic operations. 

After the demonstration, a portion of the meeting was dedicated to independent exploration by the participants. This phase included replicating the demonstration, applying the tool to simulate operations like banking transactions, and experimenting with different knowledge sources to enhance the tool’s usage in line with the participant’s field.

The user study concluded with a group discussion, in which the participants reflected on their experience with the StackSpot AI tool and provided feedback for its development. These discussions were instrumental in building rapport and ensuring engagement throughout the study.

What are the users’ perceptions of using StackSpot AI?

Now, we’ll learn some of our users’ perceptions next. For more details about the architecture or the users’ perception of using StackSpot AI, please read our detailed research report.

Generation of Accurate Codes for Swift Integration with Internal APIs 

Initially, participants expressed skepticism about the accuracy and completeness of the code generated by the tool, particularly for complex API integration. During the user study, the participants were handed a few API files, which they used to generate integration using the tool. 

Their experience revealed the tool’s ability to generate functional code snippets, acknowledging this as a crucial feature for accelerating development cycles and reducing errors. This was highlighted by one of the participants, who mentioned the following:

“Thus, the tool accurately grasped the context, generating a code with Spring Framework — it was impressive. I did not expect it to work out so well. The tool created the class as I requested. Indeed, I am quite amazed here.”

Aggregating Knowledge Sources Within the IDE

As mentioned among the benefits, participants specifically said that the tool saved time by eliminating the need to search for knowledge sources because they are now “available” within the IDE. This feature was highlighted as a significant time-saver, influencing some study participants’ productivity perception. 

For instance, one respondent mentioned, “It saves time, as it’s a shortcut for accessing information. If done through conventional means, you’d have to search a search engine, consult books, or find people with that information to assist you, which would take longer.” 

Another participant mentioned that the tool could reduce interruptions by providing knowledge sources within the IDE: “I believe it can save our time, as it allows us to reduce interruptions when seeking specific information.”

In summary, providing instant access to relevant knowledge sources directly within the tool enabled users to access necessary information or code samples without disrupting their workflow.

Iterative refinement of generated codes through chat interaction

Since the StackSpot AI tool operates as a conversational agent equipped with internal memory to record past interactions, users can engage with it like a chatbot. 

Maintaining a history of previous conversations was initially viewed as interesting. It facilitates interactive code refinement and assists in honing code outputs to align precisely with specific project requirements, enabling users to achieve more optimal coding solutions.

Productivity gains unlock only if users know how to use StackSpot AI

A participant commented that, like any AI tool, it only brings time-saving benefits if used correctly with refined prompts and proper settings. If misused, it could even lead to time wastage. For instance, one participant complemented the following: “However, if used carelessly or by less experienced people, it may result in more work for the more experienced developers.”

This insight underscores the importance of understanding how to interact with AI tools effectively. Properly formulated queries and a clear understanding of StackSpot AI’s capabilities are essential to harness its full potential and avoid counterproductive outcomes.

Figuring out what is a good knowledge source 

Knowledge sources are representative documents that enrich the prompts for RAG’s generation component, providing essential context for task development. Without these sources, responses from StackSpot AI would be less contextualized, resembling the answers from general-purpose coding AI assistants. 

Thus, identifying effective knowledge sources is vital for StackSpot’s performance. 

Our user study revealed that not all participants could understand what constitutes a (good) knowledge source. This was observed during group discussions about other kinds of knowledge sources they would use based on their team context. While a few participants could give interesting examples (e.g., using a database schema as a knowledge source and asking the tool to create SQL queries based on it), other participants could not come up with one single example. Furthermore, others gave examples that were not based on coding tasks, and a few participants even mentioned that they “need to understand more about it, but were able to use it in a very basic way.”

Concluding

In conclusion, StackSpot AI emerges as a significant advancement in coding AI assistants tailored to individual developers’ unique needs and specific codebases. 

Our user studies highlight the tool’s effectiveness in generating accurate code, aggregating knowledge sources within the IDE, and refining codes through interactive chat, demonstrating its potential to enhance productivity and streamline development processes. 

However, the user study also underscores the importance of understanding how to effectively use such AI tools, including selecting appropriate knowledge sources and formulating precise queries. 

As AI continues to evolve, tools like StackSpot AI user study are pivotal in bridging the gap between general-purpose AI assistants and the specialized demands of software development, ultimately driving innovation and efficiency in the industry. 

Leave a comment on this article about “StackSpot AI User Study” and check out our blog for more content!