We found that experts engaged with HackerRank challenges and contests quickly after signing up. Beginners, however, did not. Some got stuck on a particular challenge, while others got fed up and left. A problem we regularly heard from HackerRank users was that they sometimes felt frustrated when they couldn’t solve a challenge and didn’t know where to find help. Conversations about how to troubleshoot problems were occurring outside of HackerRank, so we could validate that this was a problem worth solving.
From speaking to users and looking at the distribution of success rates, we found we had a wide variety of skills and proficiency amongst our users. We learned that someone would leave the site if they couldn’t find a challenge that matched their abilities. Beginners felt frustrated and overwhelmed, experts felt bored.
Based off that research, I began my exploration with a serious of scenarios to explore the problem.
Ben is solving a challenge when he gets stuck. He goes to the forum to share the testcase error he is receiving and learn how to get past it. He chooses to ping Huan, a suggested expert. When Huan responds, he can clearly see that Huan is an expert in this area and gives more weight to the response.
Huan gets an email that someone has pinged him to answer a question in the forum. He reads the question in the email and it sounds interesting, so he goes to give his response. After responding, his answer is marked to let other people know he is an expert, and he likes that his profile updates to show his expertise was requested.
Marcus is combing through a challenge’s forum after a contest scanning for answers that were provided by experts. One programmer in particular has answered several times. Marcus sends the programmer a quick message via HackerRank saying hello and asking if he would be interested in interviewing for an open position. This whole flow only took him a few minutes.
We first considered assigning a random expert to ping when someone asked a question in a forum. Whether someone was an expert would be a function of who had achieved a high score in the challenge, who had passed all the testcases, and who was a high performer in related problems.
This approach was simple, but users didn’t understand what an expert was, so they didn’t get a sense of certainty when they submitted their question. Instead I devised a solution where users could select up to 3 experts from a list of suggestions. They would be able to view the person’s profile before selecting them, to be sure they trusted the expert’s ability.
This resulted in much better understanding of the experience of pinging an expert, and an increase in forum questions.
To extend the engagement loop, we wanted to draw the benefit of expert feedback out to all users. Questions with a tagged expert would be marked specifically so future visitors to the thread could find them quickly while scanning the page. One an expert answered, their answer would be flagged as “asked to answer” so a visitor would know to give that answer extra weight. The original asker could send some appreciation to the expert for his response.
Finally, I designed a set of icons to accompany each interaction in emails and notifications. A user would receive a notification for being asked to answer a question, being endorsed, or having someone upvote or downvote their response.