Career Mentoring in Online Communities: Seeking and Receiving Advice from an Online Community

Online Q&A Forums as a Source of Career Mentoring. Q&A forums are online communities for sharing expertise, where individuals can pose questions and receive multiple responses from a broad set of individuals [2]. In Q&A forums, the advice seeker may have little previous mentorship interactions. Prior work has examined the content and dynamics of these systems, such as on Yahoo Answers [2], Quora [47, 55], and StackOverflow [39], but has yet to explore the nature of the career mentoring interactions in such environments.

Q&A forum interactions differ in nature and structure from offline mentorship. In Q&A forums, questions are public to a relatively unknown and often anonymous audience of many potential respondents [24]. Questions on Q&A forums such as StackOverflow can also receive responses very quickly, often within 12 minutes [39]. These forums rely on a combination of intrinsic and extrinsic motivation to participate, including through visible reputation markers and connections to one's interests and identities [28]. Responses to questions can benefit the person who originally asked the question as well as the third-party observers. Posted questions can be conversational, discursive, or informational [24]. Question types include: factual (seeking objective data), advice (seeking recommendations), opinion (seeking others’ viewpoints), and non-questions (spam) [28, 39, 44]. While research demonstrates that these question patterns persist in Q&A around career advice, it is unclear what kind of challenges people experience in posting these questions online.

Research Site. Our research focused on the Workplace, a Q&A subforum in the StackExchange network self-described as “Q&A for members of the workforce navigating the professional setting” [1].Workplace acts as a career resource for its members, who are mainly software developers [4]. Stack- Exchange allows users to post questions visible to anyone, comment on others posts, and upvote/downvote questions and answers [48]. Authors of questions can then choose one answer as a “qualifying” response to their question. In 2015, 16k questions and 52k answers were posted on Workplace, with no unanswered questions [1]. To understand the context of our research site, we interviewed five participants on Workplace: one moderator and four top contributors from the IT/IS profession. Interviews focused on motivations behind seeking advice and contributing responses on Workplace. These interviews helped us understand the nature of career mentoring in this forum and gain insights on participation.

Potential Benefits of Online Career Mentoring. Based on our interviews, we identified a series of benefits in seeking career advice in an online community rather than through offline sources: validation of offline mentoring, diversity, access, and safety. We also identified two key motives to contribute as mentors in Workplace: reciprocity to the community and personal skill improvement. A summary of our findings is presented in Table 1.

Data collection: Requests and Responses. StackExchange releases “data dumps” of all of its publicly available content roughly every three months via archive.org, and offers that information through the Stack Exchange Data Explorer (SEDE). A data dump is an anonymized dump of all usercontributed content on the Stack Exchange network. Each site is formatted as a separate archive consisting of XML files zipped via 7-zip using bzip2 compression. Each site archive includes Posts, Users, Votes, Comments, PostHistory and PostLinks. The Workplace StackExchange site is 52.4mb of compressed text files.

From the StackExchange data dumps, we collected a random sample of 847 thread-starting posts (April 2012-August 2016) along with follow-up responses and post and response metrics. In terms of anonymity, 534 unique users of these posts were anonymous (89.9%), 20 users revealed their name in the url but have a nickname on StackExchange (3.4%), and 40 have both their names and reveal their names on url (6.7%). The average number of responses for the posts in our dataset was 3.14. 304 original posts had an accepted response, 182 out of which had at least one more (rejected) response.

Regarding the social interactions in the Workplace posts in our dataset, 186 out of 847 posts (27%) had no followup of any kind from the original poster after the initial request, whereas 58.5% had some follow-up from the original poster requesting advice, such as through within-post comments and answers. The follow-ups consisted of the following types of behavior: acknowledgments of the responses such as thanks and gratitude expressions (29% of posts), clarifications of the original post/question (21% of posts), asking additional questions (8.5% of posts), and “other or irrelevant” posts (14.5% of posts). Consistent with previous findings [48], these indicate that users view the online community as a mentoring platform, with a “mentoring value” ranging from a simple “thank you” to asking more questions.

Taxonomy Development. We used 597 of the 847 anonymized public posts from Workplace forum. We limited our analysis to posts with tags such as as “career development” and “careers” from 515 unique users. We excluded 127 student posts. Following previous guidelines [30, 42], a single researcher (first author) conducted all of the coding. She coded posts using an inductive process where major themes emerge through interpretation of the data [46]. This process allows the researcher to continually create and refine categories to develop a complete or saturated categorization of the data [20]. The process was to (1) review the collected data and identify preliminary themes and categories, (2) revise these categories based on their relevance to subsequent new data as collected, and (3) create typologies for main themes and categories. First-order coding categorized the naturally occurring themes in the data. We used NVivo to calculate the frequency of the codes (i.e., the number of times a code was mentioned within and across the posts, [38]). The process resulted in 470 unique posts by 424 unique users coded into three key-types and 21 sub-themes.

Taxonomy Validation. We applied the developed codes to a new set of 250 posts with the same features as the original ones. The posts were drawn in a 19-months time period, from January 2016 to August 2017. Two independent coders were cross-trained using 50 posts and then independently coded the remaining ones (200 posts from 182 unique users). Regular meetings occurred to resolve disagreement in code applications and refine the initial set of codes. Cohen's Kappa ranged from 0.75 to 0.90.

Best Practices. The first major category of requests focused on seeking advice for best practices. Best practice requests sought general solutions or approaches to career relevant problems that were superior to other alternatives or the standard approach used in the industry. This category includes seven sub-themes, such as managing interviews, time, and resumes, among others (see Table 2). For example, users sought advice on how to improve their resume as a best practice:

“I have 25 years of experience with different companies, working in high-tech software industry. I have many links, skills, and education that made my resume more than 4 pages long. How can I reduce its length without short changing my skills and accomplishments?” (Post and User's ID are removed; text is paraphrased)

Threats to Career Progress. The second major theme was seeking advice for problems that could hamper one's career progress. Career progress outcomes include likelihood of promotion, status within the organization, or marketability and employability beyond their current position. This theme was characterized by requests for advice on how to manage challenging situations that individuals encountered and were concerned could negatively impact their performance, likelihood of promotion, or reputation in the organization or beyond. These situations were long term or chronic.

Seven sub-themes fell under this category: managing relationships, motivation, failures, health, organizational inclusion, resignation, and proving oneself. Here is an example post in this category from a user asking for advice on how to manage a demotion as a potential threat to their career:

“I've been working as a software director for 3 years with a team of talented developers building a complicated piece of code. We now have about 10,000 lines of operational code... Recently, the company got a few high-profile people that want to scrap everything and rebuild it from scratch. I argued against the idea. Now they ask me to take a position as an entry level software engineer in another team that does “fun and creative” work. How can I deal with it? I don't want to resign, but my options seem limited. How bad does it look on a resume to see from “Software director ” -> “Software Engineer” being the same company.”

Time-Sensitive Decision Making. The last type of requests we identified were those seeking solutions to or input on a time-sensitive decision. These posts were characterized by a detailed description of a specific situation the poster was experiencing, a request for help in making a decision within that situation, and the time demand associated with the decision. This category includes four sub-themes: negotiations, ethics, and avoidance. For example, one user posted about managing time-sensitive job offers:

“Right now I work part-time, but previous week my boss offered a full-time position. However, I started interviews with another company just before the offer because I do not like my job and have been job hunting. I did well on the interview and they want a second round next week, but my current boss wants an answer tomorrow. I really want to do the other interview, but they have not made an offer. How can I handle the situation?”

Sample. We randomly selected 36 original posts from our dataset of 470 posts (see Study 1) that had at least one accepted response and at least one rejected response, and created 72 unique pairs (original post-accepted response; original post-rejected response). Accepted responses had a positive follow-up comment in the same thread (e.g., a “thank you”). We randomly selected rejected responses when these were more than one.

Measuring Request Types. Authors used 6-point Likerttype scales to independently rate the extent to which each of the 36 requests fell into the three main request categories: (1) was about best practices, (2) reported a threat to the requester's career, and (3) was time-sensitive. Because interrater reliability was high (ICC = .86), we averaged the three scores for each request on each of the three metrics.

Measuring Functions of Career Advice Responses. We employed Amazon Mechanical Turk (MTurk) workers to evaluate the functions and quality of the 72 responses. To ensure quality of MTurk responses, we employed only MTurk workers with 5,000 Human Intelligence Tasks (HIT, i.e., the tasks MTurk workers complete on Amazon platform) with a 98% or higher approval quality and paid them $3 for a 15-minute task. We duplicated certain items, included a screening question, and requested workers to justify their response ratings.

We applied previous literature on mentoring functions to develop a coding scheme to measure features of responses in this sample [11, 35]. We used the conceptual definitions of mentoring functions described previously (i.e., informational support, instrumental support, socio-emotional support, and role modeling) to identify specific instances of these functions in the responses in our corpus.

Second, based on examples from the corpus, we generated 13 survey items that described how the mentoring functions were enacted in the online mentorship site (Table 3). Since we saw no instances of instrumental support in the corpus, we did not create survey items for this construct.We saw two forms of informational support: general and specific information (or guidance). General information refers to the extent to which responses provide information that can apply to many situations and individuals, rather than being tailored to the original request. This type of elicited response is similar to informational support both in online and offline career mentoring. Guidance is another type of informational support that refers to problem solving strategies and tactics tailored to the original request. Socio-emotional support was shown in responses that encouraged the user to engage in some sort of activity to help the requester feel better. Encouragement was also used to provide non-emotional support, such as for promotion focus (i.e., turning one's attention toward goal pursuit [26]). We also identified aspects of role modeling, where advice-givers shared their own personal experiences and the extent to which those experiences were positive.

Third, to validate the dimensions of support that we identified and to create scales,we conducted a principal component analysis (PCA) with direct oblimin rotation method to assess the extent to which the items were measuring the same underlying construct. For each of the 72 responses, six MTurk workers evaluated the extent to which the 13 survey items described the response on a 6-point Likert scale.We dropped two survey items that did not load highly (<.40) on any factor. The top four factors from the PCA analysis explained 68% of the variance in the ratings and revealed four online mentorship functions, with general information and guidance loading into two different factors (general information, guidance, encouragement, and role modeling).

We then constructed multi-item scales for each of the mentoring functions by averaging the MTurkers’ judgments for a survey item, and then averaging the survey items designed to measure a specific mentoring function (Table 3). Intra-class correlations (ICCs) with one-way random effects ranged from .81 to .84, indicating that most of the variance in MTurker judgments could be attributed to the post being evaluated. Alphas for each sub-scale ranged from .78 to .84, indicating good consistency among items measuring the same construct.

Requesters’ Judgement of the Value of Responses. We measured the response value to the requester with a binary variable. We coded 1 if the requester (a) voted the response as “accepted” and (b) followed up with within thread comments. Rejected responses were coded as 0.

Third-Parties’ Judgement of the Value of Responses. The value of a response to third party viewers was assessed using MTurkers’ ratings. We used two items toward that purpose: “To what extent do you believe the Answer is helpful”, and “How much effort do you think the writer of the Answer put in to the response” MTurkers rated their agreement with these items on a 6-point Likert-type scale. These items had high internal consistency (ICC = .88, Cronbach's alpha = .86). Thus we averaged them to create a value score as assessed by third parties. This value indicator was moderately correlated with acceptance by the original requester (r = .57, p = .001).

Request Type and Response Features. We conducted linear regression with response nested within request, wherein standard errors are adjusted because observations are nonindependent. Here non-independence occurs because we analyzed cases where we randomly selected one rejected and one accepted response. None of the request types elicited different functions of responses.

Response Functions and the Value of the Response. We used logistic regression with response nested within request, to examine whether functions of a response predicted the likelihood of the original requester accepting the response. Responses containing more guidance were more likely to be accepted (β = 4.13, t = 2.82, p = .005), while other request features were not significantly related to acceptance (see Table 3). In a follow-up analysis not reported in Table 3, we included interactions between request type and response functions, but found no significant interactions (p = ns). This suggests that guidance is valued by the original requester across request types.

Next, we used linear regression with response nested within request to assess how fuctions of responses were associated with the value of a post as rated by MTurkers. Functions of responses explained 65% of the variance in MTurks’ judgments of response value. Responses with more general information (β = .41, t = 3.74, p = .001), encouragement (β = .23, t = 2.54, p = .011) and guidance (β = .73, t = 7.21, p = .001) were rated as more valuable. In contrast, responses with more role modeling were judged less valuable, although the effect was only marginally significant (β = −.10, t = −1.87, p = .06). When we probed for interactions between request type and response function on external perceived value of the response, we found no significant results (p = ns). Again, response functions appear to play similar roles across the different types of requests.

Table 3 shows that responses with general information, encouragement, and guidance were valued by both the original requester and outside viewers, though some of the effects were marginally significant for requesters’ judgments. The correspondence between the types of value judges is to be expected, because their judgments are moderately correlated. The associations were weaker for the requesters’ value judgments because regression analyses lose power with binary outcomes [50]. Since the requesters’ judgments rely on a single judge, they are less reliable than the MTurkers’ judgments, which were based on the average of six judges. Appendix B contrasts examples for high-low response scores.

Guidance on how to advise. New contributors can benefit from site-specific advice to improve their responses. This advice could be as simple a static document describing the type of responses that requesters find most valuable. An alternative might be just-in-time mentoring in which responders receive feedback from other members of the community. This approach has already shown promise: past work has shown that redirecting people writing questions in StackOverflow to an on-site help room staffed by experienced members of the community can increase the quality of their questions by 50% [18]. The Wikipedia Teahouse is a analogous program to help new editors of Wikipedia articles [41].

Intelligent response critic. Considering the different effects of what is valued in responses, future research can also explore the effectiveness of an natural language processing (NLP)-driven critic for increasing the quality of responses. An intelligent response critic could apply NLP to analyze the content of a response as it is being constructed, and provide guidance to the author. For example, the critic could inform the responder that their response did not use action-oriented language, and suggest words or phrases to encourage the requester to act on the problem.