CORRECT-ICSE2016

Editor's Notes

1. Hello everyone. My name is Mohammad Masudur Rahman I am a 2nd year PhD student from University of Saskatchewan, Canada. Today, I am going to talk about code reviewer recommendation based on cross-project and technology experience. I work with Dr. Chanchal Roy. The other co-author of the paper is Jason Collins from Google, USA.
2. When I searched in the web, I found this. Obviously, code review is not a very good experience all the time  If you do not select appropriate reviewers, the review could be disastrous. One way to handle the frustration about the code review is choosing the appropriate reviewers for your code.
3. We already had a talk on code review. Anyway, just to recap, code review is a systematic process of source code that identifies defects and coding standard violation of the code. It helps in early bug detection—thus reduces cost. It also ensures code quality by maintaining the coding standards. Code review has also evolved. First, it was formal inspection which was time-consuming, slow and costly. Then came a less formal code review—peer code review. Now we do tool assisted code review—also called modern code review.
4. This is an example for code review at GitHub. Once a developer submits a pull request, a way to submitting changes at GitHub, The core developers/ reviewers can review the changes and provide their feedback like this. Our goal in this research is to identify appropriate code reviewers for such a pull request. Identifying such code reviewers is very important especially for the novice developers who do not know the skill set their fellow developers. It is also very essential for distributed development where the developers rarely meet face to face. Besides an earlier study suggest that without appropriate code reviewers the whole change submission could be 12 days late on average. However, identifying such reviewers is challenging since the skill is not obvious, and it would require massive mining of the revision history.
5. The earlier study analyze line change history of source code, file path similarity of source files and review comments. In short, they mostly considered the work experience of a candidate code reviewers within a single project only. However, some skills span across multiple projects such as working experience with specific API libraries or specialized technologies. Also in an industrial setting, a developer’s contribution scatter throughout different projects within the company codebase. We thus consider external libraries and APIs included in the changed code and suggest more appropriate code reviewers.
6. This is the outline of my today’s talk. We collect commercial projects and libraries from Vendasta codebase, a medium sized Canadian software company. Then we ask 3 research questions and conduct an exploratory study to answer those questions. Based on those findings, we then propose our recommendation technique—CORRECT. Then in the experiments, we experimented commercial projects, compare with the state-of-the-art and also experimented with Open source projects. Finally, we conclude the talk.
7. We ask these three research questions. In a commercial codebase, there are two types of projects– customer project and utility projects. The utility projects are also called libraries. We ask. How frequently do the commercial software projects reuse external libraries in their code? Does working experience on such libraries matter in code reviewer selection? That means does a reviewer with such experience get preference over the others? Does working experience with specialized technologies such as mapreduce, taskqueue matter in code reviewer selection?
8. This is connectivity graph of core projects and internal libraries from Vendasta codebase. We see the graph is pretty much connected, that means most of the libraries are used by most of the projects. For the study, we chose 10 projects and 10 internal libraries, and they are chosen based on certain restriction. Each project should have 750 closed pull requests, that means they should be pretty big, and most importantly pretty much active. Each internal library should be used at least 10 times on average by each of those projects. Each technology, I mean the specialized technology is should be used at least 5 times on average by each of those projects. We consider the Google App Engine libraries as the specialized technologies. We consider 10 of them.
9. This is the usage frequency of the selected libraries in the 10 project we selected for the study. We take the latest snapshot of each of the projects, analyze their source files, and look for imported libraries using an AST parser. We try to find out those 10 libraries mostly, and this is the box plot of their frequencies. We can see that vtest, vauth and vapi are the mostly used, which a kind of make sense especially vtest and vauth, they provide possibly the generic testing and authentication support. However, vtest has a large variance, that means some projects used it extensively whereas the others didn’t use it at all. The least used libraries are vlogs and vmonitor. So, this is the empirical frequencies.
10. We investigated the ratio of pull requests that used any of the selected libraries. We note that 30%-70% of all pull requests did that in different projects. We also investigated the percentage of the library authors who are later recommended as code reviewers for the projects referring to that library. We considered a developer as library author if he/she authored at least one pull request of the library. We note that almost 100% of authors are later recommended. This is a very interesting findings that suggest that library experience really matters.
11. We also calculated the empirical frequency of the ten specialized technologies in the selected Vendasta projects And this is the box plot. We can see that mapreduce is the champion technology here, and the rest are close competitors.
12. In case of the pull requests, 20%-60% pull requests used at least one of ten specialized technologies Mostly used by ARM, CS and VBC. So, specialized technologies are also used in our selected projects quite significantly.
13. So, here are the empirical findings from the exploratory studies we conducted. They suggest that library experience and specialized technology experience really matter. These are new findings, and we exploit them to develop the recommendation algorithm later.
14. Based on those exploratory findings, we propose CORRECT– Code reviewer recommendation based on cross-project and technology experience.
15. This is our recommendation algorithm. Once a new pull request R3 is created, we analyze its commits, then source files, and look for the libraries referred and the specialized technologies used. Thus, we get a library token list and a technology token list. We combined both lists, and this list can be considered as a summary of libraries and technologies for the new pull request. Now, we consider the latest 10 but closed pull requests, and collect their library and technology tokens. It should be noted that the past requests contain their code reviewers. Now, we estimate the similarity between the new and each of the past requests. We use cosine similarity score between their token list. We add that score to the corresponding code reviewers. This way, finally, we get a list of reviewers who got accumulated score for different past reviews. Then they are ranked and top reviewers are recommended. Thus, we use pull request similarity score to estimate the relevant expertise of code review.
16. Now, to be technically specific The state-of-the-art considers two pull requests relevant/similar if they share source code files or directories. On the other hand, we suggest that two pull requests are relevant/similar if they share the same external libraries and specialized technologies. That’s the major difference in methodology and our core technical contribution.
17. We performed two evaluations– one with Vendasta codebase, and the other with Open source codebase. From that experiments, we try to answer four research questions. Are library experience and technology experience useful proxies for code review skills? Can our technique outperform the state-of-the-art technique from the literature? Does it perform equally for closed source and open source projects? Does it show any bias to any particular platform?
18. We conducted experiments using 10 projects from GitHub codebase and 6 projects from open source domain. From Vendasta, we collected 13K pull requests, and from open source, we collect 4K pull requests. Gold reviewers are collected from the corresponding pull requests. Vendasta projects are python-based whereas the OS projects are written in python, Java and Ruby. We consider four performance metrics– accuracy, precision, recall, and reciprocal rank. In case of accuracy, if the recommendation contains at least one gold reviewer, we consider the recommendation is accurate.
19. This is how we answer the first RQ. We see that both library similarity and technology similarity are pretty good proxies for code review skills. Each of them provides over 90% top-5 accuracy. However, when we combine, we get the maximum—92% top-5 accuracy. The precision and recall are also greater than 80% which is highly promising according to relevant literature.
20. We then compare with the state-of-the-art –RevFinder. We found that our performance is significantly better than theirs. We get a p-value of 0.003 for top-5 accuracy with Mann-Whitney U tests. The median accuracy 95%. The median precision and median recall are between 85% to 90% In the case of individual projects, our technique also outperformed the state-of-the-art.
21. We also experimented using 6 Open source projects, and found 85% Top-5 accuracy. For the case of precision and recall, they are not significantly different from those with Vendasta projects. For example, with precision, we get a p-value of 0.239 which is greater than 0.05.
22. This slide shows how CORRECT performed with the projects from 3 programming platforms– Python, Java and Ruby. We also find quite similar performance for each of the platforms which is interesting. This shows that our findings with commercial projects are quite generalizable.
23. Now to summarize Code review could be unpleasant or unproductive without appropriate code reviewers. We fist motivated our technique using an exploratory study which suggested that --library experience and specialized technology experience really matter for code reviewer selection. Then we proposed our technique—CORRECT that learns from past review history and then recommends. We experimented using both commercial and open source projects, and compared with the state-of-the-art. The results clearly demonstrate the high potential of our technique.
24. That’s all I have to say today. Thanks for your time. Questions!!
25. There are a few threats to the validity of our findings. -- The dataset from VA codebase is a bit skewed. Most of the projects are medium and only one project is big. --Also the projects considered from open source domain is limited. --Also, the technique could be slower for big pull requests.