[Bug][gitlab] The data in the commits table is different from the data in GitLab about incubator-devlake

[dosubot]

To address the discrepancies between the commits table and GitLab data, especially after merge operations, it's crucial to dive deeper into the specifics of your situation. Here's how we can proceed:

1. Share the Python Script: The script you used for verification could provide insights into how data is being fetched and compared. This will help in understanding the methodology behind your findings.

2. Identify Patterns: Knowing if these discrepancies follow any specific patterns, such as occurring more with larger commits or certain file types, can help in pinpointing the cause.

3. Provide Examples: Examples of commit hashes with their discrepancies in additions count between the commits table and GitLab would be invaluable for a detailed investigation.

4. Scope of the Issue: Clarifying whether this issue is isolated to specific repositories or is more widespread will aid in diagnosing the problem more effectively.

This information will be instrumental in uncovering the root cause of the discrepancies you're observing.

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[Startrekzky]

Thanks for reporting.

[klesh]

It appears the libgit2 library we're using has a different diff algorithm compared to GitLab. Unfortunately, this is likely an expected behavior rather than a bug and wouldn't be easily configurable.

[Startrekzky]

Hi, we checked the logic. The difference between additions from GitLab APIs and DevLake is because:

• DevLake doesn't use GitLab APIs, but the gitextractor plugin to collect commits
• The gitextractor plugin uses the libgit2 library to calculate the commits additions and deletions.
• The logic of calculating the additions in gogit and GitLab APIs are slightly different.

Thus, it's a problem that can not be addressed in DevLake for now. We can't switch gitextractor to GitLab APIs or GitHub APIs or Bitbucket APIs or Azure DevOps APIs to collect commits (Git) data, as it will increase the collection time by 10x times.