GrimoireLab

Join us!

Community Health Analytics for Open Source Software

GrimoireLab is one of CHAOSS Software founding projects

Free, libre, open source tools for
software development analytics

MSR & GrimoireLab

Are you researching in the area of mining software repositories or analyzing data about software development?

Consider joining the GrimoireLab MSR Virtual Hackathon. You will receive training & assistance on how to use GrimoireLab, discussion of your research project, short paper and hopefully a new tool in your toolbox.

The Mining Software Repositories (MSR) conference is the premier conference for data science, machine learning, and artificial intelligence in software engineering.

More information: Mining Software Repositories Conference - 2022

msr

How it works

The aim of this project is to provide an open source toolkit for:

  1. Automatic and incremental data gathering from almost any tool (data source) related with contributing to Open Source development (source code management, issue tracking systems, forums, etc.)
  2. Automatic gathered data enrichment, merging duplicated identities, adding additional information about contributors affiliation, calculation delays, geographical data, etc.
  3. Data consumption and visualization, allowing filtering by time range, project, repository, contributor, etc.
GrimoireLab architecture

Supported data sources

Source Code Management

Git, Baazar(*), Mercurial(*) (* by exporting to git)

Issues / Tasks Management

Bugzilla, GitHub Issues (including comments), GitHub project boards events(*), GitLab Issues, JIRA, Launchpad(*), Phabricator Maniphest, Pagure(*), Redmine (* only data gathering and enrichment)

Source Code Review

Gerrit, GitHub Pull Requests (including comments), GitLab Merge Requests

Mailing lists / Forums

Askbot, Discourse, Gmane, Groups.io, Hyperkitty, mbox files, NNTP, Pipermail, Stack Exchange (and Stack Overflow)

Continuous Integration

Jenkins

Synchronous Communication

Gitter, IRC (Supybot), Mattermost, Slack, Telegram

Wikis

Confluence, MediaWiki

Meetings Management

Meetup

Others

DockerHub, GitHub repository stats (stars, watchers, forks), Google hits, RSS, Twitter

Components

The platform provides the following components:

Perceval

Sir Perceval goes on the quest to retrieve and gather data from git, GitHub, Bugzilla, JIRA, Gerrit, mbox, pipermail, StackExchange, Discourse, etc.

Check Perceval repository

Graal

Graal leverages on the Git backend of Perceval and enhances it to set up ad-hoc source code analysis. Thus, it fetches the commits from a Git repository and provides a mechanism to plug third party tools/libraries focused on source code analysis.

Check Graal repository

Arthur

King Arthur commands his loyal knight Sir Perceval mananging the tasks to retrieve data for analysis. It manages data incremental update, parallel downloading, etc.

Check Arthur repository

GrimoireELK

Playground for testing the whole set of tools as a platform for software development analytics. It's is a prototype of the Grimoire Open Development Analytics platform.

Check GrimoireELK repository

Ceres

Ceres is a library that aims at dealing with data in general, and software development data in particular. The initial goal of Ceres is to parse information in several ways from the Perceval tool in the GrimoireLab project.

Check Ceres repository

Sorting Hat

Tool to manage people identities information, to merge multiple person identities across different data sources, enrich profile information with affiliation information, etc.

Check Sorting Hat repository

HatStall

Hatstall is a web interface for SortingHat databases developed mainly with Django.

Check HatStall repository

KiBiter

Custom fork of Kibana to work on new ideas for metrics & data visualization to be used by GrimoireLab Panels.

Check KiBiter repository

Sigils

Set of pre-defined widgets and dashboard templates to visualize Elasticsearch indexes generated by GrimoireELK with KiBiter.

Check Sigils repository

Kidash

Kidash is a tool for managing Kibana-related dashboards from the command line. The standard GrimoireLab dashboards are available in the Sigils repository.

Check Kidash repository

Manuscripts

Automatic generation of reports from the enriched indexes with items from Perceval data sources (git commits, github pull requests, bugzilla bugs ...) enriched using GrimoireELK.

Check Manuscripts repository

SirMordred

Docker container to help you deploying the Grimoire Lab using a set of configuration files.

Check SirMordred repository

Built with GrimoireLab

Some projects and services built using GrimoireLab Toolset

About

GrimoireLab is an evolution of the work done during +10 years by Bitergia, LibreSoft URJC research group and several contributors in Metrics Grimoire, VizGrimorie and GrimoireLib projects, to provide Open Development Analytics.

Since September 2017, GrimoireLab is part of The Linux Foundation CHAOSS Software community as one of the founding projects.

We truly free, libre, open source software and the power of the Community, so these set of tools are and always will be free, open source software. Feel free to contribute!