Dev Lake
What is Dev Lake?
Dev Lake is the one-stop solution that integrates, analyzes, and visualizes the development data throughout the software development life cycle (SDLC) for engineering teams.
Why choose Dev Lake?
- Supports various data sources and quickly growing
- Comprehensive dev metrics built-in
- Customizable visualizations and dashboard
- Easy-to-setup via docker
- Extensible plugin system to add your own data collectors
- Designed to process enterprise-scale data
Contents
| Section | Description | Link |
|---|---|---|
| Requirements | Underlying software used | View Section |
| Installation | Getting all the required files | View Section |
| Setup | Steps to get up and running | View Section |
| Core Usage | Using core lake features |
View Section |
| Plugin Usage | Links to specific plugin usage & details | View Section |
| Configuration | Local file config settings info | Link |
| Contributing | How to contribute to this repo | Link |
Requirements
How to run this application
NOTE: If you only plan to run the product, this is the only section you should need
- Clone this repository and
cdinto it - Configure settings for services & plugins with
cp config/docker.sample.js config/docker.jsand edit the newly created file - Start the service with
npm run compose-prod
- you can see the logs with
npm run compose-logs - you can stop all docker containers with
npm run compose-down-prod
- Send a post request to the service
curl -X POST "http://localhost:3001/" -H 'content-type: application/json' \
-d '{"jira":{"boardId": 29}, "gitlab": {"projectId": 24547305}}'
- Check the console logs for docker-compose to see when the logs stop collecting your data. This can take up to 30 minutes for large projects. (gitlab 10k+ commits or jira 10k+ issues)
- Navigate to Grafana Dashboard
https://localhost:3002(Username:admin, password:admin)
Installation
-
Clone this repository
git clone https://github.com/merico-dev/lake.git
-
Install dependencies with
npm i -
Configure local settings for services & plugins, see CONFIGURATION.md
Setup
-
From the root directory, run
npm run docker
NOTE: If you get an error like this: "Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /tmp/rabbitmq/etc/"
You can fix it by creating the directories in the terminal:
mkdir -p ./rabbitmq/logs/ ./rabbitmq/etc/ ./rabbitmq/data/ -
In another tab run
npm run all
-
Create a collection job to collect data. See that the:
- collection job was published
- lake plugin collection ran
- enrichment job was published
- lake plugin enrichment ran
This process will run through each lake plugin, collecting data from each
From Postman (or similar), send a request like (
branchis optional):POST http://localhost:3001/ { "jira": { "boardId": 8 }, "gitlab": { "projectId": 8967944, "branch": "<your-branch-name>", } }
Or, by using
curl# ee curl -X POST "http://localhost:3001/" -H 'content-type: application/json' \ -d '{"jira":{"boardId": 8}, "gitlab": {"projectId": 8967944}}' # small data set for test curl -X POST "http://localhost:3001/" -H 'content-type: application/json' \ -d '{"jira":{"boardId": 29}, "gitlab": {"projectId": 24547305}}'
-
Visualize data in Grafana dashboard
From here you can see existing data visualized from collected & enriched data
- Navigate to http://localhost:3002 (username:
admin, password:admin) - You can also create/modify existing/save dashboards to
lake - For more info: Provisioning a Dashboard
- Navigate to http://localhost:3002 (username:
Migrations
- Revert all current migrations
npx sequelize-cli db:migrate:undo:all - Run migration with
npx sequelize-cli db:migrate
Core Usage
| Section | Section Info |
|---|---|
| Collections | Create a Collection Job |
| Grafana | Logging In |
| Grafana | Provisioning a Dashboard |
Collections: Create a Collection Job
Details
- From the terminal, execute
npm run all - From Postman (or similar), send a request like:
POST http://localhost:3001/
{
"jira": {
"boardId": 8
},
"gitlab": {
"projectId": 8967944,
"branch": "<your-branch-name>", // branch is optional, we fetch Gitlab default branch if this arg is absent
}
}
Or, by using curl
# ee
curl -X POST "http://localhost:3001/" -H 'content-type: application/json' \
-d '{"jira":{"boardId": 8}, "gitlab": {"projectId": 8967944}}'
# small data set for test
curl -X POST "http://localhost:3001/" -H 'content-type: application/json' \
-d '{"jira":{"boardId": 29}, "gitlab": {"projectId": 24547305}}'
- See that the:
- collection job was published
- jira collection ran
- enrichment job was published
- jira enrichment ran
Grafana: Logging In
Details
Once the app is up and running, visithttp://localhost:3002 to view the Grafana dashboard.
Default login credentials are:
- Username:
admin - Password:
admin
Grafana: Provisioning a Dasboard
Details
To save a dashboard in the lake repo and load it:
- Create a dashboard in browser (visit
/dashboard/new, or use sidebar) - Save dashboard (in top right of screen)
- Go to dashboard settings (in top right of screen)
- Click on JSON Model in sidebar
- Copy code into a new
.jsonfile in/grafana/dashboards
Plugin Usage
| Section | Section Info | Docs |
|---|---|---|
| Jira | Metrics, Generating API Token, Find Project/Board ID | Link |
| Gitlab | Metrics, Generating API Token | Link |
3 May 18, 2022
4 May 3, 2022
14 Jan 3, 2023
1.2k Jan 7, 2023
20.6k Jan 4, 2023
4 Nov 13, 2022
331 Jan 4, 2023
11 Dec 21, 2022
7 Dec 20, 2022
50 Dec 14, 2022
261 Dec 30, 2022
2 Jan 29, 2022
12 Dec 3, 2022
5 May 23, 2022
20 Dec 29, 2022
97 Dec 28, 2022
6 Dec 23, 2022
3 Feb 22, 2022
37 Dec 22, 2022