Allow the dashboard to show historic data of pipelines and jobs (maybe by add "id" label to the metrics)

[ErezArbell]

This is a great tool and it is easy to deploy and use. Thank you.

The Grafana dashboard in the example shows only the latest pipeline for the selected project/branch, and the latest job for the selected project/branch/job name.
I would like to see all the pipelines/jobs that were collected since I started collecting the data.
For example, in the pipeline dashboard, when I filter by project and branch, I would like to see in the table all the pipelines that Prometheus has data on them and match those project and branch.
This would be great since it will give us easier was to see the results in a better way than the GitLab GUI allow.
One use case is to use the jobs dashboard, filter by project, branch and job name, and see the history of a job, like when it started to fail. There can be many use cases for it.
As I wrote, the dashboards in the example only show the latest one for each set of labels (pipeline, ref).
I tried to manipulate the queries but could not make it show multiple results.

Do you have idea for how to change the dashboard to do it?

I have an idea for what is missing. I think that we can add a label "id" to the metrics:

• gitlab_ci_pipeline_status
• gitlab_ci_pipeline_id
• gitlab_ci_pipeline_job_status
• gitlab_ci_pipeline_job_id

the value of the label "id" will be the pipeline/job ID.
If these metrics will have the label "id" then the above will be possible by adding "id" to the "by ()" list in the queries.

This is similar to the fact that the metrics:

• gitlab_ci_pipeline_status
• gitlab_ci_pipeline_job_status

have the label "status".
Actually, I think that all the metrics can include the label "id" that will include the relevant id for the pipeline/job/MR etc.

I will try to do it but I am not familiar with the code so it will not be easy for me.
@mvisonneau , if you can add it is this way or another, it would be great and increase the usability of this project.

BTW, I think that this was the intention of the author of Pull Request 200 although there he added new metrics while it be better to have "id" label to the exiting metrics.

[ErezArbell]

I added 'pipeline_id' and 'job_id' labels to the relevant metrics and also 'status' label.
See PR 455
I was able to create a dashboard that will show more than the last pipeline/job. I sill have problem with the 'status' column.
It is probably due to my lack of experience with PromQL and Grafana, but also may be that I am trying to use those tools to goal there are not designed for.

I kept looking and couldn't find any other tool that can do that we want (show history of GitLab pipeline/job results with option to filter) so this way is the best we have.

[DivyaSethy]

i started exploring this tool, as I wanted the pipeline matrix, unfortunately I am able to pull the latest pipeline only of a project with one branch. I am able to pull the branches but only one pipeline per branch. I want to see the more pipeline or list of the pipelines per branch. if I change the time like " from now to past 7 days, it should show all the pipelines executed in past 7 days. but however, this is not happening.

I saw the code, which uses ref.LatestPipeline, maybe for that reason it is collecting the latest pipeline only.

can anyone help on this. @mvisonneau could you please look into this.

[ruanwenjun]

I added 'pipeline_id' and 'job_id' labels to the relevant metrics and also 'status' label. See PR 455 I was able to create a dashboard that will show more than the last pipeline/job. I sill have problem with the 'status' column. It is probably due to my lack of experience with PromQL and Grafana, but also may be that I am trying to use those tools to goal there are not designed for.

I kept looking and couldn't find any other tool that can do that we want (show history of GitLab pipeline/job results with option to filter) so this way is the best we have.

This will cause the metrics got high cardinality, I search some docs told prometheus cannot support high cardinality.