Access Prometheus queries using the Prometheus HTTP API

In the last long blog post we covered the topic Monitor your custom operator with Prometheus. That means we did a setup of a Prometheus operator and we created a Prometheus service instance. In our operator we registered an example counter called goobers_total at the Prometheus server to monitor the invocations for our controller inside the operator application.

Now we want to access the counter information goobers_total by using the  Prometheus HTTP API from a local Golang application.

The following simplified diagram shows the dependencies. Prometheus gets the counter values over the /metrics endpoint and we will use the Prometheus HTTP API /api/v1/query?query=goobers_total to extract the goobers_total counter values from Prometheus.

The GitHub project Go Access Prometheus API contains the example source code related to that blog post.

That blog post is structured in following sections:

1. Basic implementation preparation flow
2. Basic programming steps
3. Run the example
4. Summary
5. Additional resources

1. Basic implementation preparation flow

• Examine the Prometheus HTTP API request and the HTTP response result. The following JSON shows an example HTTP response result. That example JSON is also available in the GitHub project.

{"status":"success",
 "data": {"resultType":"vector",
          "result":[{"metric": {"__name__":"goobers_total",
                                  "container":"kube-rbac-proxy",
                                  "endpoint":"https",
                                  "instance":"172.17.88.145:8443",
                                  "job":"frontendoperator-controller-manager-metrics-service",
                                  "namespace":"frontendoperator-system",
                                  "pod":"frontendoperator-controller-manager-6f7b64bf8f-2cv5p",
                                  "service":"frontendoperator-controller-manager-metrics-service"},
                                  "value":[1649681715.685,"7"]
                                }
                    ]
        }
}

• Reflect the given JSON format in the Golang programming language by using type struct definitions for later unmarshaling the JSON. Therefor the Visual Studio code extension Json to Go is useful. We find the struct definitions for the example JSON in the GitHub project.
• We need to know how to handle interfaces in combination with map definitions in the Go language for various datatypes. Therefor the stackoverflow entry How can I type an array of mixed types in golang? is useful and helps to understand how that will be implemented, because the HTTP response result in the JSON format contains different nested types.

2. Basic programming steps¶

Each step contains the link to the line in the source code inside the GitHub repository.

1. Define the Prometheus HTTP API request
2. Create HTTP request
3. Define HTTP header
4. Invoke HTTP request
5. Provide HTTP request response information
6. Verify the HTTP response request status
7. Get only the body from the HTTP response request
8. Convert HTTP response request body to JSON
9. Show values from the HTTP response request in JSON format
10. Decode the interface datatypes

The following source code shows the relevant sections for step 10. Therefor the stackoverflow entry How can I type an array of mixed types in golang? is useful and helps to understand how the interface will be used to implement the management of various datatypes in a nested array of a JSON.

...
type Result struct {
    ...
    Value  []interface{} `json:"value"`
}
...
if counter_float, ok := dat.Data.Result[0].Value[0].(float64); ok {
            // act on float
            fmt.Printf("-> counter_float: %f\n", counter_float)
        } else {
            // not float
            fmt.Printf("-> Error counter_float")
        }
...

3. Run the example

These are the steps you need to follow to run the example on your local machine.

Note: You should have followed the steps in Monitor your custom operator with Prometheus before you use that example code or you customize the Prometheus counter value to your needs.

Step 1: Clone the project

git clone https://github.com/thomassuedbroecker/go_access_prometheus_api.git

Step 2: Verify the that “go.mod” file exists

cd go_access_prometheus_api
ls

Step 3: Set the environment variable

export PROMETHEUS_URL=http://YOUR_HOST:YOUR_PORT

Step 5: Execute the go program

go run  ./main.go

• Example output:

*******************************
-> Use Prometheus API to get counter values
Prometheus REST API request: http://MY_HOST:MY_PORT/api/v1/query?query=goobers_total
Request Host: MY_HOST:MY_PORT
resp.Header: map[Content-Type:[application/json] Date:[Tue, XX Apr 2022 XX:XX:XX GMT]]
resp.StatusCode: 200
-> Response bodyString: {"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"goobers_total","container":"kube-rbac-proxy","endpoint":"https","instance":"172.17.88.145:8443","job":"frontendoperator-controller-manager-metrics-service","namespace":"frontendoperator-system","pod":"frontendoperator-controller-manager-6f7b64bf8f-2cv5p","service":"frontendoperator-controller-manager-metrics-service"},"value":[1649770144.356,"10"]}]}}
-> dat.Data.ResultType: vector
-> dat.Data.Result[0].Metric.Name: goobers_total
-> counter_float: 1649770144.356000
-> counter_string: 10

4. Summary

We used the Prometheus HTTP API to get the counter information from the Prometheus server.

I also took a brief look at the Prometheus Go client library, that library mainly covers the topic: how to define a Prometheus client to provide a /metrics endpoint and data in the right format for Prometheus, so that Prometheus can gathering data.

The Prometheus Go client library also provides a Promethis API alpha version, which does say: “It allows you to write Go applications that query time series data from a Prometheus server. It is still in alpha stage“, that wasn’t for me the right option in my current situation.

But we saw that Prometheus HTTP API works, the only challenge for the implementation in Golang is the JSON return values with the various type combinations in arrays ( Example: string and float64 in one array: “value”:[1649681715.685,”7″]), which isn’t optimal supported in Golang ;-).

5. Additional resources

• Example (CustomMarshalJSON)
• Documentation Prometheus query API
• Prometheus Model: Metrics and Labels
• Json to Go

I hope this was useful to you and let’s see what’s next?

Greetings,

Thomas

#golang, #prometheus, #metrics, #restapi, #json