How to sniff HTTPS traffic without certificate manipulation?

Introduction

Recently, I was working with a third-party SDK written in Python. I had to find out which headers this SDK sends to the server. I could have used dedicated software like Charles or mimproxy, but I didn’t have much experience with manipulating certificates in Python, and someone else used SDK directly. Instead, I came up with something simpler.

The solution

Let’s say I have an SDK that sends requests to https://www.example.com. I can configure the SDK’s base URL.

I start with running ngrok - a reverse proxy that creates a secure tunnel to localhost:

ngrok http 8090

This script starts a tunnel that is publicly accessible through HTTPS, it points to my local machine on port 8090. The URL looks somewhat like: https://b2a1-93-174-30-35.ngrok-free.app

I pass this URL to the SDK.

Then I start Spring Cloud Gateway with the following configuration:

spring:
  cloud:
    gateway:
      routes:
        - id: all
          uri: https://www.example.com
          predicates:
            - Path=/**
server:
  port: 8090

This configuration starts server on port 8090 and forwards all incoming requests to https://www.example.com.

Then it’s possible to view requests and responses in the ngrok dashboard - http://127.0.0.1:4040/

The following diagram shows how the traffic is sniffed:

Conclusion

In this article, I showed how to sniff HTTPS traffic without manipulating certificates. So, if you don’t want or can’t manipulate certificates (but you can change the base URL), it can be a solution for you. The source code can be found on GitHub.