GO: Debugging Each Service While It Is Running In Its Own Docker Container

Part 3 of 3 – GO: API Dialing GRPC Endpoints – Debug Each Service While It Is Running In Its Own Docker Container

The purpose of this article is to demonstrate how to setup a development environment that is containerized and has multiple microservices in a mono-repo that is all debugable with breakpoints at each point of the call chain from the HTTP Request, to the GRPC Request, the GRPC Response, and the HTTP Response.

3-Part Overview Review

In this overview we will be creating 3 microservices:

An HTTP API
A Strings Service

A Numbers Service

We will be defining the Strings and Numbers service through Protobuff files and then generate server code. The HTTP API will implement the necessary client side code to communicate with these GRPC services.

We want to setup the Strings Service to convert an input string to a fully Upper-Case or fully Lower-Case string.

We want to setup the Numbers service to accept two numbers and then Add them or Subtract them.

We want the API to have the following HTTP endpoints and then communicate with the Strings or Numbers service as appropriate over GRPC and then return the GRPC response as the HTTP Response body:

api/StringsService/MakeUpperCase
api/StringsService/MakeLowerCase
api/NumbersService/AddTwoNumbers

api/NumbersService/SubtractTwoNumbers

We also want to be able to debug the entire operation from the HTTP Request, to the GRPC Request, to the Service Operations, to the GRPC Response, and the final HTTP Response.

Show me the code: https://github.com/woodman231/api_dialing_grpc

Part 1: Setting Up The Development Container And Building The Services — Link
Part 2: Running Each Service In Its Own Docker Container — Link
Part 3: Debug Each Service While It Is Running In Its Own Docker Container

Part 3 – Debug Each Service While It Is Running In Its Own Docker Container

Thanks to DLV, we can have remote debugging. Since services are supposed to run in their own containers in production, we might as well debug them on our computers in the same way. It is a little tricky to get set up due to some security features we have to manipulate on our local containers, but that is not so bad.

Thus far when the services are being built via the command we gave it via docker it is building with the default settings and as such is what is known as an optimized and compiled binary for the OS that it was built under. We also deleted the source code from the server after the build was complete. These builds are not able to be debugged and therefore we need to both build and run our services differently than we have been building them. We will demonstrate using different arguments to the build command, installing dlv in to the container and telling dlv to run our service as well as tell visual studio code how to map the source files on the remote container to the source files in our development container.

Debug the strings_service Service

To begin we will need a new Dockerfile which will compile our source code to a binary that is able to be debugged. To the root of the project i.e /workspaces/api_dialing_grpc add a new file called: “StringsServiceDebugDockerfile”. Give it the following code:

# syntax=docker/dockerfile:1

FROM golang:1.18

RUN go install github.com/go-delve/delve/cmd/dlv@latest

WORKDIR /app
COPY protos/ protos/
COPY strings_service/ strings_service/

WORKDIR /app/strings_service

RUN go build -gcflags="all=-N -l"

EXPOSE 50051 40001

CMD ["/go/bin/dlv", "--listen=:40001", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "./strings_service"]

A couple of important differences between this file and the StringsServiceRunDockerfile. In this case we are installing go-delve package to the container. We are still copying the protos and strings_service folders to the /app folder we are then building the stings_service using the -gcflags=”all=-N -l” which is essentially telling it not to optimize the build at all for run time, rather it will be built optimized for debugging. Furthermore we are exposing both port 50051 and 40001. We then tell dlv to run our built binary and listen to connections on port 40001. We are also leaving the source code in the container instead of deleting it as we had before, and since we ran dlv from the /app/strings_service working directory it will assume that the source files are in the directory we executed dlv from.

Next, we need to tell Visual Studio Code how to build, run, and connect to this container for debugging.

Add the following tasks to /.vscode/tasks.json

        {
            "label": "Build the strings_service Debug Container",
            "type": "shell",
            "command": [
                "docker",
                "build",
                "-t strings_service",
                "-f StringsServiceDebugDockerfile",
                "."
            ],
            "options": {
                "cwd": "${workspaceFolder}"
            }
        },
        {
            "label": "Debug the strings_service Container",
            "type": "shell",
            "command": [
                "docker",
                "run",
                "-it",
                "-d",
                "-p 50051:50051",
                "-p 40001:40001",
                "--name=strings_service",
                "--network=api_dialing_grpc",
                "--hostname=strings_server",
                "--cap-add=SYS_PTRACE",
                "--security-opt=\"seccomp=unconfined\"",
                "strings_service"
            ],
            "options": {
                "cwd": "${workspaceFolder}"
            }            
        },
        {
            "label": "Build and Debug the strings_service Container",
            "dependsOn": [
                "Build the strings_service Debug Container",
                "Debug the strings_service Container"
            ],
            "dependsOrder": "sequence"
        }

Notice the different between the “Debug the strings_service Container” that we just created and the “Run the strings_service Container” task that we created earlier. We now have parameters for cap-add and security-opt. These are necessary parameters to use in order for the container to accept connections from Visual Studio Code to the DLV service.

Do the CTRL+SHIFT+P to Run a Task and Run the “Build and Debug the strings_sevice Container”.

Your containers area should look like this:

Do a right-click on the strings_service and select View Logs. The logs should look something like this:

This statement here saying “API server listening at: [::}40001” is not referring to our API service, rather it is referring to the dlv API service awaiting connections. After a debugger has attached to the server, then it will execute the command we told dlv to execute as per the Dockerfile that we just built as the CMD. We just did this to confirm that the dlv server is running in the container. We will come back to this but for now remove the current strings_service individual container.

Create a new file in the .vscode folder called “launch.json”. Give it the following code

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Attach to strings_service Debug Container",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/app/strings_service",
            "cwd": "${workspaceFolder}/strings_service",            
            "host": "127.0.0.1",
            "port": 40001,
            "trace": "verbose",
            "preLaunchTask": "Build and Debug the strings_service Container",
            "postDebugTask": "Remove the strings_service Container"
        }
    ]
}

What we have here is a Debug Configuration that will attach to the strings_service Debug Container. We have told Visual Studio Code that this is a go type of request that will attach to a remotely running service. It also tells Visual Studio Code to use the “Build and Debug the strings_service Container” task before trying to attach to this. Furthermore, pay attention to that “remotePath”, “cwd”, and the final “WORKDIR” of the StringsServiceDebugDockerfile. It is very important that these settings match up and that the files are in the same folder structure on the remote path as in your Visual Studio Code path, or else DLV and Visual Studio Code will not be able to create break points. In addition once the debugger has been stopped it will remove the strings_service container.

If you click on the Run and Debug button within your Visual Studio you will now have this “Attach to strings_Service Debug Container” as an option to use. Select that option and click on the green play button.

(If you have problem at this point a possible cause is that you did not remove the running strings_service on the Docker tab of Visual Studio Code.)

Once everything is done it should show you a “Debug Console” similar to this:

Furthermore, if you go to the Docker tab and do a right-click and View Logs on the strings_service that is running your terminal should look something like this.

This tells us that DLV accepted the connection from Visual Studio Code and started our strings_service which is now listening on port 50051. Use our Postman collection and the gRPC endpoints that we made to confirm that it is running.

As you do it should log it to the strings_service container log that you are viewing right now.

You will also notice that this bar is in your Visual Studio Code which is great because it tells us that a debugger is attached

In Visual Studio Code open the /strings_service/main.go file. Go to line 27 and press F9 to create a breakpoint at that line.

Use the MakeUpperCase gRPC Request that we built in Postman earlier.

As you do Visual Studio Code should turn orange in your task bar indicating that it needs attention.

Click on it and you should be able to get allot of debug information on your screen

Inspect the in parameter and the InputString it should be the same string as your input_string in the Postman gRPC request

Press F10 to step through the program, or press F5 again when you are ready for the debugger to just complete and allow the request to complete.

When you are done testing click on the Disconnect button in the Debug toolbar.

Because we told the postDebugTask to remove the container the container should also be removed and the service no longer works.

You can also return to line 27 in /strings_service/main.go and press F9 if you no longer want the breakpoint. But I am going to leave mine there for now.

Debug the numbers_service Service

Debugging the numbers_service will be very much like debugging the strings_service.

In the root of the application i.e /workspaces/api_dialing_grpc create a new file called “NumbersServiceDebugDockerfile”. Give it the following code:

# syntax=docker/dockerfile:1

FROM golang:1.18

RUN go install github.com/go-delve/delve/cmd/dlv@latest

WORKDIR /app
COPY protos/ protos/
COPY numbers_service/ numbers_service/

WORKDIR /app/numbers_service

RUN go build -gcflags="all=-N -l"

EXPOSE 50052 40002

CMD ["/go/bin/dlv", "--listen=:40002", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "./numbers_service"]

Once again, we are installing dlv, building our code to be optimized for debugging, exposing ports 50052 and 40002 (so that we can run the services and debuggers at the same time with the strings_service and api service).

Add the following tasks to /.vscode/tasks.json

        {
            "label": "Build the numbers_service Debug Container",
            "type": "shell",
            "command": [
                "docker",
                "build",
                "-t numbers_service",
                "-f NumbersServiceDebugDockerfile",
                "."
            ],
            "options": {
                "cwd": "${workspaceFolder}"
            }
        },
        {
            "label": "Debug the numbers_service Container",
            "type": "shell",
            "command": [                
                "docker",
                "run",
                "-it",
                "-d",
                "-p 50052:50052",
                "-p 40002:40002",
                "--name=numbers_service",
                "--network=api_dialing_grpc",
                "--hostname=numbers_server",
                "--cap-add=SYS_PTRACE",
                "--security-opt=\"seccomp=unconfined\"",
                "numbers_service"                
            ],
            "options": {
                "cwd": "${workspaceFolder}"
            }
        },
        {
            "label": "Build and Debug the numbers_service Container",
            "dependsOn": [
                "Build the numbers_service Debug Container",
                "Debug the numbers_service Container"
            ],
            "dependsOrder": "sequence"
        }

Add the following configuration to /.vscode/launch.json

        {
            "name": "Attach to numbers_service Debug Container",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/app/numbers_service",
            "cwd": "${workspaceFolder}/numbers_service",            
            "host": "127.0.0.1",
            "port": 40002,
            "trace": "verbose",
            "preLaunchTask": "Build and Debug the numbers_service Container",
            "postDebugTask": "Remove the numbers_service Container"
        }

Go to the Debug tab of Visual Studio Code, select the “Attach to numbers_service Debug Container” and then press the green arrow.

Put a line break in /numbers_service/main.go @ line 26 by pressing F9. Use the Postman collection to invoke a request to the AddTowNumbers method. Once again you will have Visual Studio come up orange. Press F10 / F5 to continue through the debugging and inspect the information that is on your screen before hand.

Press the disconnect button when your done doing the testing and debugging.

You can test both the strings_service and numbers_service at the same time by using the drop down in the Debug menu and pressing the green button each time you have a new item selected in the drop down.

When they are both running your Debug bar will look like this and you will need to disconnect from each.

Using the MakeUpperCase and AddTwoNumbers via Postman will cause both break points to occur while both of these debuggers are running.

Debug the API / HTTP Service

Debugging the API / HTTP Service will be much like we did for the others. Of course we have a dependency on the others with this but for now we will just focus on getting it running and hitting our /api/hello endpoint.

Create a new file called “ApiDebugDockerfile”. Give it the following code:

# syntax=docker/dockerfile:1

FROM golang:1.18

RUN go install github.com/go-delve/delve/cmd/dlv@latest

WORKDIR /app
COPY protos/ protos/
COPY api/ api/

WORKDIR /app/api

RUN go build -gcflags="all=-N -l"

EXPOSE 8080 40000

CMD ["/go/bin/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "./api", "--", "--strings_server_host=strings_server", "--numbers_server_host=numbers_server"]

That is quite the long “CMD” and that is because as you might recall in the run file we specify the strings_service_host name and numbers_server_host name well luckily with DLV we are still able to provide those parameters after the executable by using the — parameter that tells DLV that all parameters after — are to be passed to our GO application.

Add the following tasks to /.vscode/tasks.json

        {
            "label": "Build the api Debug Container",
            "type": "shell",
            "command": [
                "docker",
                "build",
                "-t api",
                "-f ApiDebugDockerfile",
                "."
            ],
            "options": {
                "cwd": "${workspaceFolder}"
            }
        },
        {
            "label": "Debug the api Container",
            "type": "shell",
            "command":[
                "docker",
                "run",
                "-it",
                "-d",
                "-p 8080:8080",
                "-p 40000:40000",
                "--name=api",
                "--network=api_dialing_grpc",
                "--hostname=api",
                "--cap-add=SYS_PTRACE",
                "--security-opt=\"seccomp=unconfined\"",
                "api"                                
            ],
            "options": {
                "cwd": "${workspaceFolder}"
            }
        },
        {
            "label": "Build and Debug the api Container",
            "dependsOn": [
                "Build the api Debug Container",
                "Debug the api Container"
            ],
            "dependsOrder": "sequence"
        }

Add the following configuration to launch.json

        {
            "name": "Attach to api Debug Container",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/app/api",
            "cwd": "${workspaceFolder}/api",            
            "host": "127.0.0.1",
            "port": 40000,
            "trace": "verbose",
            "preLaunchTask": "Build and Debug the api Container",
            "postDebugTask": "Remove the api Container"
        }

At this point when you go to the Debug area of Visual Studio Code you will have all three services available as something that you can debug. Remember you can run each of them all at the same time after selecting one and hitting the green run button.

If you put break points in lines 81 and 119 in /api/main.go you should get a break point every time you use an HTTP Request in Postman. You will also hit the break point in the strings or numbers service if the code executes as part of the HTTP Request. This allows you to be able to debug the request from the beging of the HTTP Request, the Beginning of the gRPC request, the end of the gRPC Response and the end of the HTTP Response despite the fact that there are three different services running on three different containers, all debuggers are in one IDE with this setup.

Stop the services when you are done testing this sort of connectivity.

Use Docker Compose to Debug All 3 Services

Of course, selecting these debuggers and running them over and over again could be tedious. However, they are also useful if you are only debugging one service at a time. To get all three services up and running in debug mode and our Visual Studio Code connected to all 3 debuggers with a single click this is how we do it.

Create a new file in the root of the application called ” docker-debug-compose.yml”.

version: '3'
services:
  strings:
    build:
      dockerfile: StringsServiceDebugDockerfile
    hostname: strings_server
    security_opt:
      - "seccomp:unconfined"
    cap_add:
        - SYS_PTRACE    
    ports:
      - 50051:50051
      - 40001:40001
  numbers:
    build:
      dockerfile: NumbersServiceDebugDockerfile
    hostname: numbers_server
    security_opt:
      - "seccomp:unconfined"
    cap_add:
        - SYS_PTRACE    
    ports:
      - 50052:50052
      - 40002:40002
  api:
    build:
      dockerfile: ApiDebugDockerfile
    security_opt:
      - "seccomp:unconfined"
    cap_add:
        - SYS_PTRACE
    ports:
      - 8080:8080
      - 40000:40000
    depends_on:
      - strings
      - numbers

This is very similar to our docker-run-compose.yml file except that we have added the security_opt and cap_add parameters to the file that we used when we did “docker run…” manually. This ensures that the security changes to allow the debuggers to work will be automatically placed when this compose is built.

Add the following tasks to tasks.json

        {
            "label": "Build All 3 Debug Containers - Compose",
            "type": "shell",
            "command": [
                "docker",
                "compose",
                "-f docker-debug-compose.yml",
                "build"
            ],
            "options": {
                "cwd": "${workspaceFolder}"
            }
        },
        {
            "label": "Debug All 3 Containers - Compose",
            "type": "shell",
            "command":[
                "docker",
                "compose",
                "-f docker-debug-compose.yml",
                "up",
                "-d"
            ],
            "options": {
                "cwd": "${workspaceFolder}"
            }
        },
        {
            "label": "Build and Debug All 3 Containers - Compose",
            "dependsOn":[
                "Build All 3 Debug Containers - Compose",
                "Debug All 3 Containers - Compose"
            ],
            "dependsOrder": "sequence"
        },
        {
            "label": "Remove All 3 Debug Containers - Compose",
            "type": "shell",
            "command": [
                "docker",
                "compose",
                "-f docker-debug-compose.yml",
                "down"
            ],
            "options": {
                "cwd": "${workspaceFolder}"
            }
        }

Add the following configurations to launch.json

        {
            "name": "Attach to composed strings_service Debug Container",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/app/strings_service",
            "cwd": "${workspaceFolder}/strings_service",            
            "host": "127.0.0.1",
            "port": 40001,
            "trace": "verbose"
        },
        {
            "name": "Attach to composed numbers_service Debug Container",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/app/numbers_service",
            "cwd": "${workspaceFolder}/numbers_service",            
            "host": "127.0.0.1",
            "port": 40002,
            "trace": "verbose"            
        },
        {
            "name": "Attach to composed api Debug Container",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/app/api",
            "cwd": "${workspaceFolder}/api",            
            "host": "127.0.0.1",
            "port": 40000,
            "trace": "verbose",            
            "postDebugTask": "Remove All 3 Debug Containers - Compose"
        }

Then add a new “coumpounds” key with the following code:

    "compounds": [
        {
            "name": "Build and attach to composed Debug Containers",
            "configurations": [
                "Attach to composed strings_service Debug Container",
                "Attach to composed numbers_service Debug Container",
                "Attach to composed api Debug Container"
            ],
            "preLaunchTask": "Build and Debug All 3 Containers - Compose",
            "stopAll": true
        }
    ]

You will notice that the new configurations we added are very similar to the other one’s except this time we did not add prelaunch tasks to them. Furthermore only the api has a postDebugTask. The reason for this is because the compound that we made only needs to build the composer, and not each individual container. In the compound there is also no such thing as a “postDebugTask” therefore just one of the services needs to call the compose down when the compound debugger is shut down.

In the Debug Area you will see that you have something similar to this now. It’s a little bit of a bummer to see these with the composed in it since you will only be using the compound one, but that just is what it is at this time. So really when you use this project you will most likely be using just the “Build and Debug All 3 Containers – Compose” configuration every time you make a change to the service code.

The non-composed ones are still able to be used if you just want to debug a single microservice without compiling all of them.

Retrospective

The world of microservices are meant to be one code repository for one service. This architecture is more of a mono repo with all of the services. There are some pros and cons to consider for this sort of a setup. On the one hand while working on these as a team it will be pretty clear how changes in one service could impact changes in another. On the other hand if there are some major changes needed just on one service and not the others then the code base could get stale quickly from which the developer has checked out their branch from.

A couple of things that we could have done differently include having each proto file in each service instead of having a single point / package for the protos. This would certainly make the separation of concerns more clear, but might make the orchestrator / api a little bit more difficult to manage since the client and server compiled protofiles usually need to be changed at the same time.

Regardless this is still a great example of spinning up microservices in to their own containers, running and debugging them and experiencing the orchestration in order to keep a grip on what is going on in the big picture.

Here is a challenge for you if you are up to it: Can you add a MultipleTwoNumbers method to the numbers_service service?

Conclusion

Microservices are a great way to separate processing power and later coordinate services all together to accomplish the greater goal. I hope that you enjoyed this article.

Show me the code: https://github.com/woodman231/api_dialing_grpc

About Intertech

Intertech is a Software Development Consulting Firm that provides single and multiple turnkey software development teams, available on your schedule and configured to achieve success as defined by your requirements independently or in co-development with your team. Intertech teams combine proven full-stack, DevOps, Agile-experienced lead consultants with Delivery Management, User Experience, Software Development, and QA experts in Business Process Automation (BPA), Microservices, Client- and Server-Side Web Frameworks of multiple technologies, Custom Portal and Dashboard development, Cloud Integration and Migration (Azure and AWS), and so much more. Each Intertech employee leads with the soft skills necessary to explain complex concepts to stakeholders and team members alike and makes your business more efficient, your data more valuable, and your team better. In addition, Intertech is a trusted partner of more than 4000 satisfied customers and has a 99.70% “would recommend” rating.

+651.288.7000

info@intertech.com

Request A Quote

GO: API Dialing GRPC Endpoints (3 of 3-Parts) – Debug Each Service While It Is Running In Its Own Docker Container

Part 3 of 3 – GO: API Dialing GRPC Endpoints – Debug Each Service While It Is Running In Its Own Docker Container

3-Part Overview Review

Part 3 – Debug Each Service While It Is Running In Its Own Docker Container

Debug the strings_service Service

Debug the numbers_service Service

Debug the API / HTTP Service

Use Docker Compose to Debug All 3 Services

Retrospective

Conclusion

About Intertech

info@intertech.com

651. 288. 7000

Consulting Services

Education Services

About Us

Careers

GO: API Dialing GRPC Endpoints (3 of 3-Parts) – Debug Each Service While It Is Running In Its Own Docker Container

Part 3 of 3 – GO: API Dialing GRPC Endpoints – Debug Each Service While It Is Running In Its Own Docker Container

3-Part Overview Review

Part 3 – Debug Each Service While It Is Running In Its Own Docker Container

Debug the strings_service Service

Debug the numbers_service Service

Debug the API / HTTP Service

Use Docker Compose to Debug All 3 Services

Retrospective

Conclusion

About Intertech

We Use Cookies From Time to Time