Troubleshooting
Something went wrong with my GameServer
If there is something going wrong with your GameServer, there are a few approaches to determining the cause:
Depending on what is happening, you may want to run kubectl describe <gameserver name>
to view the events
that are associated with that particular GameServer
resource. This can give you insight into the lifecycle of the
GameServer
and if anything has gone wrong.
For example, here we can see where the simple-udp example has been moved to the Unhealthy
state
due to a crash in the backing GameServer
Pod container’s binary.
root@6a71afd42291:/go/src/agones.dev/agones# kubectl describe gs simple-udp-zqppv
Name: simple-udp-zqppv
Namespace: default
Labels: <none>
Annotations: agones.dev/sdk-version: 1.0.0-dce1546
API Version: agones.dev/v1
Kind: GameServer
Metadata:
Creation Timestamp: 2019-08-16T21:25:44Z
Finalizers:
agones.dev
Generate Name: simple-udp-
Generation: 1
Resource Version: 1378575
Self Link: /apis/agones.dev/v1/namespaces/default/gameservers/simple-udp-zqppv
UID: 6818adc7-c06c-11e9-8dbd-42010a8a0109
Spec:
Container: simple-udp
Health:
Failure Threshold: 3
Initial Delay Seconds: 5
Period Seconds: 5
Ports:
Container Port: 7654
Host Port: 7058
Name: default
Port Policy: Dynamic
Protocol: UDP
Scheduling: Packed
Template:
Metadata:
Creation Timestamp: <nil>
Spec:
Containers:
Image: gcr.io/agones-images/udp-server:0.21
Name: simple-udp
Resources:
Limits:
Cpu: 20m
Memory: 32Mi
Requests:
Cpu: 20m
Memory: 32Mi
Status:
Address: 35.230.59.117
Node Name: gke-test-cluster-default-590db5e4-4s6r
Ports:
Name: default
Port: 7058
Reserved Until: <nil>
State: Unhealthy
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal PortAllocation 72s gameserver-controller Port allocated
Normal Creating 72s gameserver-controller Pod simple-udp-zqppv created
Normal Scheduled 72s gameserver-controller Address and port populated
Normal RequestReady 67s gameserver-sidecar SDK state change
Normal Ready 66s gameserver-controller SDK.Ready() complete
Warning Unhealthy 34s health-controller Issue with Gameserver pod
The backing Pod has the same name as the GameServer
- so it’s also worth looking at the
details and events for the Pod to see if there are any issues there, such as restarts due to binary crashes etc.
For example, you can see the restart count on the gcr.io/agones-images/udp-server:0.21 container
is set to 1
, due to the game server binary crash
root@6a71afd42291:/go/src/agones.dev/agones# kubectl describe pod simple-udp-zqppv
Name: simple-udp-zqppv
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: gke-test-cluster-default-590db5e4-4s6r/10.138.0.23
Start Time: Fri, 16 Aug 2019 21:25:44 +0000
Labels: agones.dev/gameserver=simple-udp-zqppv
agones.dev/role=gameserver
Annotations: agones.dev/container: simple-udp
agones.dev/sdk-version: 1.0.0-dce1546
cluster-autoscaler.kubernetes.io/safe-to-evict: false
Status: Running
IP: 10.48.1.80
Controlled By: GameServer/simple-udp-zqppv
Containers:
simple-udp:
Container ID: docker://69eacd03cc89b0636b78abe47926b02183ba84d18fa20649ca443f5232511661
Image: gcr.io/agones-images/udp-server:0.21
Image ID: docker-pullable://gcr.io/agones-images/udp-server@sha256:6a60eff5e68b88b5ce75ae98082d79cff36cda411a090f3495760e5c3b6c3575
Port: 7654/UDP
Host Port: 7058/UDP
State: Running
Started: Fri, 16 Aug 2019 21:26:22 +0000
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 16 Aug 2019 21:25:45 +0000
Finished: Fri, 16 Aug 2019 21:26:22 +0000
Ready: True
Restart Count: 1
Limits:
cpu: 20m
memory: 32Mi
Requests:
cpu: 20m
memory: 32Mi
Liveness: http-get http://:8080/gshealthz delay=5s timeout=1s period=5s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from empty (ro)
agones-gameserver-sidecar:
Container ID: docker://f3c475c34d26232e19b60be65b03bc6ce41931f4c37e00770d3ab4a36281d31c
Image: gcr.io/agones-mark/agones-sdk:1.0.0-dce1546
Image ID: docker-pullable://gcr.io/agones-mark/agones-sdk@sha256:4b5693e95ee3023a2b2e2099d102bb6bac58d4ce0ac472e58a09cee6d160cd19
Port: <none>
Host Port: <none>
State: Running
Started: Fri, 16 Aug 2019 21:25:48 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 30m
Liveness: http-get http://:8080/healthz delay=3s timeout=1s period=3s #success=1 #failure=3
Environment:
GAMESERVER_NAME: simple-udp-zqppv
POD_NAMESPACE: default (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from agones-sdk-token-vr6qq (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
empty:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
agones-sdk-token-vr6qq:
Type: Secret (a volume populated by a Secret)
SecretName: agones-sdk-token-vr6qq
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m32s default-scheduler Successfully assigned default/simple-udp-zqppv to gke-test-cluster-default-590db5e4-4s6r
Normal Pulling 2m31s kubelet, gke-test-cluster-default-590db5e4-4s6r pulling image "gcr.io/agones-mark/agones-sdk:1.0.0-dce1546"
Normal Started 2m28s kubelet, gke-test-cluster-default-590db5e4-4s6r Started container
Normal Pulled 2m28s kubelet, gke-test-cluster-default-590db5e4-4s6r Successfully pulled image "gcr.io/agones-mark/agones-sdk:1.0.0-dce1546"
Normal Created 2m28s kubelet, gke-test-cluster-default-590db5e4-4s6r Created container
Normal Created 114s (x2 over 2m31s) kubelet, gke-test-cluster-default-590db5e4-4s6r Created container
Normal Started 114s (x2 over 2m31s) kubelet, gke-test-cluster-default-590db5e4-4s6r Started container
Normal Pulled 114s (x2 over 2m31s) kubelet, gke-test-cluster-default-590db5e4-4s6r Container image "gcr.io/agones-images/udp-server:0.21" already present on machine
Finally, you can also get the logs of your GameServer
Pod
as well via kubectl logs <pod name> -c <game server container name>
, for example:
root@6a71afd42291:/go/src/agones.dev/agones# kubectl logs simple-udp-zqppv -c simple-udp
2019/08/16 21:26:23 Creating SDK instance
2019/08/16 21:26:24 Starting Health Ping
2019/08/16 21:26:24 Starting UDP server, listening on port 7654
2019/08/16 21:26:24 Marking this server as ready
The above commands will only give the most recent container’s logs (so we won’t get the previous crash), but
you can use kubectl logs --previous=true simple-udp-zqppv -c simple-udp
to get the previous instance of the containers logs, or
use your Kubernetes platform of choice’s logging aggregation tools to view the crash details.
For more tips and tricks, the Kubernetes Cheatsheet: Interactive with Pods also provides more troubleshooting techniques.
How do I see the logs for Agones?
If something is going wrong, and you want to see the logs for Agones, there are potentially two places you will want to check:
- The controller: assuming you installed Agones in the
agones-system
namespace, you will find that there is a single pod calledagones-controller-<hash>
(where hash is the unique code that Kubernetes generates) that exists there, that you can get the logs from. This is the main controller for Agones, and should be the first place to check when things go wrong.
- To get the logs from this controller run:
kubectl logs --namespace=agones-system agones-controller-<hash>
- To get the logs from this controller run:
- The SDK server sidecar: Agones runs a small gRPC + http server for the SDK in a container in the
same network namespace as the game server container to connect to via the SDK.
The logs from this SDK server are also useful for tracking down issues, especially if you are having trouble with a particularGameServer
.
- To find the
Pod
for theGameServer
look for the pod with a name that is prefixed with the name of the owningGameServer
. For example if you have aGameServer
namedsimple-udp
, it’s pod could potentially be namedsimple-udp-dnbwj
. - To get the logs from that
Pod
, we need to specify that we want the logs from theagones-gameserver-sidecar
container. To do that, run the following:
kubectl logs simple-udp-dnbwj -c agones-gameserver-sidecar
- To find the
Agones uses JSON structured logging, therefore errors will be visible through the "severity":"info"
key and value.
I uninstalled Agones before deleted all my GameServers
and now they won’t delete
Agones GameServers
use Finalizers
to manage garbage collection of the GameServers
. This means that if the Agones controller
doesn’t remove the finalizer for you (i.e. if it has been uninstalled), it can be tricky to remove them all.
Thankfully, if we create a patch to remove the finalizers from all GameServers, we can delete them with impunity.
A quick one liner to do this:
kubectl get gameserver -o name | xargs -n1 -P1 -I{} kubectl patch {} --type=merge -p '{"metadata": {"finalizers": []}}'
Once this is done, you can kubectl delete gs --all
and clean everything up (if it’s not gone already).
I’m getting Forbidden errors when trying to install Agones
Ensure that you are running Kubernetes 1.12 or later, which does not require any special clusterrolebindings to install Agones.
If you want to install Agones on an older version of Kubernetes, you need to create a clusterrolebinding to add your identity as a cluster admin, e.g.
# Kubernetes Engine
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user `gcloud config get-value account`
# Minikube
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin --serviceaccount=kube-system:default
On GKE, gcloud config get-value accounts
will return a lowercase email address, so if
you are using a CamelCase email, you may need to type it in manually.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.