Sumologic appengine

In an enterprise world, with a SAS solution for your project one of the most important tools for debugging is log files. There are many solution for aggregating and storing log files. One of the SAS solutions is sumologic. One of the simpler ways to use sumologic is to install an agent and for it to upload all logs to the server.

Since appengine is a docker container, it would make send to create a base container for all appengines (be them java or python) so that we could upload logs to our sumo server.

Though the way that google has created its own base images, we need to separate ours to two types: java (gcr.io/google_appengine/openjdk8) & python(gcr.io/google-appengine/python), but the logic in the end is the same for both.

Note: Depending on the logger aggregator you are using, you might want to change the way you write your logs. For instance in sumo the best practice format is k=v, k=v (the sumo can extract all keys and allow you to select for value per key)

Sumologic installation

To install sumologic agent, we need to add the sumo.conf file, that will include the accesskey and accessid to the server (this will be fixed for all appengines). In addition we need to add the syncSources entry (/opt/SumoCollector/config/sumologic_sources.json) that will have the application specific configuration for sumologic. For more information see installation of agent.

Our base image will install the basic sumo agent installation with a startup script to run the sumo agent. All sumo configuration will be done in a sumologic_sources.json file that will be added in on the next layer of the docker stack (per application)

Sumologic Docker file layer

FROM gcr.io/google_appengine/openjdk8
VOLUME /tmp

ENV DOCKER_BASEDIR /opt/docker
ENV DOCKER_LOGDIR ${DOCKER_BASEDIR}/log
ENV DOCKER_SCRIPTS_DIR ${DOCKER_BASEDIR}/scripts
ENV SUMOLOGIC_BASEDIR /opt/SumoCollector
ENV SUMOLOGIC_CONFDIR ${SUMOLOGIC_BASEDIR}/config

RUN apt-get update && apt-get install -y curl nano

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# add sumo files
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

ENV SUMO_LATEST_DEBPKG_NAME /tmp/sumocollector.deb
ENV SUMO_INSTALLER_URL https://collectors.sumologic.com/rest/download/deb/64

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# INSTALL SUMO LOGIC COLLECTOR
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
RUN wget -O ${SUMO_LATEST_DEBPKG_NAME} ${SUMO_INSTALLER_URL} && \
if [ -f ${SUMO_LATEST_DEBPKG_NAME} ] ; then \
dpkg -i ${SUMO_LATEST_DEBPKG_NAME} ; \
chown -R root:root /opt/SumoCollector ; \
rm ${SUMO_LATEST_DEBPKG_NAME} ; \
fi

COPY sumo.conf /etc/sumo.conf.comp

RUN mkdir -p ${DOCKER_SCRIPTS_DIR}
COPY startup.sh ${DOCKER_SCRIPTS_DIR}/startup.sh
RUN chmod +x ${DOCKER_SCRIPTS_DIR}/startup.sh
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

RUN mkdir -p ${DOCKER_LOGDIR}
CMD tail -f /dev/null

As you can see, we install the latest version of the sumo agent, and add a default configuration file.

The startup script for this would be:

#!/bin/bash
set -x #echo on

exec > >(tee /tmp/start.log|logger -t user-data -s 2>/dev/console) 2>&1

HOSTNAME=`curl -s http://169.254.169.254/computeMetadata/v1beta1/instance/hostname`

SUMO_CONF_FILE=”/etc/sumo.conf”
SUMO_CONF_COMP_TEMPLATE=”/etc/sumo.conf.comp”

if [ -f ${SUMO_CONF_COMP_TEMPLATE} ] ; then
sed -i ‘s/ //g’ ${SUMO_CONF_COMP_TEMPLATE}
mv ${SUMO_CONF_COMP_TEMPLATE} ${SUMO_CONF_FILE}
fi

sed -i “s|@@INSTANCE_ID@@|${HOSTNAME}|g” /opt/SumoCollector/config/sumologic_sources.json

/etc/init.d/collector start

java -Djava.security.egd=file:/dev/./urandom -jar /app.jar

This scripts assumes that in the next level you will call your app app.jar, and that you will add the sumologic_sources.json file.

Application layer

So and example of the next level that inherits from this docker would be:

FROM gcr.io/ordinal-ember-163410/appengine-sumo-base
VOLUME /tmp
COPY migration-dataflow-appengine-1.0.0-SNAPSHOT.jar app.jar

ENV SUMOLOGIC_BASEDIR /opt/SumoCollector
ENV SUMOLOGIC_CONFDIR ${SUMOLOGIC_BASEDIR}/config
COPY sumologic_sources.json ${SUMOLOGIC_CONFDIR}/sumologic_sources.json

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# INSTALL gcp credentials
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
RUN mkdir -p /opt/cto/migration-dataflow/log && mkdir -p /opt/cto/migration-dataflow/gcp/log

ENV DOCKER_BASEDIR /opt/docker
ENV DOCKER_LOGDIR ${DOCKER_BASEDIR}/log
ENV DOCKER_SCRIPTS_DIR ${DOCKER_BASEDIR}/scripts

CMD bash -x ${DOCKER_SCRIPTS_DIR}/startup.sh | tee -a ${DOCKER_LOGDIR}/startup.log

Also we will need to add our own sumologic_sources.json so that the agent will know where to get the log files from, and the format. So an example could be:

{
“api.version”: “v1”,
“sources”: [
{
“name”: “docker-container-cto-migration-dataflow-log”,
“hostName”: “@@INSTANCE_ID@@”,
“category”: “prod/us/docker/container/microservices/cto/migration-dataflow/logs”,
“automaticDateParsing”: true,
“multilineProcessingEnabled”: true,
“useAutolineMatching”: false,
“forceTimeZone”: false,
“timeZone”: “Etc/UTC”,
“filters”: [],
“cutoffTimestamp”: 1464382800000,
“encoding”: “UTF-8”,
“pathExpression”: “/opt/cto/migration-dataflow/log/*.log”,
“blacklist”: [],
“sourceType”: “LocalFile”
},
{
“name”: “docker-container-bootstrap-log”,
“hostName”: “@@INSTANCE_ID@@”,
“category”: “prod/us/docker/container/bootstrap/logs”,
“automaticDateParsing”: true,
“multilineProcessingEnabled”: false,
“useAutolineMatching”: false,
“forceTimeZone”: false,
“timeZone”: “Etc/UTC”,
“filters”: [],
“cutoffTimestamp”: 1464382800000,
“encoding”: “UTF-8”,
“pathExpression”: “/opt/docker/log/*.log”,
“blacklist”: [],
“sourceType”: “LocalFile”
}
]
}

Summary

If you have a lot of app-engines that are part of your deployment, and would like to get the logs into sumologic you can easily do this using a base image with the basic sumologic agent install, and then only have the configuration on each app-engine docker.

Advertisements