at Social Point

Golang BCN - Apr '16

by @gonzaloserrano

Question everything


Who am i

Who are we

  • catalan startup based in Barcelona
  • casual games (mobile & FB)
  • ~275 employees
  • several million DAUs

Our games


This is based in our personal experience only.

Several million DAUS?

  • High traffic
  • Hundreds or thousands OPS
  • Scalabity
  • Availability
  • Performance: latency, throughput...

Backend Stack

  • PHP/Symfony2: stateless, request/response
  • Golang, Erlang/OTP: stateful, CSP/actor model
  • JS, Python, BASH, Groovy...
  • AWS: VPC/EC2, Redshift, RDS, DynamoDB, S3, SNS, SQS, EMR, Lambda, CodeDeploy, Gateway...
  • MySQL, Redis, Cassandra, Riak, ElasticSearch...

From stateless to stateful

  • Some game features have state or need soft-real time
  • Request/Response model is wasteful or not performant enough
  • Erlang/OTP, Elixir, Golang, Java/Scala/Akka, NodeJS, Rust, Haskell, Stackless Python and others provide:
    • better performance than PHP
    • stateful servers
    • real-time capabilities
    • concurrency model

But what about...

  • Learning curve
  • Performance limits
  • Ecosystem
  • Documentation
  • DX
  • Ops
  • Evolution

What we did 2 years ago

  • Bet for Erlang/OTP
  • Created a chat and some microservices
  • Found some issues:
    • lack of tutorials and good documentation
    • small library ecosystem
    • not easy to deploy
    • difficult to master:
      • functional programming
      • recursion
      • immutability
      • distributed systems
    • some of this issues are solved in Elixir

Pivot to Golang

  • simplicity & readability
    • small language
    • easy to understand and fix
  • easy but powerful concurrency model if you know Erlang
  • also not very difficult from PHP:
    • most of the the team has finished the go tour
    • ~15% code full time, 25% part time
    • ~35% has some go code in prod, and increasing
    • our frontend is doing Golang too! :-)

Pivot to Golang

  • very good documentation: go tour, godoc, wiki, tutorials...
  • big and active ecosystem: github libs (aws-sdk-go), blog posts, mailing list, slack...
  • tooling
  • big ones (Google and others) betting on Go

Pivot to Golang

  • we are pretty happy about the results
  • huge performance gains
  • fast development rate
  • small time to deploy
  • working in good DX: tools and frameworks

Lessons learnt

Automation: Makefile

  • There are many useful go commands, flags and tools
  • Automate its usage
  • Learn from the masters:
    • github.com/hashicorp/terraform/blob/master/Makefile
    • github.com/hashicorp/consul/blob/master/GNUmakefile
    • github.com/kubernetes/kubernetes/blob/master/Makefile
    • github.com/cockroachdb/cockroach/blob/master/Makefile

Makefile rules examples

REPOSITORY_NAME=$(shell basename $(CURDIR))
SOURCES=$(shell find . -name "*.go" | grep -v vendor/)
PACKAGES=$(shell go list ./... | grep -v vendor/)

default: test

	GORACE="halt_on_error=1" go test -race -v $(PACKAGES)

Vendor handling

  • need reproducible builds: go get unreliable (e.g aws-sdk-go)
  • From not managing to GO15VENDOREXPERIMENT to 1.6
  • https://github.com/golang/go/wiki/PackageManagementTools
  • We chose Godep: good enough
  • Go < 1.6 => Godeps/_workspace
  • Go >= 1.6 => vendor/
  • Make rules for lock & restore

Vendor handling rules for Go 1.6

	go get -u github.com/tools/godep

	go get -v -t ./...

	rm -rf Godeps vendor
	godep save -t $(PACKAGES)

	godep restore

Code check tooling

  • gofmt: all code looks the same
  • go vet: find errors not caught by the compiler
  • golint: find style mistakes (implements https://github.com/golang/go/wiki/CodeReviewComments)
  • gometalinter: optional. Sometimes broken.
  • fgt: exit 0/1, stdout/stderr

Code check tooling

FGT := $(shell command -v fgt)
GOLINT := $(shell command -v golint)
ifndef FGT
	# use fgt to exit when some lint writes to stderr
	# https://github.com/golang/lint/issues/65
	go get -u github.com/GeertJohan/fgt
ifndef GOLINT
	go get -u github.com/golang/lint/golint

Code check tooling

# for travis, circle CI
test-ci: linters-ci test

	fgt gofmt -l $(SOURCES)

	go list ./... | grep -v vendor/ | xargs -L1 fgt golint

	fgt go vet $(PACKAGES)

Other tooling

  • godef
  • godeps
  • gorename
  • oracle / guru (1.7): referrers
  • go-carpet: test coverage in your CLI
  • use a IDE that integrates them!

Deploy: from local to prod

  • most of us work in OSX
  • started with .tgz with xcompiled binary plus assets/config
  • moved from .tgz to .deb generated in Docker with a Ruby tool
  • ported the tool to BASH, got rid of Docker
  • migrated to AWS CodeDeploy package
  • compile, package and deploy takes ~1m
  • next step: graceful connection restarts

Concurrency: Mutex vs chan

  • We started the erlang way: actor model, channels all around. Something felt wrong.
  • Go proverb #2:
Channels orchestrate; mutexes serialize.
― Rob Pike

Concurrency: Mutex vs chan

  • Typical mutex usages:
    • concurrent shared state access
    • use RWMutex and RLock() if possible
  • Typical channel usages:
    • goroutine coordination for a linear behaviour
    • websocket connection communication
    • fan-in / fan-out, cheap map/reduce (e.g parallel downloading of s3 files)
    • ctx.Done() for goroutine cleanup
    • sync.WaitGroup uses channels behind the scenes

Go VM metrics

  • expvar package
  • Memory stats: allocations, frees, lookups...
  • GC stats: num GC, last GC, paused times...
  • Runtime: num goroutines, num CPUs...
  • Process stats: ulimits, max fds, opened fds...

Monster Legends teamwars

Monster Legends teamwars (Go)

  • WAMP protocol: WS, RPC & Pub/Sub

Monster Legends teamwars

  • Our first Go project
  • Forked the turnpike project
  • Marcos Quesada is working in his own implementation.
  • Chat messages => pub/sub
  • Game commands => RPC
  • Dispatcher (observer pattern) for message passing between components
  • Scheduler for attacks and wars endings
  • JSON and big numbers issue
  • Infrastructure design: cluster vs big machine
  • Saving chat messages in S3


  • First Go microservice
  • Banners between games logic
  • Receives all the login requests from all the games in 2 servers
  • JSON-based configuration uploaded to S3 and a reading goroutine (next step: Consul?)
  • Tried XP as methodology

Push notifier

  • Send Apple and Google game push notifications
  • Antispam and delay features
  • Integration with different AWS services
  • Datadog metrics reporter


  • WIP
  • In-house log analysis tool
  • Enrich up to 50K logs/second/instance
  • Copy up to 15M rows/minute to Redshift

Best resources: