Who are we
- catalan startup based in Barcelona
- casual games (mobile & FB)
- ~275 employees
- several million DAUs
Our games
Disclaimer
This is based in our personal experience only.
Several million DAUS?
- High traffic
- Hundreds or thousands OPS
- Scalabity
- Availability
- Performance: latency, throughput...
Backend Stack
- PHP/Symfony2: stateless, request/response
- Golang, Erlang/OTP: stateful, CSP/actor model
- JS, Python, BASH, Groovy...
- AWS: VPC/EC2, Redshift, RDS, DynamoDB, S3, SNS, SQS, EMR, Lambda, CodeDeploy, Gateway...
- MySQL, Redis, Cassandra, Riak, ElasticSearch...
From stateless to stateful
- Some game features have state or need soft-real time
- Request/Response model is wasteful or not performant enough
- Erlang/OTP, Elixir, Golang, Java/Scala/Akka, NodeJS, Rust, Haskell, Stackless Python and others provide:
- better performance than PHP
- stateful servers
- real-time capabilities
- concurrency model
But what about...
- Learning curve
- Performance limits
- Ecosystem
- Documentation
- DX
- Ops
- Evolution
What we did 2 years ago
- Bet for Erlang/OTP
- Created a chat and some microservices
- Found some issues:
- lack of tutorials and good documentation
- small library ecosystem
- not easy to deploy
- difficult to master:
- functional programming
- recursion
- immutability
- distributed systems
- some of this issues are solved in Elixir
Pivot to Golang
- simplicity & readability
- small language
- easy to understand and fix
- easy but powerful concurrency model if you know Erlang
- also not very difficult from PHP:
- most of the the team has finished the go tour
- ~15% code full time, 25% part time
- ~35% has some go code in prod, and increasing
- our frontend is doing Golang too! :-)
Pivot to Golang
- very good documentation: go tour, godoc, wiki, tutorials...
- big and active ecosystem: github libs (aws-sdk-go), blog posts, mailing list, slack...
- tooling
- big ones (Google and others) betting on Go
Pivot to Golang
- we are pretty happy about the results
- huge performance gains
- fast development rate
- small time to deploy
- working in good DX: tools and frameworks
Automation: Makefile
- There are many useful go commands, flags and tools
- Automate its usage
- Learn from the masters:
- github.com/hashicorp/terraform/blob/master/Makefile
- github.com/hashicorp/consul/blob/master/GNUmakefile
- github.com/kubernetes/kubernetes/blob/master/Makefile
- github.com/cockroachdb/cockroach/blob/master/Makefile
Makefile rules examples
SP_GOPATH=github.com/socialpoint/
REPOSITORY_NAME=$(shell basename $(CURDIR))
REPOSITORY_PATH=$(CURDIR)
BINARY_NAME=$(REPOSITORY_NAME)
SOURCES=$(shell find . -name "*.go" | grep -v vendor/)
PACKAGES=$(shell go list ./... | grep -v vendor/)
default: test
test:
GORACE="halt_on_error=1" go test -race -v $(PACKAGES)
Vendor handling
- need reproducible builds: go get unreliable (e.g aws-sdk-go)
- From not managing to GO15VENDOREXPERIMENT to 1.6
- https://github.com/golang/go/wiki/PackageManagementTools
- We chose Godep: good enough
- Go < 1.6 => Godeps/_workspace
- Go >= 1.6 => vendor/
- Make rules for lock & restore
Vendor handling rules for Go 1.6
godep-get:
go get -u github.com/tools/godep
deps-get:
go get -v -t ./...
deps-lock:
rm -rf Godeps vendor
godep save -t $(PACKAGES)
deps-restore:
godep restore
Code check tooling
- gofmt: all code looks the same
- go vet: find errors not caught by the compiler
- golint: find style mistakes (implements https://github.com/golang/go/wiki/CodeReviewComments)
- gometalinter: optional. Sometimes broken.
- fgt: exit 0/1, stdout/stderr
Code check tooling
FGT := $(shell command -v fgt)
GOLINT := $(shell command -v golint)
linters-ci-get:
ifndef FGT
# use fgt to exit when some lint writes to stderr
# https://github.com/golang/lint/issues/65
go get -u github.com/GeertJohan/fgt
endif
ifndef GOLINT
go get -u github.com/golang/lint/golint
endif
Code check tooling
# for travis, circle CI
test-ci: linters-ci test
fmt-ci:
fgt gofmt -l $(SOURCES)
lint-ci:
go list ./... | grep -v vendor/ | xargs -L1 fgt golint
vet-ci:
fgt go vet $(PACKAGES)
Other tooling
- godef
- godeps
- gorename
- oracle / guru (1.7): referrers
- go-carpet: test coverage in your CLI
- use a IDE that integrates them!
Deploy: from local to prod
- most of us work in OSX
- started with .tgz with xcompiled binary plus assets/config
- moved from .tgz to .deb generated in Docker with a Ruby tool
- ported the tool to BASH, got rid of Docker
- migrated to AWS CodeDeploy package
- compile, package and deploy takes ~1m
- next step: graceful connection restarts
Concurrency: Mutex vs chan
- We started the erlang way: actor model, channels all around. Something felt wrong.
- Go proverb #2:
Channels orchestrate; mutexes serialize.
― Rob Pike
Concurrency: Mutex vs chan
- Typical mutex usages:
- concurrent shared state access
- use RWMutex and RLock() if possible
- Typical channel usages:
- goroutine coordination for a linear behaviour
- websocket connection communication
- fan-in / fan-out, cheap map/reduce (e.g parallel downloading of s3 files)
- ctx.Done() for goroutine cleanup
- sync.WaitGroup uses channels behind the scenes
Go VM metrics
- expvar package
- Memory stats: allocations, frees, lookups...
- GC stats: num GC, last GC, paused times...
- Runtime: num goroutines, num CPUs...
- Process stats: ulimits, max fds, opened fds...
Monster Legends teamwars
Monster Legends teamwars (Go)
- WAMP protocol: WS, RPC & Pub/Sub
Monster Legends teamwars
- Our first Go project
- Forked the turnpike project
- Marcos Quesada is working in his own implementation.
- Chat messages => pub/sub
- Game commands => RPC
- Dispatcher (observer pattern) for message passing between components
- Scheduler for attacks and wars endings
- JSON and big numbers issue
- Infrastructure design: cluster vs big machine
- Saving chat messages in S3
Crosspromotion
- First Go microservice
- Banners between games logic
- Receives all the login requests from all the games in 2 servers
- JSON-based configuration uploaded to S3 and a reading goroutine (next step: Consul?)
- Tried XP as methodology
Push notifier
- Send Apple and Google game push notifications
- Antispam and delay features
- Integration with different AWS services
- Datadog metrics reporter
Brain
- WIP
- In-house log analysis tool
- Enrich up to 50K logs/second/instance
- Copy up to 15M rows/minute to Redshift
