Who are we?

• casual games (mobile & FB)
• > 200 employees
• several million DAUs

Backend stack

• LAMP on AWS
• PHP/Symfony2
• Python, Cassandra, Redis, Riak...
• ... and now Erlang/OTP ;-)

Our games

Why Erlang/OTP?

Why Erlang/OTP?

Disclaimer !

SP erlang projects:

chat

session length

crashup

chat

chat

the problem

• high concurrency
• (soft) real time
• FB & mobile connectivity
• 2 rooms:
  
  
  • public (language)
  • alliance

chat: existing solutions

> ejabberd / mongooseIM

• erlang-based
• XMPP protocol
• many modules
• too complex

chat: our custom solution

• minimum viable product
• multiple iterations
• learn as you go

chat: building blocks

Chat > Cowboy

• HTTP server, supports TLS
• suports WS and WSS
• custom app + admin protocol over it

• [
      CommandId,
      TransactionId,
      ArrayOfArguments,
      Timestamp (S2C)
  ]

Chat > GProc

• Process dictionary: register a process under an alias
• multi alias
• counters
• aggregated counters

GProc examples

• gproc:reg({p, l, {chat_client, UID}}),

• gproc:reg({c, l, {room_sessions, ?ROOM_TYPE_PUBLIC, Id}}, 1)

• gproc:send({p, l, {room_session, alliance, Id}}, Msg),

• gproc:send(?ROOM_NAME_KEY(?ROOM_TYPE_ALLIANCE, Id), {update_history, Msg})

• S = gproc:select([{MatchByKey, Guard, Result}])

• gproc:select_count({all, all}, [{MatchHead, Guard, Result}])

chat

tools

• make
• rebar
• sync
• reltool
• observer
• crash parser

chat

demo

chat

lessons learned

• erlang is hard
• library ecosystem and erlang version support
• 3G connections problems
• WSS for the win
• memory leak -> app error (observer)

session length

session length

the problem

• Measure Users Session Length
• All Games & All platforms
• High Scalability
• High Stability

session length: the solution

• Stateful Processes
• Auto expire
• Heartbeats
• Clustering

session length: building blocks

session length: tools

• Riak Core Vs Own Cluster
• Ranch
• Eunit

session length: Scaling Strategy

• Consistent hashing ring

session length: Scaling Strategy

• Global Registry

session length: Scaling Strategy

• spawn process on destination Node

session length: Ranch

session length: Ranch

session length: Basic Eunit Test

session length: demo

lessons learned

• Distributed Apps are hard!
• Don't reinvent the wheel

crashup

High available mobile tracking service

crashup: the problem

• Mobile Crash Uploads
• High Performance
• Autoscale

crashup: the solution

• Microservices
• S3 Erlcloud
• VM Metrics

crashup: building blocks

crashup: tools

• erlang.mk & relx
• Eunit
• Meck
• Folsom

crashup: Advanced Tests

crashup: Advanced Tests

crashup: Folsom

crashup: demo

crashup: lessons learned

• TDD: Development Speed Up
• Advanced Mocking is Hard
• Tests Teardown are tricky
• Folsom is great

Future projects

Stateful game sessions

Real time strategy games

full stack erlang game

The end

Questions?