Erlang/OTP

in Social Point

@gonzaloserrano
@marcos_quesada
@ronnylt

Who are we?


  • casual games (mobile & FB)
  • > 200 employees
  • several million DAUs

Backend stack


  • LAMP on AWS
  • PHP/Symfony2
  • Python, Cassandra, Redis, Riak...
  • ... and now Erlang/OTP ;-)

Our games

Why Erlang/OTP?

Why Erlang/OTP?

Disclaimer !

SP erlang projects:


chat

session length

crashup

chat

chat

the problem

  • high concurrency
  • (soft) real time
  • FB & mobile connectivity
  • 2 rooms:

    • public (language)
    • alliance

chat: existing solutions


> ejabberd / mongooseIM

  • erlang-based
  • XMPP protocol
  • many modules
  • too complex

chat: our custom solution


  • minimum viable product
  • multiple iterations
  • learn as you go

chat: building blocks

Chat > Cowboy

  • HTTP server, supports TLS
  • suports WS and WSS
  • custom app + admin protocol over it 
  • [
    CommandId,
    TransactionId,
    ArrayOfArguments,
    Timestamp (S2C)
]

Chat > GProc

  • Process dictionary: register a process under an alias
    • multi alias
    • counters
    • aggregated counters

GProc examples

  • 
gproc:reg({p, l, {chat_client, UID}}),
                        
  • 
gproc:reg({c, l, {room_sessions, ?ROOM_TYPE_PUBLIC, Id}}, 1)
                        
  • 
gproc:send({p, l, {room_session, alliance, Id}}, Msg),
                        
  • 
gproc:send(?ROOM_NAME_KEY(?ROOM_TYPE_ALLIANCE, Id), {update_history, Msg})
                        
  • 
S = gproc:select([{MatchByKey, Guard, Result}])
                        
  • 
gproc:select_count({all, all}, [{MatchHead, Guard, Result}])

chat

tools

  • make
  • rebar
  • sync
  • reltool
  • observer
  • crash parser

chat

demo

chat

lessons learned

  • erlang is hard
  • library ecosystem and erlang version support
  • 3G connections problems
  • WSS for the win
  • memory leak -> app error (observer)

session length

session length

the problem

  • Measure Users Session Length
  • All Games & All platforms
  • High Scalability
  • High Stability

session length: the solution

  • Stateful Processes
  • Auto expire
  • Heartbeats
  • Clustering

session length: building blocks

session length: tools

  • Riak Core Vs Own Cluster
  • Ranch
  • Eunit

session length: Scaling Strategy

  • Consistent hashing ring

session length: Scaling Strategy

  • Global Registry

session length: Scaling Strategy

  • spawn process on destination Node

session length: Ranch

session length: Ranch

session length: Basic Eunit Test

session length: demo

lessons learned


  • Distributed Apps are hard!
  • Don't reinvent the wheel

crashup

High available mobile tracking service

crashup: the problem

  • Mobile Crash Uploads
  • High Performance
  • Autoscale

crashup: the solution


  • Microservices
  • S3 Erlcloud
  • VM Metrics

crashup: building blocks

crashup: tools


  • erlang.mk & relx
  • Eunit
  • Meck
  • Folsom

crashup: Advanced Tests

crashup: Advanced Tests

crashup: Folsom

crashup: demo

crashup: lessons learned

  • TDD: Development Speed Up
  • Advanced Mocking is Hard
  • Tests Teardown are tricky
  • Folsom is great

Future projects

Stateful game sessions


Real time strategy games


full stack erlang game

The end

Questions?