I made a Server Program in Go.

MiddleMan: A Pub/Sub and Request/Response server in Go.

This is my first Go project. It is a rewrite of an existing Python server, based on some previous work: https://entitycrisis.blogspot.de/2016/09/python3-asyncio-pubsub-plaything.html

All code available here: https://github.com/simonwittber/middleman

I can see Go becoming my preferred language for game servers.

Python3 Asyncio PubSub Plaything

#!/usr/bin/env python
import io
import asyncio
import websockets
import logging
import collections

logger = logging.getLogger('websockets.server')
logger.setLevel(logging.ERROR)
logger.addHandler(logging.StreamHandler())
events = collections.defaultdict(lambda: set())
#-----------------------------------------------------------------------
async def handle_outgoing_queue(websocket):
    while websocket.open:
        msg = await websocket.outbox.get()
        await websocket.send(msg)
#-----------------------------------------------------------------------
async def pubsub(websocket, path):
    websocket.prefix = path.encode()
    websocket.outbox = asyncio.Queue()
    websocket.subscriptions = set()
    sender_task = asyncio.ensure_future(handle_outgoing_queue(websocket))
    while True:
        msg = await websocket.recv()
        if msg is None: break
        if isinstance(msg, str): msg = msg.encode()
        stream = io.BytesIO(msg)
        await handle_message(websocket, stream)
    sender_task.cancel()
    for name in websocket.subscriptions:
        try:
            events[name].remove(websocket)
        except KeyError:
            pass
#-----------------------------------------------------------------------
async def handle_message(websocket, stream):
    cmd = stream.readline().strip()
    name = websocket.prefix + stream.readline().strip()
    print(cmd, name);
    if cmd == b"SUB":
        events[name].add(websocket)
        websocket.subscriptions.add(name)
    elif cmd == b"UNS":
        subscribers = events[name]
        try:
            websocket.subscriptions.remove(name)
        except KeyError:
            pass
        try:
            subscribers.remove(websocket)
        except KeyError:
            pass
    elif cmd == b"PUB":
        stream.seek(0)
        msg = stream.read()
        for subscriber in events[name]:
            await subscriber.outbox.put(msg)
#-----------------------------------------------------------------------
start_server = websockets.serve(pubsub, '0.0.0.0', 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

Python on Hardware

Just bought a MicroPython pyboard. Plugged it in, and I have a python prompt. Never thought this day would come. Amazing.

Python-3 Game Server

I've been working on something new. I get a lot of projects where I have to add multiuser functionality. Finally I've made the jump to Python 3, and built a generic multi-user game server, so I can stop rewriting this sort of thing for every new project. It's using the most excellent asyncio framework, with aiopg for Postgresql support.

The server code is on GitHub, the Unity3D client is still in progress.

Those tedious functions, such as Registration, Authentication, Reset Password, Messaging, Rooms and Object storage are solved, may I never have to write a "Reset Password" module again!

Footy Boss Game has launched!

The project I've been working on for the last 9 months has now launched!

footyboss.com.au

It is a very big, very complicated game where the player manages every aspect of a team of Australian Football players.

The main simulator engine is written in Python, which I generated from the client's specification. It's not often a spec is detailed enough, and so well structured you can generated code from it! In total there is over 10000 lines of code just in the match simulator. The simulator generates a 1MB match log for each football game played, which can then be viewed as a real game thanks to a 3D visualisation written with Unity3D.

Some code statistics:

Sim Visualisation: 50000 lines of C#
Game Server: 27000 lines of Python (10000 auto-generated)
Game Front End: 5000 lines of Javascript (ExtJS framework)
Database: 90 tables, 35 functions.

A Football Game

Ive been working on a small yet ambitious game over the last 3 months.

It's almost baked, so we're starting up the Hype Machine.

You can see a small preview here.

It follows my usual MO, with a Python backend and a Unity3D client. More coming soon! :-)

The Secret Project

This is the complete schema for my current project. Of course, deliberately downscaled so you can't exactly see what is going on... :-)

I'm building most of the game server within PostgreSQL itself, using the plpython extension language. The game interface will be HTML, with a tiny bit of Unity3D visualisation. Having been out of the web dev game for some years now, I'm not sure of the best tech for exposing my database as a Web API, and I'm double unsure about what client side tech to use. Shouldn't we have a one click develop and deploy solution by now? :-)

A new startup...

If all things align, I'm about to start work on a new venture.

For the last two years I've been building something epic, and I've now handed that project over to a new team in Singapore. It is time for something new!

Of course, I can't say exactly what I'll be doing, but it will all be made public in about 3 months. What I can say, is that the project is at least an order of magnitude more complex than any system I have yet built. It will run on Unity3D and Python. It's a niche market game, but there is a dedicated and unusual audience waiting to play it. And I have a stake in the new company. I think that is important. I think it guarantees a success, or at least a spectacular disaster! :-)

(Python + Unity3D) == MMO

This is something I've been helping build for quite some time now, and it is almost done.



It is an educational MMOG. It is powered by Python out the back, and Unity3D at the front. The Python server(s) are running Python 2.7, with Greenlets :-) and ZeroMQ for IPC. We had to design our own ORM, because SQLAlchemy (our first choice) didn't play well with the asynchronous Psycopg2 extension. The backend is loosely based on an Entity Component architecture, which is quite elegant and a joy to work with. It has been so easy to modify and extend, that I think I'll be using this architecture for most of my new projects.

Soon, the whole system will be battle tested by hordes of screaming children. I am confident it will weather the storm, and more. :-)

Things I like about PyPy...

The fact that PyPy needs to use scientific notation to print the results of a pystone benchmark... :-)


Pystone(1.1) time for 50000 passes = 0.016978
This machine benchmarks at 2.94499e+06 pystones/second

We're building a library!

Would you like to help build a library for a school in Thailand?

It is easy to help out, all you need is to send a FB like, and you're sending a brick for the library!

Skoolbo is funding (and even helping with the brick laying!) the project.

Skoolbo is a project I've been working for almost two years. The backend is a beautiful piece of Python, and the front end is a similar quality piece of Unity3d, which will be released Real Soon Now.

Convert CSV to TSV

import csv
open("output.tsv", "w").write("\n".join("\t".join(e.strip() for e in i) for i in csv.reader(open("input.csv"))))

Fun and Games in Boston

After three days of heavy fog, two days of bleary jet lag and one day of the lurgy, the sun came out today. So far I've met with quite a few interesting characters. Alex Schwartz of "Smuggle^H^H^H^H^H^Hnuggle Truck" fame, Ned Batchelder (an altogether nice chap) and today I had the good fortune to have lunch with Glyph Lefkowitz and Jp Calderone who assured me that Apple is not the Evil Empire we might think it is, and that PyPy is the future, as long as we can get build times down.

I'll be home in three days, and I'm super-keen to get back into some coding, and maybe have a crack at breathing life into some of my old projects. Oh, and start compiling PyPy trunk. :-)

Python Hackers in Boston

I'm in Boston MA next week, looking to hire some awesome Python hackers to help open a new game development studio. We use Python for our game server, and Unity3D/C# for the game client. It is an amazing project with an existing international team based in Perth, Western Australia.

If you're interested, in Boston and want to chat, drop me an email at simonwittber@differentmethods.com and we can arrange a time and place.

Python in Unity3D

This is amazing. My two favourite technologies working together.

Adding Methods without Inheritance

Did you know that C# will let you add methods to objects, without needing to write new sub-classes? I didn't until today. The following snippet adds a new method to generic list instances, called Shuffle, which randomises the items in the list.

static class ExtensionMethods {
 
 public static void Shuffle(this IList list)  
 {  
     var rand = new System.Random();
     int n = list.Count;  
     while (n > 1) {  
         n--;  
         int k = rand.Next(n + 1);  
         T value = list[k];  
         list[k] = list[n];  
         list[n] = value;  
     }  
 }

}

Is this possible in Python? I think it is, but i need to experiment with some code first... stay tuned.

Update:

Of course you can do this in Python, but it won't work on built in types.

>>> import random
>>> 
>>> x = range(10)
>>> def shuffle(self):
...  random.shuffle(self)
... 
>>> list.shuffle = shuffle
Traceback (most recent call last):
  File "", line 1, in 
TypeError: can't set attributes of built-in/extension type 'list'
>>> class List(list): pass
... 
>>> n = List(x)
>>> List.shuffle = shuffle
>>> n.shuffle()
>>> n
[5, 3, 7, 4, 2, 9, 1, 6, 0, 8]
>>>

Markov Name Generator

I use something like this to generate names for stars and planets in my procedural universe game thingo.

import random

class Markov(object):
    def __init__(self, samples, order, minLength):
        self.order = order
        self.minLength = minLength
        self.chains = {}
        self.used = []
        self.samples = [i.strip().upper() for i in samples if len(i) > order]
        for word in self.samples:
            for letter in xrange(len(word)-order):
                token = word[letter:letter+order]
                entries = self.chains.setdefault(token, list())
                entries.append(word[letter + order])

    def next(self):
        s = ""; 
        while True:
            n = random.choice(self.samples)
            i = random.randint(0, len(n) - self.order)
            s = n[i:i+self.order]
            while len(s) < len(n):
                i = random.randint(0, len(s) - self.order)
                token = s[i:i+self.order]
                if token not in self.chains:
                    break 
                s += random.choice(self.chains[token])
            s = s[0] + s[1:].lower()
            if not (s in self.samples or s in self.used or len(s) < self.minLength):
                break
        self.used.append(s);
        return s;

    def reset(self):
        self.used.Clear()

You can seed it with any data you like to generate any sort of name. Eg:

starnames = """
Achernar Alpha Eridani \
Achird Eta Cassiopeiae \
Acrab Beta Scorpii \
Acrux Alpha Crucis \
Acubens Alpha Cancri \
Adhafera Zeta Leonis \
Adhara Epsilon Canis Majoris \
Ain Epsilon Tauri \
Aladfar Eta Lyrae \
Alamak Gamma Andromedae \
Alathfar Mu Lyrae \
Alaraph Beta Virginis \
Albaldah Pi Sagittarii \
""".split(" ")

m = Markov(starnames, 2, 6)
for i in xrange(10):
    print m.next()

Gives this output:

Siolphh
Hafere
Ammaam
Ammaka
Ittatartar
Chieri
Scooor
Araineaa
Feriisda
Achuih

7 Bit Integer compression, in Python.

def decode7bit(bytes):
    bytes = list(bytes)
    value = 0
    shift = 0
    while True:
        byteval = ord(bytes.pop(0))
        if(byteval & 128) == 0: break
        value |= ((byteval & 0x7F) << shift)
        shift += 7
    return (value | (byteval << shift))


def encode7bit(value):
    temp = value
    bytes = ""
    while temp >= 128:
        bytes += chr(0x000000FF & (temp | 0x80))
        temp >>= 7
    bytes += chr(temp)
    return bytes


x = encode7bit(345)
print decode7bit(x)

Public and Private in Python?

I've just finished refactoring an awful C# class. I had been delaying the job for a while because I didn't want to do it.

Then, while staring at the code, I realised I could simply delete all the private methods and fields, leaving the public interface intact, and then re-implement the class in a much cleaner way. Great idea! I finished the job in a few hours, with a brand new class that doesn't give me a headache.

This is the first time I've actually seen a real value in having private and public scope for methods and variables. It makes refactoring much safer and easier! So... why don't we have them in Python?

Encryption between Python and C#

I do a lot of work in restricted environments, where I still need to secure internet traffic. I chose to use the RC4 algorithm because it is simple to implement and relatively fast. It is actually easy enough that a simple programmer like myself could write matching Python and C# implementations!

RC4 is also a stream cipher, which means you don't need to worry about breaking your cleartext into blocks, padding them out... and other related pain.

If you do use RC4, you should be aware that it has some vulnerabilities.

Python:

def RC4(data, key):
    x = 0 
    s = range(256)
    for i in range(256):
        x = (x + s[i] + ord(key[i % len(key)])) % 256 
        s[i], s[x] = s[x], s[i]
    x = y = 0 
    out = ""
    for c in data:
        x = (x + 1) % 256 
        y = (y + s[x]) % 256 
        s[x], s[y] = s[y], s[x]
        out += chr(ord(c) ^ s[(s[x] + s[y]) % 256])
    return out

C#:

using System;


public class RC4
{   
    static public void Cipher (ref byte[] bytes, string skey)
    {   
        var key = System.Text.ASCIIEncoding.ASCII.GetBytes(skey);
        byte[] s = new byte[256];
        byte[] k = new byte[256];
        byte temp;
        int i, j;

        for (i = 0; i < 256; i++) {
            s[i] = (byte)i;
            k[i] = key[i % key.GetLength (0)];
        }

        j = 0;
        for (i = 0; i < 256; i++) {
            j = (j + s[i] + k[i]) % 256;
            temp = s[i];
            s[i] = s[j];
            s[j] = temp;
        }

        i = j = 0;
        for (int x = 0; x < bytes.GetLength (0); x++) {
            i = (i + 1) % 256;
            j = (j + s[i]) % 256;
            temp = s[i];
            s[i] = s[j];
            s[j] = temp;
            int t = (s[i] + s[j]) % 256;
            bytes[x] ^= s[t];
        }
    }
}