Category Archives: Programming

Constructing the Twitch Bot

I want to give the IRC bot a post of its own as not to distract too much from the machine learning post. This bot will be used to log IRC chat messages and poll for viewer counts on Twitch. There are a ton of different ways to implement this so feel free to take what you see here and build a bot of your own. Here I’ll be using Python to construct the bot. All our data will be stored in a SQLite database, and we will spin off three threads for each Twitch streamer we plan to track. One thread will keep track of whether the streamer is online or not, another thread will log all the IRC chat activity, and the finial thread will log the viewer count. Using SQLite is great because later we’ll be able to easily access our data via Pandas.

So first let’s define the main loop that will kick everything off:

import sys;
import time;
import socket;
import sqlite3;
import requests;
from threading import Thread, Lock;
from time import gmtime, strftime, sleep;

if __name__ == "__main__":

	password  = "oauth:YOUR_CODE_HERE";
	nickname  = "YOUR_USERNAME";
	client_id = "YOUR_CLIENT_ID";

	study_channels = ["drdisrespectlive", "lirik", "manvsgame", "cohhcarnage"]

	TwitchBot.db_con = sqlite3.connect("twitch.db", check_same_thread=False);
	TwitchBot.db_con.text_factory = str;

	bots = []
	for i in range(0, len(study_channels)):
		bots.append(TwitchBot(study_channels[i], nickname, password, client_id))

	running = True;

	while running:
		command = raw_input("");
		if command == "exit":
			running = False;
			for i in range(0, len(study_channels)):
				bots[i].running = False

	TwitchBot.db_con.close();

Let’s break this down a little:

  1. if __name__ == “__main__”: checks to make sure we are running the script directly (as opposed to being imported via someone else).
  2. Next we define a few variables which we will need in order to connect to the IRC chat room and poll for viewer counts. password is our OAuth password we will use when connecting to the chat rooms. If you don’t already have one grab one from herenickname is the username that you use when logging into Twitch (therefore you’ll need to sign up for Twitch in order to log into the IRC rooms). And finally we have client_id which is our ID which Twitch gives us when we register our application (see here). Once you have this information filled out you’ll be good to go!
  3. Now comes the fun part, which streamers are you going to follow? study_channels is a list of the channels we are going to log. When picking steamers you might want to consider when they stream (i.e. if you want the bot running 24/7 find 2/3 night steamers and 2/3 daytime streamers) and how many threads you want to use for the bot (3 threads per streamer plus one main thread).
  4. TwitchBot.db_con is a static variable for the SQLite database connection. (We will write the TwitchBot class itself soon!) TwitchBot.db_con.text_factory = str just ensures that every time we attempt to save a string to the SQLite database is in the standard string format we are expecting (some emotes are sent in non-ascii format. stuff like “¯_(ツ)_/¯”, which we will ignore). I should also point out that check_same_thread=False prevents SQLite from throwing errors when accessing a single database via multiple threads. In any other scenario this would be a problem but we will be using our own mutex to ensure that only one thread writes to the database at a time.
  5. Next we create an instance of the TwitchBot class for each streamer we plan to follow and then we “listen” for when the user types “exit” and then proceed to shut down all the bots we created and close the database connection.

Next let’s define the constructor for our TwitchBot class. Here I will check to see whether our SQLite database already has the tables we need and create them if they don’t already exist. We will also spin off the three threads we’ll need for each channel we plan to log information for.

class TwitchBot:

	mutex  = Lock();
	db_con = None;

	def __init__(self, channel, nickname, password, client_id):

		self.net_data       = None;
		self.net_sock       = socket.socket();
		self.channel        = channel;
		self.nickname       = nickname;
		self.password       = password;
		self.client_id      = client_id;
		self.running        = True;
		self.is_online      = False;
		self.cur_stream_ind = None;

		TwitchBot.mutex.acquire();

		self.cur = TwitchBot.db_con.cursor();
		self.cur.execute("CREATE TABLE IF NOT EXISTS messages (id integer PRIMARY KEY AUTOINCREMENT, username text NOT NULL, message text NOT NULL, channel text NOT NULL, datetime_recv datetime NOT NULL);")
		self.cur.execute("CREATE TABLE IF NOT EXISTS viewers (id integer PRIMARY KEY AUTOINCREMENT, num_viewers integer NOT NULL, channel text NOT NULL, datetime_recv datetime NOT NULL);")
		self.cur.execute("CREATE TABLE IF NOT EXISTS streams (id integer PRIMARY KEY AUTOINCREMENT, start_time datetime NOT NULL, end_time datetime, channel text NOT NULL);")
		TwitchBot.db_con.commit();

		TwitchBot.mutex.release();

		thread_func = self.check_online_thread;
		thread = Thread(target=thread_func, args=(self,));
		thread.start();

		sleep(1)

		thread_func = self.irc_thread;
		thread = Thread(target=thread_func, args=(self,));
		thread.start();

		sleep(1)

		thread_func = self.viewer_thread;
		thread = Thread(target=thread_func, args=(self,));
		thread.start();

		sleep(1)

The first thing we do here is acquire the mutex and check whether three tables exist within our SQLite database (by the way, you can hover your mouse over the code above and use the slider to see the rest of the SQL code). The first table we need is called “messages”:

In this table we have the IRC username for who said the chat message, the message itself, the channel in which the message was said, and the date + time for when we received the message. We also need a table for our viewer counts, let’s call this table “viewers”:

In this table we have the number of viewer, the channel, and the date + time for which we polled for the viewer count. Next we need a table that will allow us to differentiate the different streams given by the same streamer (this also allows us to stop polling the viewer count when they go offline). Let’s call this table “streams”:

Here we have the start and end time for each stream and the channel that went live. Setting things up this way will allow us to easily grab all the messages and viewer counts for each of the individual streams. If you would like to view your SQLite database similar to the images above you should take a look at DB Browser for SQLite. Next we spin off our three threads, the first of which checks to see whether the streamer is online:

def check_online_thread(self, data):
	while self.running:
		try:
			resp = requests.get(
				"https://api.twitch.tv/helix/streams?user_login="+self.channel,
				headers={"Client-ID": self.client_id}
			);

			resp = resp.json();

			if "data" in resp:
				if not self.is_online:
					print("33[93mChecking to see if "+self.channel+" is online33[0m.");

				if not resp["data"]:
					if self.is_online:

						TwitchBot.mutex.acquire();
						self.cur.execute("UPDATE streams SET end_time = ? WHERE id = ?;",
							(strftime("%Y-%m-%d %H:%M:%S", gmtime()), self.cur_stream_ind)
						);
						TwitchBot.db_con.commit();
						TwitchBot.mutex.release();

						print("33[93m""+self.channel+"" is offline33[0m.");

						self.is_online = False;
					else:
						print("33[93m""+self.channel+"" is offline33[0m.");

				else:
					if not self.is_online:

						TwitchBot.mutex.acquire();
						self.cur.execute("INSERT INTO streams (start_time, end_time, channel) VALUES (?, NULL, ?);",
							(strftime("%Y-%m-%d %H:%M:%S", gmtime()), self.channel)
						);
						TwitchBot.db_con.commit();
						self.cur_stream_ind = self.cur.lastrowid;
						TwitchBot.mutex.release();

						print("33[93m""+self.channel+"" is online33[0m.");

						self.is_online = True;

			sleep(60);
		except:
			continue

Here we are simply sending a get request to Twitch using our application ID and Twitch sends us back a JSON string which we can decode and check whether there is an data available for our streamer. If the “data” vector in our get response is empty then the streamer is offline. If they were previously online but are now offline we update the end time in the database and inform all the other threads. On the other hand, if the stream was previously offline but is not online we insert a new row into the “streams” table and set the start time for the stream. The end time is set to NULL until the stream ends. This allows us to ignore streams for which we do not have complete data on as they are still in progress. Next let’s take a look at the thread which interfaces with the IRC chat room:

#--------------------------------------------------------------------------
def parse(self, line):
	prefix = "";
	trailing = [];
	if line[0] == ":":
		prefix, line = line[1:].split(" ", 1);
	if line.find(" :") != -1:
		line, trailing = line.split(" :", 1);
		args = line.split();
		args.append(trailing);
	else:
		args = line.split();
	command = args.pop(0);	
	return prefix, command, args

#--------------------------------------------------------------------------
def process(self, prefix, command, args):
	if command == "PING":
		self.net_sock.send(self.net_data.replace("PING", "PONG"));
	elif command == "376":
		self.net_sock.send("JOIN #"+self.channel+"rn");
	elif command == "PRIVMSG":
		if self.is_online:
			user_name = prefix.split("!")[0];
			user_message = args[1];
			print("33[91m"+user_name+"33[0m: "+args[1]);
			self.save_msg(user_name, user_message);

#--------------------------------------------------------------------------
def save_msg(self, user_name, user_message):
	TwitchBot.mutex.acquire();

	self.cur.execute("INSERT INTO messages (username, message, channel, datetime_recv) VALUES (?, ?, ?, ?);",
		(user_name, user_message, self.channel, strftime("%Y-%m-%d %H:%M:%S", gmtime()))
	);
	TwitchBot.db_con.commit();

	TwitchBot.mutex.release();

#--------------------------------------------------------------------------
def irc_thread(self, data):
	self.net_sock.connect(("irc.twitch.tv", 6667));

	self.net_sock.send("PASS "+self.password+"rn");
	self.net_sock.send("NICK "+self.nickname+"rn");

	while self.running:
		try:
			self.net_data = self.net_sock.recv(1024);
			if not self.net_data: break;

			lines = self.net_data.split("rn");
			lines.remove("");

			for line in lines:
				prefix, command, args = self.parse(line);
				self.process(prefix, command, args);
		except:
			continue;

	self.net_sock.close();

This thread is a bit more complicated but we first open a network socket to connect to “irc.twitch.tv” via port 6667. This is where all the IRC chat rooms are hosted. We send our password and nickname in order to log into the chat server. Then we continuously check for data coming from the Twitch IRC server. When we do receive data we pass it to self.parse(line) which grabs the prefix, command, and arguments for the response we got from the server. We only care to process a subset of all the possible responses we might receive from the IRC server and def process(self, prefix, command, args) handles this for us. Command “376” informs us that the login handshake has been finalized and we may now join an IRC channel. Command “PING” is to check whether we are still listening to the server. We have to reply with a “PONG” or the server will close our socket connection because it assumes we disconnected or timed out. The “PRIVMSG” command tells us we got a chat message and we should save it to the SQLite database. Saving occurs in the def save_msg(self, user_name, user_message) method and is relatively straight forward. Next we’ll need a thread for polling the viewer count. Polling the viewer count is very similar to checking whether the streamer is online:

#--------------------------------------------------------------------------
def grab_num_viewers(self):
	resp = requests.get(
		"https://api.twitch.tv/helix/streams?user_login="+self.channel,
		headers={"Client-ID": self.client_id}
	);

	try:
		resp = resp.json();
	except:
		return -1;

	if "data" in resp:
		if not resp["data"]:
			return -1;
		return resp["data"][0]["viewer_count"];
	else:
		return -1;

#--------------------------------------------------------------------------
def save_viewers(self, num_viewers):
	TwitchBot.mutex.acquire();

	self.cur.execute("INSERT INTO viewers (num_viewers, channel, datetime_recv) VALUES (?, ?, ?);",
		(num_viewers, self.channel, strftime("%Y-%m-%d %H:%M:%S", gmtime()))
	);
	TwitchBot.db_con.commit();
	
	TwitchBot.mutex.release();

#--------------------------------------------------------------------------
def viewer_thread(self, data):
	while self.running:
		try:
			if self.is_online:
				num_viewers = self.grab_num_viewers();
				if num_viewers != -1:
					print("33[94mNumber of viewers33[0m 33[93m(""+self.channel+"")33[0m: "+str(num_viewers));
					self.save_viewers(num_viewers);
				sleep(10);
			else:
				sleep(3);
		except:
			continue;

Here we again send a get request using the Twitch API, decode the JSON string, and check for the data list – however this time we are actually making use of the information within the data list!

Using the data collected by this bot we can construct plots like the one below!

Do you see that big spike between the 10:00 and 12:00 marks? Well if you check the VOD Bikeman (another streamer) actually raids ManVsGame 6 hours and 48 minutes into the stream and we were able to capture it using our bot!

You should make your own bot and give this a try! Let it run for a few days and collect plenty of data because I am planning to make a follow-up post were we will apply a statistical classifier to the IRC messages and see how well we can predict which stream they came from!

Knight Online Terrain (GTD format for 1299)

I want to talk a little bit about reading from binary files using C/C++ and the file structure for the Knight Online terrain files (i.e. files with the extension “.gtd”). To follow along you need two things:

  1. A C/C++ compiler, here I’ll be using Visual C++ (Community 2015)
  2. A 1299 terrain file, I’m using “karus_start.gtd” but any .gtd will work as long as it’s from the 1299 version of the game – however I’d recommend that you also use “karus_start.gtd” so that our number match up.

The basic concepts I’ll discuss here are valid for reading any of the Knight Online content files. Also, if a particular map spans multiple versions of the game (with each version varying in the file structure) you can use the data you know is required for the 1299 version (discussed in this post) to figure out what was added/removed in other versions. Therefore, even though we’ll be discussing the 1299 file format this information is relevant for cracking all versions of the Knight Online terrain files. Here I’ll be assuming you have little to no C/C++ knowledge. However, I am assuming you know general programming concepts and the difference between memory on the stack vs. memory on the heap.

First let’s read in the .gtd file.

/*
*/

#include "stdio.h"

//-----------------------------------------------------------------------------
int main(int argc, char** argv) {

	char filename[] = "karus_start.gtd";

	FILE* fp = fopen(filename, "rb");
	if(fp == NULL) {
		printf("ERROR: Unable to open file "%s"n", filename);
		return -1;
	}

	fclose(fp);

	return 0;
}

The function int main(int argc, char** argv) is the main entry point for the program. #include “stdio.h” includes the declarations for all standard input and output operations. char filename[] = “karus_start.gtd”; defines a variable name “filename” which is a pointer to an array of “char”s allocated on the stack. FILE* fp = fopen(filename, “rb”); opens the file “karus_start.gtd” in the “read binary” mode. This means we will only be reading from (as opposed to writing to) the file and we wish to treat the contents of the file as if it were all binary data. The rest of the code here checks to make sure we successfully opened the file (printing an error if we were unable to open it) and immediately closing the file and exiting the program.

So far this program don’t do much. We are just opening the file (if it exists) and closing it. What we want to do is start reading in the contents of the file. We can’t read in the contents unless we know a little bit about how the terrain information is structured. So before we start read in all the binary information let’s take a moment to discuss the .gtd file at a higher level.

Having played the game you have probably noticed how the maps are broken up into these little squares.

Each of these rectangles is referred to as a “tile” and each tile is a 4×4 rectangle. The map data for a tile consists of four points in 3D space. In this coordinate system X and Z run along the floor and Y points up into the sky. Therefore, since we know that tiles are 4×4 rectangles, the X and Z coordinates for all the tiles on the map can be set just by knowing how big the map is (this will be important later because only the Y coordinate for these four points is stored in the .gtd file). The idea of a tile isn’t enough though, we also need the concept of a “patch.” A patch consists of a rectangle of 8×8 tiles. Patches carry absolutely no information about how the map should be rendered. Patches carry information for how the map should be updated and are useful for increasing computational efficiency. We eventually plan to have trees blowing in the wind or NPCs performing particular animations, etc. – it would be a waste of CPU power to have trees located on the opposite side of the map blowing in the wind when the player isn’t anywhere near them. Patches are useful for only updating objects which are close to the player.

Interestingly, the 1299 .gtd files start with an 4-byte integer which we do not know the function of (I’d guess it’s likely a version number).

int iIdk;
fread(&iIdk, sizeof(int), 1, fp);

An int in Visual C++ is 4 bytes. int iIdk; allocates the 4 bytes on the stack which we will use to store the first 4 bytes of the .gtd file. fread(&iIdk, sizeof(int), 1, fp); actually reads 4 bytes from the file and copies them into the 4 bytes we have allocated. Next we read in the name of the map.

int iNL;
fread(&iNL, sizeof(int), 1, fp);

char m_szName[0xFF] = "";
if(iNL > 0) fread(m_szName, sizeof(char), iNL, fp);

Here we are reading in another 4-byte integer, although this time it is the length of the map name. This integer tells us how many characters there are in the map’s name. Now we’ll read in the size of the map, set the number of patches based on the map size, and grab all the map data.

int m_ti_MapSize = 0;
fread(&m_ti_MapSize, sizeof(int), 1, fp);
int m_pat_MapSize = (m_ti_MapSize - 1) / 8;

_N3MapData* m_pMapData = new _N3MapData[m_ti_MapSize*m_ti_MapSize];
fread(m_pMapData, sizeof(_N3MapData), m_ti_MapSize*m_ti_MapSize, fp);

Based on the code we have looked at up to this point it should be relatively straight forward to understand what’s going on here. However, we haven’t talked at all about _N3MapData , let’s take a look.

struct _N3MapData {
	float fHeight;
	unsigned int bIsTileFull : 1;
	unsigned int Tex1Dir : 5;
	unsigned int Tex2Dir : 5;
	unsigned int Tex1Idx : 10;
	unsigned int Tex2Idx : 10;

	_N3MapData(void) {
		bIsTileFull = true;
		fHeight = FLT_MIN;
		Tex1Idx = 1023;
		Tex1Dir = 0;
		Tex2Idx = 1023;
		Tex2Dir = 0;
	}
};

Each of the four points making up a tile has one of these map data structs. fHeight is the Y coordinate for a point (remember we can get the X and Z coordinate based on how big the map is). bIsTileFull is less straight forward but I believe it gets used when setting which textures to use for a particular tile (here I’m taking “full” to mean “stuff is covering this tile”). I would be interested to hear possible alternative theories about this. The rest of the variables are used for deciding which texture to blend together (Tex1Idx and Tex2Idx) and how they should be oriented (Tex1Dir and Tex2Dir).

Next we read in the patch information.

float** m_ppPatchMiddleY = new float*[m_pat_MapSize];
float** m_ppPatchRadius = new float*[m_pat_MapSize];

for (int x = 0; x<m_pat_MapSize; x++) {
	m_ppPatchMiddleY[x] = new float[m_pat_MapSize];
	m_ppPatchRadius[x] = new float[m_pat_MapSize];

	for (int z = 0; z<m_pat_MapSize; z++) {
		fread(&(m_ppPatchMiddleY[x][z]), sizeof(float), 1, fpMap);
		fread(&(m_ppPatchRadius[x][z]), sizeof(float), 1, fpMap);
	}
}

This simply gives us how high each 8×8 patch is and the radius of its bounding sphere. Next we’ll read in the grass information.

unsigned char* m_pGrassAttr = new unsigned char[m_ti_MapSize*m_ti_MapSize];
fread(m_pGrassAttr, sizeof(unsigned char), m_ti_MapSize*m_ti_MapSize, fp);

char* m_pGrassFileName = new char[260];
fread(m_pGrassFileName, sizeof(char), 260, fp);

This information is used for rendering the grass which pops up out of the tile. Then we read in information about the textures which we will use for each of the tiles on the map.

int m_NumTileTex;
fread(&m_NumTileTex, sizeof(int), 1, fp);

int NumTileTexSrc;
fread(&NumTileTexSrc, sizeof(int), 1, fp);

char** SrcName = new char*[NumTileTexSrc];
for (int i = 0; i<NumTileTexSrc; i++) {
	SrcName[i] = new char[260];
	fread(SrcName[i], 260, 1, fp);
}

short SrcIdx, TileIdx;
for (int i = 0; i<m_NumTileTex; i++) {
	fread(&SrcIdx, sizeof(short), 1, fp);
	fread(&TileIdx, sizeof(short), 1, fp);
}

m_NumTileTex is the number of textures which get used for this particular map. NumTileTexSrc is the number of files from which all these textures will be loaded from. SrcIdx is the index into the source name array for which this texture is located and TileIdx is the index of this texture within the file. Basically the DTEX folder has a bunch of files in it and each of these files contains a bunch of textures put together one by one. In order to get one of these textures you first must know which file to look into and then how far into that file to look in order  to get your texture.

Here are the last few variables we need.

int NumLightMap;
fread(&NumLightMap, sizeof(int), 1, fp);

int m_iRiverCount;
fread(&m_iRiverCount, sizeof(int), 1, fp);

int m_iPondMeshNum;
fread(&m_iPondMeshNum, sizeof(int), 1, fp);

If you have a few rivers or ponds on the map then each files have to be loaded and looked through.

This is the basic structure of the 1299 .gtd file. If you would like all the code along with a simplistic OpenGL implementation of the rendering go here.

Getting setup with the OpenKO project

I recently made a video on how to get setup with the OpenKO source code and another video explaining the basic structure and history of the codebase. Feel free to take the time and get the code running! If you have any questions feel free to send me an email (info@stephenmeier.net).

This second video explains a little about where the source code originates from and what major changes have taken place since I first started working on everything.

Kodev – Updating the UIQuestMenu

I recently made a video explaining how the quest menu GUI works for the game Knight Online. This is the first part in a two part series where we look at why these GUI components work the way they do and how we can use the GUI editor to help us to implement functionality for later versions. Part two for this series has already been recorded I just need to upload it! I hope to do that in the next week or so. If you have any questions or would like to suggest future topics for videos please email me at “info@stephenmeier.net”. Thanks!

EDIT: I was recently able to upload part 2! In the future I will try not to make the videos so long because they quickly become hard to upload.

 

SDL 2.0 Tutorial-06: Networking 3

In the last tutorial we setup a basic TCP server. In this tutorial we will start by modifying the server we setup and finish by writing a player client application. In order to write a client application with SDL 2.0 we will need a lot of the ideas which where presented in my first string of SDL tutorials (here). A lot of my personal preferences for setting up SDL have changed since writing those tutorials so instead of using code found within those tutorials I will be using a collection of C++ files I have settled into using (found here).

First let’s start with a few server-side modifications. Let’s start by adding a few extra packet flags:

//-----------------------------------------------------------------------------
#define FLAG_QUIT 0x0000
#define FLAG_WOOD_QUEST 0x0011
#define FLAG_WOOD_UPDATE 0x0010
#define FLAG_WOOD_GETTIME 0x0012

Now we have a packet flag for starting and completing a quest, as well as a packet flag for getting the amount of time left until the quest is completed. Next we will update our server-side “RecvData” function:

uint8_t* RecvData(int index, uint16_t* length) {
	uint8_t temp_data[MAX_PACKET];
	int num_recv = SDLNet_TCP_Recv(sockets[index], temp_data, MAX_PACKET);

	if(num_recv <= 0) {
		CloseSocket(index);

		const char* err = SDLNet_GetError();
		if(strlen(err) == 0) {
			printf("DB: client disconnectedn");
		} else {
			fprintf(stderr, "ER: SDLNet_TCP_Recv: %sn", err);
		}

		return NULL;
	} else {
		*length = num_recv;

		uint8_t* data = (uint8_t*) malloc(num_recv*sizeof(uint8_t));
		memcpy(data, temp_data, num_recv);

		return data;
	}
}

What I have done here is removed the work this function was doing to grab the packet flag from the start of the incoming socket data. This is because when a socket is ready for us to process it may have several packets all lined together. Therefore the data buffer returned by “RecvData” may contain several packets depending on how quickly the client is sending packets to us; so any work “RecvData” does to grab the flag from the first packet doesn’t help us identify the packet flags for all the other packets possibly following the first. Therefore the extra work being done isn’t necessary at this level.

Next we will add a new function called “ProcessData.” This function will continuously run on each packet we find within the data buffer returned by “RecvData.”

void ProcessData(int index, uint8_t* data, uint16_t* offset) {
	if(data == NULL) return;

	uint16_t flag = *(uint16_t*) &data[*offset];
	*offset += sizeof(uint16_t);

	switch(flag) {
		case FLAG_WOOD_UPDATE: {
			uint16_t send_offset = 0;
			uint8_t send_data[MAX_PACKET];

			memcpy(send_data+send_offset, &clients[index].amt_wood, sizeof(uint8_t));
			send_offset += sizeof(uint8_t);

			SendData(index, send_data, send_offset, FLAG_WOOD_UPDATE);
		} break;

		case FLAG_WOOD_GETTIME: {
			uint16_t send_offset = 0;
			uint8_t send_data[MAX_PACKET];

			uint32_t time_left;
			if(clients[index].questing) {
				time_left = WOOD_WAIT_TIME-(SDL_GetTicks()-clients[index].timer_wood);
			} else time_left = 0;

			memcpy(send_data+send_offset, &time_left, sizeof(uint32_t));
			send_offset += sizeof(uint32_t);

			SendData(index, send_data, send_offset, FLAG_WOOD_GETTIME);
		} break;

		case FLAG_WOOD_QUEST: {
			if(!clients[index].questing) {
				clients[index].questing = 1;
				clients[index].timer_wood = SDL_GetTicks();
			}
		} break;

		case FLAG_QUIT: {
			running = 0;
			printf("DB: shutdown by client id: %dn", index);
		} break;
	}
}

This is where we do the work and grab the packet’s flag. Everything here should look similar to what we where doing before when the server found client sockets which were ready with information to be processed – except for the fact that we have added two extra flags. The “GETTIME” flag checks if the player is currently questing; if not then the time left on the quest is 0, otherwise the server sends the time left on the quest to the player. The other flag is the “QUEST” flag; this flag checks whether the client is currently questing and starts the quest if they aren’t currently in a quest.

The finial server-side update we make will be to the main processing loop. First let’s crank up how often we check for ready sockets:

int num_rdy = SDLNet_CheckSockets(socket_set, 50);

If num_rdy <= 0 then we do not have any sockets ready for processing so we enter the server’s idle tasks. Previously we were simply adding four wood to the client’s resource count continuously. This is quite a silly thing to do! Instead let’s check for whether a client has completed any quests and if so we’ll send out a quest completed packet and a wood update packet. In addition, we’ll continuously send out “GETTIME” packets to keep the client updated with the server’s timer.

int ind;
for(ind=0; ind<MAX_SOCKETS; ++ind) {
	if(!clients[ind].in_use) continue;

	if(clients[ind].questing &&
		(SDL_GetTicks()-clients[ind].timer_wood)>WOOD_WAIT_TIME
	) {
		clients[ind].questing = 0;
		clients[ind].amt_wood += 4;
		SendData(ind, NULL, 0, FLAG_WOOD_QUEST);

		uint16_t send_offset = 0;
		uint8_t send_data[MAX_PACKET];

		memcpy(send_data+send_offset, &clients[ind].amt_wood, sizeof(uint8_t));
		send_offset += sizeof(uint8_t);

		SendData(ind, send_data, send_offset, FLAG_WOOD_UPDATE);
	}

	uint16_t send_offset = 0;
	uint8_t send_data[MAX_PACKET];

	uint32_t time_left;
	if(clients[ind].questing) {
		time_left = WOOD_WAIT_TIME-(SDL_GetTicks()-clients[ind].timer_wood);
	} else time_left = 0;

	memcpy(send_data+send_offset, &time_left, sizeof(uint32_t));
	send_offset += sizeof(uint32_t);

	SendData(ind, send_data, send_offset, FLAG_WOOD_GETTIME);
}

And for the final server touch we’ll update the code which runs if there are sockets ready to be processed. Notice that how we handle the server socket (for new connections) isn’t changing but how we handle the client sockets is changing.

if(SDLNet_SocketReady(server_socket)) {
	int got_socket = AcceptSocket(next_ind);
	if(!got_socket) {
		num_rdy--;
		continue;
	}

	// NOTE: get a new index
	int chk_count;
	for(chk_count=0; chk_count<MAX_SOCKETS; ++chk_count) {
		if(sockets[(next_ind+chk_count)%MAX_SOCKETS] == NULL) break;
	}

	next_ind = (next_ind+chk_count)%MAX_SOCKETS;
	printf("DB: new connection (next_ind = %d)n", next_ind);

	num_rdy--;
}

int ind;
for(ind=0; (ind<MAX_SOCKETS) && num_rdy; ++ind) {
	if(sockets[ind] == NULL) continue;
	if(!SDLNet_SocketReady(sockets[ind])) continue;

	uint8_t* data;
	uint16_t length;
				
	data = RecvData(ind, &length);
	if(data == NULL) {
		num_rdy--;
		continue;
	}

	int num_processed = 0;

	uint16_t offset = 0;
	while(offset < length) {
		num_processed++;
		ProcessData(ind, data, &offset);
	}

	printf("num_processed from ID: %d was %dn", ind, num_processed);
	fflush(stdout);

	free(data);
	num_rdy--;
}

I am also keeping track of the number of packets which the server processes for each of the ready clients for debugging purposes. Now the client-side stuff isn’t that different when it comes to the network side of things. A lot of the helper functions we have server-side will have analogous client-server versions. Here is the client-side code in its entirety:

/*
*/

#include "AdBase.h"
#include "AdLevel.h"
#include "AdScreen.h"
#include "AdSpriteManager.h"

//-----------------------------------------------------------------------------
#define MAX_PACKET 0xFF

//-----------------------------------------------------------------------------
#define FLAG_QUIT 0x0000
#define FLAG_WOOD_QUEST 0x0011
#define FLAG_WOOD_UPDATE 0x0010
#define FLAG_WOOD_GETTIME 0x0012

//-----------------------------------------------------------------------------
TCPsocket socket;
SDLNet_SocketSet socket_set;

//-----------------------------------------------------------------------------
int questing;
uint8_t amt_wood;
uint32_t timer_wood;

//-----------------------------------------------------------------------------
void CloseSocket(void) {
	if(SDLNet_TCP_DelSocket(socket_set, socket) == -1) {
		fprintf(stderr, "%sn", SDLNet_GetError());
		system("pause");
		exit(-1);
	}

	SDLNet_FreeSocketSet(socket_set);
	SDLNet_TCP_Close(socket);
}

//-----------------------------------------------------------------------------
void SendData(uint8_t* data, uint16_t length, uint16_t flag) {
	uint8_t temp_data[MAX_PACKET];

	int offset = 0;
	memcpy(temp_data+offset, &flag, sizeof(uint16_t));
	offset += sizeof(uint16_t);
	memcpy(temp_data+offset, data, length);
	offset += length;

	int num_sent = SDLNet_TCP_Send(socket, temp_data, offset);
	if(num_sent < offset) {
		fprintf(stderr, "ER: SDLNet_TCP_Send: %sn", SDLNet_GetError());
		CloseSocket();
	}
}

//-----------------------------------------------------------------------------
uint8_t* RecvData(uint16_t* length) {
	uint8_t temp_data[MAX_PACKET];
	int num_recv = SDLNet_TCP_Recv(socket, temp_data, MAX_PACKET);

	if(num_recv <= 0) {
		CloseSocket();

		const char* err = SDLNet_GetError();
		if(strlen(err) == 0) {
			printf("DB: server shutdownn");
		} else {
			fprintf(stderr, "ER: SDLNet_TCP_Recv: %sn", err);
		}

		return NULL;
	} else {
		*length = num_recv;

		uint8_t* data = (uint8_t*) malloc(num_recv*sizeof(uint8_t));
		memcpy(data, temp_data, num_recv);

		return data;
	}
}

//-----------------------------------------------------------------------------
void ProcessData(uint8_t* data, uint16_t* offset) {
	if(data == NULL) return;

	uint16_t flag = *(uint16_t*) &data[*offset];
	*offset += sizeof(uint16_t);

	switch(flag) {
		case FLAG_WOOD_UPDATE: {
			amt_wood = *(uint8_t*) &data[*offset];
			*offset += sizeof(uint8_t);
		} break;

		case FLAG_WOOD_GETTIME: {
			timer_wood = *(uint32_t*) &data[*offset];
			*offset += sizeof(uint32_t);
		} break;

		case FLAG_WOOD_QUEST: {
			// NOTE: quest completed
			questing = 0;
		} break;
	}
}

//-----------------------------------------------------------------------------
void InitNetwork(const char* pIP, int iPort) {
	IPaddress ip;
	if(SDLNet_ResolveHost(&ip, pIP, iPort) == -1) {
		fprintf(stderr, "%sn", SDLNet_GetError());
		system("pause");
		exit(-1);
	}

	socket = SDLNet_TCP_Open(&ip);
	if(socket == NULL) {
		fprintf(stderr, "%sn", SDLNet_GetError());
		system("pause");
		exit(-1);
	}

	socket_set = SDLNet_AllocSocketSet(1);
	if(socket_set == NULL) {
		fprintf(stderr, "%sn", SDLNet_GetError());
		system("pause");
		exit(-1);
	}

	if(SDLNet_TCP_AddSocket(socket_set, socket) == -1) {
		fprintf(stderr, "%sn", SDLNet_GetError());
		system("pause");
		exit(-1);
	}
}

//-----------------------------------------------------------------------------
bool CheckSocket(void) {
	if(SDLNet_CheckSockets(socket_set, 0) == -1) {
		fprintf(stderr, "%sn", SDLNet_GetError());
		system("pause");
		exit(-1);
	}

	return SDLNet_SocketReady(socket);
}

//-----------------------------------------------------------------------------
int SDL_main(int argc, char* argv[]) {
	if(AdBase::Init(8*40, 8*30, 3) == false) {
		fprintf(stderr, "ERROR: Failed to initiate.n");
		system("pause");
		return -1;
	}

	// TESTING
	InitNetwork("PUT_SERVER_IP_HERE!!!", 8099); // arg #2 is the port

	AdLevel* testLvl = new AdLevel();
	//

	SDL_Event sdlEvent = {};
	while(sdlEvent.type != SDL_QUIT) {
		SDL_PollEvent(&sdlEvent);

		AdScreen::Clear();

		// TESTING
		if(CheckSocket()) {
			uint16_t length, flag;
			uint8_t* data = RecvData(&length);

			uint16_t offset = 0;
			while(offset < length) {
				ProcessData(data, &offset);
			}

			free(data);
		}
		//

		// TESTING
		testLvl->Update(&sdlEvent);

		char string[0xFF];
		sprintf(string, "Wood: %d", amt_wood);

		SDL_Point pnt1 = {0, 0};
		SDL_Color color1 = {0xFF, 0x00, 0x00, 0x00};
		SDL_Surface* pSurf = AdSpriteManager::BuildSprite(string, color1);
		AdScreen::DrawSprite(pnt1, pSurf);
		SDL_FreeSurface(pSurf);

		pnt1.y += 8;
		sprintf(string, "Timer: %d", (int) ceil((double)timer_wood/1000.0f));
		pSurf = AdSpriteManager::BuildSprite(string, color1);
		AdScreen::DrawSprite(pnt1, pSurf);
		SDL_FreeSurface(pSurf);
		//

		//
		if(timer_wood == 0) {
			SDL_Point pnt2 = {128, 8};
			SDL_Rect rec2 = {pnt2.x, pnt2.y, 5*8, 8};

			if(
				testLvl->m_iMouseX>=rec2.x && testLvl->m_iMouseX<=(rec2.x+rec2.w) &&
				testLvl->m_iMouseY>=rec2.y && testLvl->m_iMouseY<=(rec2.y+rec2.h)
			) {
				SDL_Color color2 = {0x00, 0xFF, 0x00, 0x00};
				pSurf = AdSpriteManager::BuildSprite("Quest", color2);

				if(testLvl->m_bMouseLeft && !questing) {
					questing = 1;
					SendData(NULL, 0, FLAG_WOOD_QUEST);
				}
			} else {
				SDL_Color color2 = {0x00, 0x00, 0xFF, 0x00};
				pSurf = AdSpriteManager::BuildSprite("Quest", color2);
			}
			
			AdScreen::DrawSprite(pnt2, pSurf);
			SDL_FreeSurface(pSurf);
		}
		//

		AdScreen::Present();
	}

	// TESTING
	delete testLvl;

	SendData(NULL, 0, FLAG_QUIT);
	CloseSocket();
	//

	AdBase::Quit();

	return 0;
}

All the SDL graphics processing and input handling is being wrapped with the custom files I wrote and linked at the beginning of this post. If you find these posts helpful or would like me to dive deeper into a particular topic covered here please send me an email at “info@stephenmeier.net”.

SDL 2.0 Tutorial-05: Networking 2

In the last tutorial we finished with a hallow if-statement within our processing loop.

int running = 1;
while(running) {
	int num_rdy = SDLNet_CheckSockets(socket_set, 1000);

	if(num_rdy <= 0) {
		// NOTE: none of the sockets are ready
	} else {
		// NOTE: some number of the sockets are ready
	}
}

If there aren’t any sockets which we need to process (i.e. num_rdy <= 0 ) then we’ll loop over all the currently connected client sockets and perform any tasks which may be pending. This is basically the server’s chance to catch up with game state and perform any security checks. This will work well for a simple TCP game where the server and client need not be in sync the entire time. For something more real-time we would prioritize game state and perhaps use UDP instead.

For now let’s add four wood to the resource count for each connected client.

int ind;
for(ind=0; ind<MAX_SOCKETS; ++ind) {
	if(!clients[ind].in_use) continue;
	clients[ind].amt_wood += 4;
}

Remember this will run every second assuming there aren’t any sockets to be processed (i.e. num_rdy <= 0 ). Why am I saying it’ll run every second? It is because of the following function call:

SDLNet_CheckSockets(socket_set, 1000);

The second argument to this function is 1000, which is the number of milliseconds this function will block while waiting for a socket connection.

Now, if there are sockets ready to be processed we’ll have to check whether it is a client socket or the server socket. If the server socket is ready then we likely have a new client connection. For the client connections we’ll have to check each one until we find the ones which are ready and then process the data that we have received from them.

First let’s handle the case when the server socket is ready:

if(SDLNet_SocketReady(server_socket)) {
	int got_socket = AcceptSocket(next_ind);
	if(!got_socket) {
		num_rdy--;
		continue;
	}

	// NOTE: get a new index
	int chk_count;
	for(chk_count=0; chk_count<MAX_SOCKETS; ++chk_count) {
		if(sockets[(next_ind+chk_count)%MAX_SOCKETS] == NULL) break;
	}

	next_ind = (next_ind+chk_count)%MAX_SOCKETS;
	printf("DB: new connection (next_ind = %d)n", next_ind);

	num_rdy--;
}

Here the function int AcceptSocket(int index); is a shorthand for accepting a new socket connection using sockets[index] and clients[index] . This function returns whether or not the connection was successfully accepted. Here is the function itself:

int AcceptSocket(int index) {
	if(sockets[index]) {
		fprintf(stderr, "ER: Overriding socket at index %d.n", index);
		CloseSocket(index);
	}

	sockets[index] = SDLNet_TCP_Accept(server_socket);
	if(sockets[index] == NULL) return 0;

	clients[index].in_use = 1;
	if(SDLNet_TCP_AddSocket(socket_set, sockets[index]) == -1) {
		fprintf(stderr, "ER: SDLNet_TCP_AddSocket: %sn", SDLNet_GetError());
		exit(-1);
	}

	return 1;
}

The function void CloseSocket(int index); was presented in the previous part of this tutorial. After we accept the new client connection we also update the next_ind variable so that we are ready for the next client connection. As it’s currently written, if a new client connects when the server is full then another client will be kicked to make room for the new connection.

Now let’s process any clients which might have data for us.

int ind;
for(ind=0; (ind<MAX_SOCKETS) && num_rdy; ++ind) {
	if(sockets[ind] == NULL) continue;
	if(!SDLNet_SocketReady(sockets[ind])) continue;

	uint8_t* data;
	uint16_t flag;
	uint16_t length;
				
	data = RecvData(ind, &length, &flag);
	if(data == NULL) {
		num_rdy--;
		continue;
	}

	switch(flag) {
		case FLAG_WOOD_UPDATE: {
			uint16_t offset = 0;
			uint8_t send_data[MAX_PACKET];

			memcpy(send_data+offset, &clients[ind].amt_wood, sizeof(uint8_t));
			offset += sizeof(uint8_t);

			SendData(ind, send_data, offset, FLAG_WOOD_UPDATE);
		} break;

		case FLAG_QUIT: {
			running = 0;
			printf("DB: shutdown by client id: %dn", ind);
		} break;
	}

	free(data);
	num_rdy--;
}

What we are doing here is looping over all the client sockets and checking to see whether a client is connected and whether that connected client has some information ready for us. We also want to make sure that we aren’t working harder than we have to by breaking out of the for-loop when num_rdy == 0 ; because at that point we have processed all the ready client sockets so there is no point in checking any remaining client connections for information.

Once we know that we have a connected socket and that the socket have information ready for us, we fill a buffer with the data they sent us. This is what the uint8_t* RecvData(int index, uint16_t* length, uint16_t* flag); function does for us:

uint8_t* RecvData(int index, uint16_t* length, uint16_t* flag) {
	uint8_t temp_data[MAX_PACKET];
	int num_recv = SDLNet_TCP_Recv(sockets[index], temp_data, MAX_PACKET);

	if(num_recv <= 0) {
		CloseSocket(index);
		const char* err = SDLNet_GetError();
		if(strlen(err) == 0) {
			printf("DB: client disconnectedn");
		} else {
			fprintf(stderr, "ER: SDLNet_TCP_Recv: %sn", err);
		}

		return NULL;
	} else {
		int offset = 0;
		*flag = *(uint16_t*) &temp_data[offset];
		offset += sizeof(uint16_t);

		*length = (num_recv-offset);

		uint8_t* data = (uint8_t*) malloc((num_recv-offset)*sizeof(uint8_t));
		memcpy(data, &temp_data[offset], (num_recv-offset));

		return data;
	}
}

This function gets data from sockets[index] and returns a pointer to a new buffer. The in/out function arguments length and flag are the length of the newly allocated buffer and the packet flag for the information we are receiving from the client.

If the client sends us a request for the amount of wood they current possess (i.e. FLAG_WOOD_UPDATE ) then we reply by sending them a packet with the information they are requesting. To send information to the connected client we use the following shorthand function:

void SendData(int index, uint8_t* data, uint16_t length, uint16_t flag) {
	uint8_t temp_data[MAX_PACKET];

	int offset = 0;
	memcpy(temp_data+offset, &flag, sizeof(uint16_t));
	offset += sizeof(uint16_t);
	memcpy(temp_data+offset, data, length);
	offset += length;

	int num_sent = SDLNet_TCP_Send(sockets[index], temp_data, offset);
	if(num_sent < offset) {
		fprintf(stderr, "ER: SDLNet_TCP_Send: %sn", SDLNet_GetError());
		CloseSocket(index);
	}
}

For debugging purposes the client can also shutdown the server with a FLAG_QUIT . In the next tutorial we will write a simple client which will display the amount of wood the client current has. This amount will continue to increase because the server currently adds four wood each time it enters idle processing. Once we have achieved this we’ll add a “quest” button and timer on the client-side and a few more packet flags allowing the client to request the amount of time left on a quest and to start a quest. The server will check for when a quest completes and will increase the amount of wood for that client as a result.

Till next time!

SDL 2.0 Tutorial-04: Networking 1

For the next few posts I plan to run through some of the basics of TCP communication using the SDLNet library. The goal here is to write a simple “resource” game where the client sends a “quest” request and, after some amount of time has passed, the quest is considered complete and the client’s resource count increases a little. This series will provide an excellent setup for developing a player driven economy type game.

Our first goal will be to write a server which manages all the client connections. To keep things simple we will be working with one single resource; here we are using wood as the resource. We’ll start with a few #define ‘s

//-----------------------------------------------------------------------------
#define MAX_PACKET 0xFF
#define MAX_SOCKETS 0x10

//-----------------------------------------------------------------------------
#define WOOD_WAIT_TIME 5000

//-----------------------------------------------------------------------------
#define FLAG_QUIT 0x0000
#define FLAG_WOOD_UPDATE 0x0010

Here we have three different type of #define ‘s

  • MAX_PACKET and MAX_SOCKETS define the maximum size of the TCP packets we are willing to process and the maximum number of clients we will allow online at once.
  • WOOD_WAIT_TIME is the amount of time in milliseconds to wait before the quest for obtaining additional wood units is complete.
  • FLAG_QUIT and FLAG_WOOD_UPDATE are 16 bit packet flags used to determine the type of packet we are receiving and we then process the rest of the packet’s data based on this flag.

Next we need a structure which we can use to keep all the client’s information separate from one another.

typedef struct {
	int in_use;
	int questing;
	uint8_t amt_wood;
	uint32_t timer_wood;
} Client;

Here we have a few important variables which we need in order to efficiently work with each of the clients which connect to the server. in_use allows us to determine whether the current client is being used by an active socket connection or not. This will prevent us from running checks on client structures which don’t have corresponding network connections. questing is a simple boolean which tells us whether the current client is running a wood fetching quest or not. amt_wood is the amount of the wood resource which the current client has. timer_wood is a timestamp for when the wood fetching quest for this particular client was started – this will help us determine when the wood fetching quest has been completed.

Next we need some global state variables which will make writing our little server a bit easier.

int next_ind = 0;
TCPsocket server_socket;
Client clients[MAX_SOCKETS];
SDLNet_SocketSet socket_set;
TCPsocket sockets[MAX_SOCKETS];

next_ind is the index into the sockets and clients arrays for the next player that will connect to the server. When one client connects next_ind gets updated so that we never overwrite a connected clients information.

First thing we do in int main(int argc, char** argv); is initialize a few SDL components and the SDLNet library.

if(SDL_Init(SDL_INIT_TIMER|SDL_INIT_EVENTS) != 0) {
	fprintf(stderr, "ER: SDL_Init: %sn", SDL_GetError());
	exit(-1);
}

if(SDLNet_Init() == -1) {
	fprintf(stderr, "ER: SDLNet_Init: %sn", SDLNet_GetError());
	exit(-1);
}

Next we’ll want to open the server’s TCP socket on a specific port, here I’m using port 8099. Any function specific information can be found within the SDLNet online API.

IPaddress ip;
if(SDLNet_ResolveHost(&ip, NULL, 8099) == -1) {
	fprintf(stderr, "ER: SDLNet_ResolveHost: %sn", SDLNet_GetError());
	exit(-1);
}

server_socket = SDLNet_TCP_Open(&ip);
if(server_socket == NULL) {
	fprintf(stderr, "ER: SDLNet_TCP_Open: %sn", SDLNet_GetError());
	exit(-1);
}

Finally, before the main processing loop, we will setup an SDLNet socket set and add the server_socket to the socket set so that we may be informed when a net connection is looking to be accepted by the server.

socket_set = SDLNet_AllocSocketSet(MAX_SOCKETS+1);
if(socket_set == NULL) {
	fprintf(stderr, "ER: SDLNet_AllocSocketSet: %sn", SDLNet_GetError());
	exit(-1);
}

if(SDLNet_TCP_AddSocket(socket_set, server_socket) == -1) {
	fprintf(stderr, "ER: SDLNet_TCP_AddSocket: %sn", SDLNet_GetError());
	exit(-1);
}

It’s worth noting that the size of the socket set is MAX_SOCKETS+1 , the plus one accounts for the addition of the server socket. The main processing loop is setup like the following (currently hollow):

int running = 1;
while(running) {
	int num_rdy = SDLNet_CheckSockets(socket_set, 1000);

	if(num_rdy <= 0) {
		// NOTE: none of the sockets are ready
	} else {
		// NOTE: some number of the sockets are ready
	}
}

While the server is running we check for any sockets from the socket set which might have packets for us to process; we check the socket set for a second (1000 milliseconds). If there aren’t any sockets ready to be processed then we use this time to run other various server activities like checking is quests have been completed, etc. If there is some number of sockets ready to be processed then we process those sockets.

In the next tutorial we will fill in these two parts of the if-statement above; this is where the real magic is! To clean up everything after the server is done running we will run the following code.

if(SDLNet_TCP_DelSocket(socket_set, server_socket) == -1) {
	fprintf(stderr, "ER: SDLNet_TCP_DelSocket: %sn", SDLNet_GetError());
	exit(-1);
} SDLNet_TCP_Close(server_socket);

int i;
for(i=0; i<MAX_SOCKETS; ++i) {
	if(sockets[i] == NULL) continue;
	CloseSocket(i);
}

SDLNet_FreeSocketSet(socket_set);
SDLNet_Quit();
SDL_Quit();

The missing piece here is the void CloseSocket(int index); function which is basically just a shorthand for closing a client socket and is stated below:

void CloseSocket(int index) {
	if(sockets[index] == NULL) {
		fprintf(stderr, "ER: Attempted to delete a NULL socket.n");
		return;
	}

	if(SDLNet_TCP_DelSocket(socket_set, sockets[index]) == -1) {
		fprintf(stderr, "ER: SDLNet_TCP_DelSocket: %sn", SDLNet_GetError());
		exit(-1);
	}

	memset(&clients[index], 0x00, sizeof(Client));
	SDLNet_TCP_Close(sockets[index]);
	sockets[index] = NULL;
}

For the next tutorial we’ll pick up from here and fill in the if-statement mentioned earlier. Once the server-side code has been setup to process a simple player request to perform a quest to gather wood, we’ll start working on a simple client-side quest button, quest timer, and resource counter. If there is time or interest we will add persistence to the resource count for each player using SQLite.