WebDownload

Sometimes you want to have your mod download something from a website to be used in game or something like that. Unreal gives you the ability to do that using a [TCPLink] class.

This document will describe how you can implement a basic HTTP client.

Note: You might want to take a look at LibHTTP, it's a general purpose library which contains a better HTTP client.

HTTP

This document is focused on using [HTTP/1.0] to retreive information, ofcourse it's possible to use any other protocol like [FTP], [QOTD (TCP)], etc. But you have to find out how to do that yourself, FTP requires a lot more work than HTTP does.

HTTP Requests

First some information about a HTTP requests works.

To get a file from a webserver you have send a query like this (without the line numbers ofcourse):

 001  GET /wiki/Recent_Changed HTTP/1.0
 002  Host: wiki.beyondunreal.com
 003  Connection: close
 004  User-agent: UT2003-webdownload (version 100; UT2003 version 2136; http://wiki.beyondunreal.com/WebDownload)
 005

First of the most important part of the query is line 001, this will request the document from the server. The location has to be the absolute path to the file, it may not contain spaces, to use spaces you have to URL escape them (%20 is a space), read the [RFC] for more details.

I use a HTTP/1.0 request because HTTP/1.1 will introduce some extra difficulties.

Another very important part of the request is line 002, this will define the host you want to connect to. You will connect on IP to the server, but you have to identify the server name you want to retreive files from. This is important because a lot of servers perform Virtual Hosting allowing diffirent servers on the same IP address.

Line 003 isn't realy that important since the default action of HTTP/1.0 is to close the connection, but better safe then sorry.

Line 004 is just usefull to identify the client, this string has to have the format of "client-name (extra info; extra info; ...)"

HTTP replies

After the HTTP request the server will reply with a HTTP reply and the requested document.

Here's an example HTTP reply (ignore the line numbers):

 001  HTTP/1.1 200 OK
 002  Date: Thu, 21 Nov 2002 20:57:34 GMT
 003  Server: Apache/1.3.27 (Unix) AuthMySQL/2.20 mod_gzip/1.3.19.1a PHP/4.2.3
 004  Connection: close
 005
 006  ... document data ...
 007  ... document data ...
 ...

From the above example only line 001 is intresting. Note that the server replies with a HTTP/1.1 reply, this doesn't matter much at this point, HTTP/1.1 and HTTP/1.0 are only important in the request. The code after HTTP/1.1 tells us what the server thought of our request. If the code is 200 everything is ok and the requested document is attachted in the reply. If the code is 404 it means that the page could not be found, check the RFC for the other codes.

A empty line (line 005 is the example) is the devider between the HTTP header and the document data.

Note about new lines

UNIX systems only use a newline character ('\n' or hex 0x0a) to identify a new line. Windows will use the combo carriage return + newline ('\r\n' , '\r' has hex value 0x0d). It's best to split lines on the newline and ignore the carriage return if it's there.

The Code

class WebDownload extends TCPLink config;

var config string sHostname;
var config int iPort;
var config string sRequest;

var private string buffer;

Well this is pretty much the default stuff, you have to decide for yourself how to use this class, the way I used it was from another class that provided the config options.

sHostname should contain the hostname and only the hostname, for example: sHostname = "wiki.beyondunreal.com";

iPort needs to be the port the webserver runs on, usualy 80

sRequest needs to be the absolute url of the document on that server, for example: sRequest = "/wiki/Recent_Changes";

buffer is the private variable that will contain the data retreived from the server.

function StartDownload ()
{
  Resolve(sHostname);
}

This is where we start, we first have to resolve the hostname, when the hostname resolves we will get a event. StartDownload has to be called from somewhere...

event ResolveFailed()
{
  log("Error, resolve failed");
}

event Resolved( IpAddr Addr )
{
  Addr.Port = iPort;
  BindPort();
  ReceiveMode = RMODE_Event;
  LinkMode = MODE_Line;
  Open(Addr);  
}

When the resolve fails we will receive a ResolveFailed event, you may want to do some stuff there

When the resolve is successfull we will get the event Resolved, from this point we will open the connection to the server.

Addr will contain the IP that has been resolved, we fill in the Port that we want to connect to, for HTTP is is 80 by default. Now we have to bind a local port, usualy you only bind a port when you want to listen to connections, but in the Unreal engine this is used to create a socket. Unreal can receive data via two ways, the first is by trying to read from the port, the other is by waiting for an event when there is data to be received. Ofcourse we want to receive events, thus we set the ReceiveMode' to RMODE_Event. After that we set the LinkMode to MODE_Line this will mean we will receive ReceivedLine events.

Now we will Open the connection to the remote server. When the open succeeds we will get Opened event.

event Opened()
{
  buffer = "";
  SendText("GET "$sRequest$" HTTP/1.0");
  SendText("Host: "$sHostname);
  SendText("Connection: close");
  SendText("User-agent: UT2003-webdownload (version 100; UT2003 version "$Level.EngineVersion$"; http://wiki.beyondunreal.com/WebDownload)");
  SendText("");
}

When the connection is open we will send the HTTP request, be sure to finish with a empty line. SendText will automatically add the trailing newlines.

Now we have sent the request we will get receive data from the Server.

event ReceivedLine( string Line )
{
  buffer = buffer$Line;
}

Just add the receiving data to the buffer, we don't want to process it yet, after the connection has closed we will do that.

event Closed()
{
  local array<string> lines;
  local string header;
  local int i;
  
  // split buff into header and doc
  if (divide(buffer, Chr(13)$Chr(10)$Chr(13)$Chr(10), header, buffer) == false) {
    // error no valid data
    return;
  };
  
  // do some header parsing
  class'wString'.static.split2(header, chr(10), lines);  // split on newlines (may still contain carriage returns)
  if (!class'wString'.static.MaskedCompare(lines[0], "HTTP/1.? 200*", false))
  {
    // error no valid data
    return;
  }
  for (i = 0; i < lines.length; i++)
  {
    if (class'wString'.static.MaskedCompare(lines[i], "Content-Type: *", false))
    {
      if (!class'wString'.static.MaskedCompare(lines[i], "content-type: text/plain*", false))
      {
        // only text/plain supported
        return;
      }
    }
  }

  // data should be correct
  class'wString'.static.(buffer, chr(10), lines);
  for (i = 0; i < lines.length; i++)
  {
    // cut off CR
    if (InStr(lines[i], Chr(13)) > -1)
    {
      lines[i] = Left(lines[i], InStr(lines[i], Chr(13)));
    }
    if (lines[i] != "")
    {
      // add the line to something
    }
  }
}

The event closed will be called when the connection gets close, since we specified Connection: close in the HTTP request the server will do it for us.

The buffer will contain the HTTP reply and the data, you may want to check the HTTP reply for status codes, and possibly for a new Location.

I've included an example on how to parse the data. It will first header the headers if the server returned a code 200, then it will check if the content type is text/plain. When these check out it will start to read data, since we split our data on a newline we have to check if there's a orphan carriage return, if so cut it off. If the resulting line is not empty we will use it.

Notice that I've used the MaskedCompare from wUtils.