Replacing LIVE functionality in XNA 4.0

Although the XNA framework supports multiple platforms, including Xbox and Windows on PC, there are parts that are not available on Windows PC.

One such part is dedicated to interacting with Xbox LIVE, such as gamer profile information, permissions, avatars. (Actually, these parts are available to developers, but not to users). That can be a problem if you want to release your Xbox game to PC. Even if you don't intend a commercial release and only want to distribute your game to your friends and family, they typically don't want to install the full SDK just so that they can run your game.

Before XNA Game Studio 4.0, if you wanted to support the PC platform, you had to write an abstraction layer over these restricted features, or sprinkle your code with #ifdef. There was no way you could replace the implementation of restricted features with your own as the default implementation was embedded into the same DLLs as non-restricted parts of XNA (and you don't want to reimplement these).

The release of XNA Game Studio 4.0 saw a clean-up of the organization of all assemblies. All restricted features are now confined to specific DLLs, which you can replace with your own.

I have started work on my own implementation. My level of ambition is not pretty low, I only intend to achieve the kind of coverage I need for my own code to compile and run. In any case, it's available on bitbucket, so feel free to grab it, modify and extend it to your liking.

I have started with Microsoft.Xna.Framework.Storage, as this is what I use in the code I mentioned in a previous post about safe IO on Xbox. Here are some random observations and small things I learned in the process.

References to record types can't be null. The XNA framework is primarily meant for C#, and it uses null (to the delight of generations of evil peer-reviewers). I initially used records for StorageDevice and StorageContainer, and I had to change to classes when I tried my replacement with existing game code. Even F# classes aren't null-friendly by default, you have to mark classes with a special attribute to change that:

[<AllowNullLiteral>]
type StorageDevice =
    class
        val drive : DriveInfo
        val path : string
        new (d, p) = { drive = d; path = p }
    end 

Separating methods from data in class declarations help avoid circular dependencies. StorageDevice depends on StorageContainer because a method of StorageDevice is used to create instances of StorageContainer. StorageContainer depends on StorageDevice because each StorageContainer is owned by a StorageDevice.
It would seem there is a circular dependency, which in F# must be dealt with explicitly. A simple solution consists of declaring the types as mutually dependent (using "and" instead of "type" for the second type). Another one is to move the definition of StorageDevice.BeginOpenContainer and StorageDevice.EndOpenContainer after StorageContainer.

[<AllowNullLiteral>]
type StorageDevice =
    class
        val drive : DriveInfo
        val path : string
        new (d, p) = { drive = d; path = p }
    end


[<AllowNullLiteral>]
type StorageContainer =
    class
        val name : string
        val device : StorageDevice
        val mutable is_disposed : bool
    
        new(name, dev) = { name = name; device = dev; is_disposed = false }
    end
with interface IDisposable with
        member this.Dispose() = this.is_disposed <- true

type StorageDevice with
    member this.BeginOpenContainer(displayName : string, cb : AsyncCallback, state : Object) =
        let f = new Task<_>(StorageInternals.openContainer(this, displayName), state)

        StorageInternals.doThenMaybeCallback(f, cb)

        f :> IAsyncResult

    member this.EndOpenContainer(result : IAsyncResult) =
        let task = result :?> Task<StorageContainer>
        task.Wait()
        task.Result

Not everything needs to be in a module. By default, each new F# source file starts with

module Module1.fs

You can use "namespace" instead:

namespace Microsoft.Xna.Framework.Storage

If you need F# functions at the root-level (i.e. outside a class or a method), you can start a module right in the middle of your source file:

type StorageContainer =
    class
        val name : string
        val device : StorageDevice
        val mutable is_disposed : bool
    
        new(name, dev) = { name = name; device = dev; is_disposed = false }
    end
with interface IDisposable with
        member this.Dispose() = this.is_disposed <- true


module StorageInternals =
    let checkConnected(dev : StorageDevice) =
        if not dev.drive.IsReady then raise(StorageDeviceNotConnectedException())

type StorageContainer with

    member this.CreateDirectory(dir) =
        StorageInternals.checkConnected(this.device)
        let path = Path.Combine(this.device.path, this.name, dir)
        Directory.CreateDirectory(path) |> ignore 

Windows.Forms.FolderBrowserDialog isn't very useful. I was initially using it to let the user select the StorageDevice and StorageContainer, but it turned out I can't use it in the main thread because it's modal, and I can't use in a separate thread because I can't. Actually, that's not true, I can use a separate thread if I create it myself using some special options. It the end, that's not what I did. I just rolled up my own browser using a TreeView.



How do I create an IAsyncResult? There may be many different solutions, but an easy one that work for me was to use a Task (they implement IAsyncResult).

let f = new Task<_>(StorageInternals.openContainer(this, displayName), state)

F# Asyncs are nice for GUI. The following code shows the browser on the right thread (assuming you can get hold of the Threading.SynchronizationContext of the GUI thread), and disposes of the dialog nicely.

let openContainer(dev : StorageDevice, name) _ =
        let dialog = new FolderBrowser(sprintf "Select container directory [%s]" name, Some dev.path)
        async {
            try
                do! Async.SwitchToContext(gui_context)
                dialog.Show()
                let! _ = Async.AwaitEvent(dialog.Closed)
                return
                    match dialog.Selected with
                    | Some path ->
                        if not(Directory.Exists(path)) then
                            Directory.CreateDirectory(path) |> ignore
                        new StorageContainer(name, dev)
                    | None ->
                        null
            finally
                dialog.Dispose()
        }
        |> Async.RunSynchronously

Playing and testing with your library code is easy, just use an F# script. No need for an extra project building an executable.



That's all for this article. Lots of small interesting details packed in little code.

Why we need better garbage collection on the Xbox

George Clingerman started a discussion on the App Hub where he asks developers what they would like George and other MVPs to discuss with the XNA dev team.

I took the chance to request that improvements be made to the garbage collector (GC for short) on Xbox. What's wrong with the current one, you may ask? Unlike GC on the PC platform, the Xbox GC is non-generational. A generational (also called ephemeral) GC distinguishes between newly allocated objects and older objects. Reclaiming memory early from "young" objects makes it possible to diminish the frequency of reclaiming memory from older objects, because many young objects are short-lived. This is important as reclaiming memory from older objects is an expensive operation.

To work around this limitation, Shawn Hargreaves has written articles on how to minimize or avoid altogether garbage collection during gameplay phases which must run at a constant 60 fps. To summarize, there are two ways: Make sure each collection is fast (the so-called low latency path) or avoid allocating memory to avoid triggering garbage collections.

The good thing with the first approach is that you don't need to worry much about allocations that happen "behind your back", i.e. allocations that are introduced by the compiler or the core library. The bad thing is that you must avoid complex object graphs. Note that this includes even objects allocated outside of the performance-critical parts of your code, such as menu code, art assets...

The second approach is by far the most popular. It works well with object-oriented designs which rely on mutability. This approach also has problems. Developers must avoid all kinds of allocation at all time during gameplay, which implies refraining from using many language constructs. This includes events, delegates and lexical closures in C#, tuples and closures in F#.

Having two options to avoid performance drops due to GC sounds nice until you realize you can't combine the two. You have to choose one or the other, but you can't mix the two. That's sad, because in practice an attractive design for games is to use OOP and mutability for menus and low-level array-based computations for gameplay.

And this is why we need better garbage-collection on the Xbox.

Death by exception

The mini-OS I developed for cooperative multi-tasking did not support killing tasks at first. This limitation made it a bit tricky to implement a specific feature I wanted to include in the small game I am developing to demonstrate the library's features. The feature in question consists of sending the player back to the "log-in" screen when the player signs out.

My first attempt simply checked if the player had signed out, and if that was the case, the current screen was exited as if the player had aborted. This was possible because every screen supports abortion. The problem with this approach is that all animations and sound effects associated to screen transitions would be played at once.

To avoid that problem, I could have added a special abortion path where all these effects are not played, but it did not feel right.

Another alternative is to "restart" the game, i.e. kill all tasks and let the top level reinitialize the game. Instant death is actually easy: it's a matter of not calling Scheduler.RunFor and discarding the scheduler, replacing it by a new empty one.

There is a problem, though. The screen manager will still have references to the screens, only the tasks have been killed. This particular problem is easy solve by the addition of a RemoveAll method, but there is more to it. The real problem is that clean-up code in tasks is never executed.

The definitive solution requires all clean-up code to be located in finally blocks, or in Dispose methods of IDisposable objects bound with "use" or "using". That's where all clean-up code should be anyway. The code below shows a new method in ScreenBase which makes it easy to add a screen, do something the remove the screen.

type ScreenManager(game, ui_content_provider : IUiContentProvider) =
    ...
    member this.AddDoRemove(s : Screen, t : Eventually<'T>) = task {
        try
            this.AddScreen(s)
            return! t
        finally
            this.RemoveScreen(s)
    }

Once this condition is met, the process of killing can be implemented by throwing an exception. The environment is given a new field indicating whether killing is going on. All system calls check this field before and after yielding or sleeping, and raise a specific exception if needed.

exception TaskKilled of obj

type Environment(scheduler : Scheduler) =
    // When killing all tasks, the data to embed in TaskKilled exceptions.
    let killing_data : obj option ref = ref None

    let checkAndKill = task {
        match !killing_data with
        | Some o -> raise(TaskKilled(o))
        | None -> ()
    }

    member this.StartKillAll(data) =
        killing_data := Some data
        scheduler.WakeAll()

    member this.StopKillAll() =
        killing_data := None

    member this.Wait dt = task {
        do! checkAndKill
        do! wait dt
        do! checkAndKill
    }

Using an exception makes it possible to survive a killing attempt by catching the exception and discarding it. There are two ways to go at this point: Either refrain from doing such a thing (in which case one should never ever catch and discard all exceptions), or embrace the idea and use it for "targeted killing". The exception thrown during killing (TaskKilled) carries an object which can be used in any way programmers see fit. It can for instance be a predicate which when evaluated indicates if the exception should be rethrown or discarded. Beware though: Tasks wake up early from long sleeps when they are killed. If the task is meant to survive, it's up to the programmer to make sure the task "goes back to bed".

The last piece in the puzzle is to detect sign-outs and react by killing and reinitializing the game. This process should not be enabled at all time though, it depends in what "top state" the game is. What I call "top state" here is an abstract view of the game state. See the code below for the complete list of states:

type TopState =
    | Initializing
    | AtPressStartScreen
    | AnonPlayer
    | Player of SignedInGamer
    | KillingAllTasks

State AnonPlayer is active when a player is playing the game (i.e. has press "start" on the "press start screen") without being signed in.
State Player corresponds to a signed-in player playing the game.

A typical flow is Initializing -> AtPressStartScreen -> Player or AnonPlayer -> AtPressStartScreen...
Another flow involving signing out: Initializing -> AtPressStartScreen -> Player or AnonPlayer-> KillingAllTasks -> Initializing -> ...

The code below updates the state machine.

type TopState =
    | Initializing
    | AtPressStartScreen
    | AnonPlayer
    | Player of SignedInGamer
    | KillingAllTasks
with
    member this.Update(transition) =
        match this, transition with
        | Initializing, InitDone -> AtPressStartScreen
        | Initializing, _ -> invalidOp "Invalid transition from Initializing"

        | AtPressStartScreen, AnonPressedStart -> AnonPlayer
        | AtPressStartScreen, PlayerPressedStart(p) -> Player p
        | AtPressStartScreen, _ -> invalidOp "Invalid transition from AtPressStartScreen"

        | AnonPlayer, Back -> AtPressStartScreen
        | AnonPlayer, _ -> invalidOp "Invalid transition from AnonPlayer"

        | Player p, SignOut -> KillingAllTasks
        | Player p, Back -> AtPressStartScreen
        | Player p, _ -> invalidOp "Invalid transition from Player"

        | KillingAllTasks, AllTasksKilled -> Initializing
        | KillingAllTasks, _ -> invalidOp "Invalid transition from KillingAllTasks"

and TopStateTransition =
    | InitDone
    | AnonPressedStart
    | PlayerPressedStart of SignedInGamer
    | SignOut
    | AllTasksKilled
    | Back

See how I used parallel pattern-matching? Whenever I have to write this kind of code in C# I find myself swearing silently...

Finally, the piece of code doing the dirty business.

// Initialization and killing of tasks.
        // Killing happens when a signed in player signs out.
        // Initialization happens during start-up and after killing.
        match !top_state with
        | Initializing ->
            scheduler.AddTask(main_task)
            top_state := (!top_state).Update(InitDone)
        | Player p when not(Gamer.IsSignedIn(p.PlayerIndex)) ->
            top_state := (!top_state).Update(SignOut)
            sys.StartKillAll(null)
        | KillingAllTasks when not(scheduler.HasLiveTasks) ->
            sys.StopKillAll()
            top_state := (!top_state).Update(AllTasksKilled)
            // Ideally, screen removal is done in finally handlers, and
            // killing should take care of removing all screens.
            // Nevertheless, we remove all screens here to be on the safe side.
            screen_manager.RemoveAllScreens()
        | _ -> ()

Game state management using cooperative multitasking

The game state management sample on the App Hub shows how to organize a game in screens. I strongly recommend all new-comers to game programming to study it, regardless of their amount of programming experience. Modern graphical non-game applications seldom follow this pattern, with the notable exception of step-by-step wizards.

The overview of this sample states:

The sample implements a simple game flow with a main menu, an options screen, some actual gameplay, and a pause menu. It displays a loading screen between the menus and gameplay, and uses a popup message box to confirm whether the user really wants to quit.
The ScreenManager class is a reusable component that maintains a stack of one or more GameScreen instances. It coordinates the transitions from one screen to another, and takes care of routing user input to whichever screen is on top of the stack.

A typical problem with this sample is that it is sometimes difficult to handle transitions from a screen to the next screen in the game flow.
For me, it's easier to specify the flow of screens from a bird's view:

The game starts with the press start screen, then the main menu is showed. Depending on the entry selected, the game may go into actual gameplay, show a credits screen...

When you implement your game, this bird view is not available by default. Instead, it's up to each screen to tell the screen manager which is the next screen to show when it's time for the first screen to remove itself. It's not unlike continuation-passing-style, in a way.

It is possible to do the inter-screen wiring from the top using events and handlers, but I find that using events and handlers for workflows is a bit ugly. The introduction of async in C# 5 indicates I'm not alone thinking that way.

To make things worse, providing the next screen to show with data it needs to initialize itself can get tricky if not carefully planned during the design phase. It also tends to cause "field overgrowth" in screen classes.

For all these reasons, I've been looking for a nicer solution using Eventually workflows and cooperative multitasking.

From the original game state management I have kept screens and the screen manager (on a conceptual level, implementation is new). These encapsulate state well, and I like the decomposition of a game into individual screens.

In particular, screens each have their ContentManager and load specific content when added to the screen manager, and release assets when removed. Sharing a ContentManager and keeping assets in memory is of course still possible using conventional methods.

The significant change is the removal of Update() and HandleInput() methods from screens. Instead, each screen has a number of Eventually computation expressions (which I also call tasks) which implement the logic of the screen.

To give a more concrete picture of the whole thing, here are bits from the MenuScreen. For full code, see github.

type MenuScreen<'I when 'I : equality>(
   player : PlayerIndex,
   sys : Environment,
   items : ('I * string)[],
   anim : AnimationParameters,
   placement : PlacementParameters) =

'I is a type parameter used to distinguish between entries. For instance, this discriminated union can be used for the main menu of a typical game:

type MainMenuEntries =
    | Play
    | Instructions
    | Options
    | Scores
    | Credits
    | BuyFullVersion
    | Exit

Back to MenuScreen, each constructor argument has the following role:

• player is the index of the player who has control, i.e. the player who pressed "start" on the "press start screen"
• sys is the object used to interact with the scheduler, which is used to spawn new tasks, wait, yield control...
• items is a list of (menu entry - text) pairs
• anim is a record whose fields control fade-in and fade-out effects
• placement is another records controlling where the menu is rendered

inherit ScreenBase()

    // A mutable cell holding the index of the entry currently selected.
    let current = ref 0

    // An object wrapping a number of variables controlling fading effects.
    let animation = new Animations.MultipleFadeIn(sys, items.Length, anim.period, anim.shift)

    // Keeps track of button presses on the gamepad of the player with control
    let input = new InputChanges.InputChanges(player)

    // Array keeping track of the state of visibility of each entry.
    // Typically entries are visible, but some must be hidden or showed as gray
    // depending on whether the user is running the full version of the game
    // or the trial.
    let visibility = items |> Array.map (fun _ -> Visible)

    do if items.Length = 0 then invalidArg "items" "items may not be empty"

In the full code functions dealing with menu entry visibility follow, I'm skipping them in this article.

The really interesting part (IMHO) comes now:

member this.Task = task {
        this.SetDrawer(this.Drawer)

        let animator = sys.Spawn(animation.Task)

        let selected = ref false
        let backed = ref false
        while not (!selected || !backed) do
            // If this screen is not active, i.e. it is not on top or the guide is visible, wait.
            // We don't want to react to input that's not for us.
            do! sys.WaitUntil(fun () -> this.IsOnTop)

            input.Update()

            if input.IsMenuDown() then moveDown()
            elif input.IsMenuUp() then moveUp()
            elif input.IsStartPressed() then selected := true
            elif input.IsBackPressed() then backed := true

            do! sys.WaitNextFrame()

        animator.Kill()
        do! sys.WaitUntil(fun() -> animator.IsDead)

        return
            if !selected then items.[!current] |> fst |> Some
            else None
    }

If you are not familiar with F#, it's worth pointing out that "!selected" means the value of mutable cell "selected", not the negation of the value of variable "selected". I keep getting confused by that when I read F# code even today.

If you have read my earlier articles about the Eventually workflow and cooperative multi-tasking, it should be clear what this code does. Critics may point out that it does not differ much from the typical content of HandleInput in the C# game state management sample, and they would be right.
A small but important difference resides in the last lines, composed of a return statement.

Interacting with the rest of the program is greatly simplified, as shown by this code snippet:

use menu =
            new MenuScreen<_>(
                controlling_player,
                sys,
                [| Play, "Play now"
                   Instructions, "How to play"
                   Options, "Options"
                   Scores, "Scores"
                   Credits, "More information"
                   BuyFullVersion, "BuyFullVersion"
                   Exit, "Exit" |],
                menu_animation,
                menu_placement
            )

        // Show the menu
        screen_manager.AddScreen(menu)
        // Execute the menu's code, and get the selected item as a result
        let! action = menu.Task
        // We are done with the menu, hide it.
        screen_manager.RemoveScreen(menu)

        // Deal with the selection in the menu.
        match action with        
        // Back to the board.
        | Some Exit ->
            exit_game := true

If you are like me, you may have experienced that dealing with seemingly trivial tasks such as showing and saving highscores after "game over" is more pain than it should be. It involves multiple screen transitions and asynchronous file I/O that must be spread over multiple update cycles.

Here how it looks in F#:

// Deal with the selection in the menu.
match action with        
// Back to the board.
| Some Exit ->
    exit_game := true
        
// Start playing.
| Some Play ->
    // Create the screen showing the game.
    use gameplay = new GameplayScreen(sys, controlling_player)

    // Show the gameplay screen. Gameplay itself is in gameplay.Task
    let! gameplay_result = screen_manager.AddDoRemove(gameplay, gameplay.Task)

    // If the game wasn't aborted, and if a new high score was achieved,
    // add it to the score table and show the table.
    match gameplay_result with
    | Aborted _ -> ()
    | TooEarly (_, points) | TooLate (_, points) ->
        use results = new ResultScreen(sys, controlling_player, gameplay_result)
        do! screen_manager.AddDoRemove(results, results.Task)

        let player_name =
            match SignedInGamer.SignedInGamers.ItemOpt(controlling_player) with
            | None -> "Unknown"
            | Some player -> player.Gamertag

        let is_hiscore = (!scores).AddScore(player_name, points)

        if is_hiscore then
            // save the scores.
            if storage.TitleStorage.IsSome then
                do! storage.CheckTitleStorage
                let! result =
                    storage.DoTitleStorage(
                       score_container,
                       saveXml score_filename !scores)

                match result with
                | Some(Some()) -> ()
                | _ -> do! doOnGuide <| fun() -> error "Failed to save scores"
            // Show the scores screen.
            do! showScores

There is a lot more to be said, but that's enough for a single article. I think that these code extracts show how F# can simplify the development of games. If you are a C# programmer used to think "functional programming is too complicated for what I need", I hope I have managed to introduce the idea there are clear benefits to be gained by introducing F# in your toolbox.

This is still work in progress, you can follow it on bitbucket.
You can also follow me on twitter, I am @deneuxj there. The full source code of a small game demonstrating task-based game state management is also available on github, under Samples/CoopMultiTasking.
This game demonstrates the following features:

• A typical "press start screen -> menu -> game" flow
• Safe IO using StorageDevice, used for best scores and user preferences
• Throwing back to the "press start screen" when sign-outs occur
• Screen transitions
• Input handling and pausing (not complete at the time of writing)
• Content loading and unloading

Notes on safe use of StorageDevice in XNA

I recently found a bug in StorageTasks.fs, an exception that I did not expect can be thrown in some situations. I am therefore going over the code and checking if I'm handling all exceptions correctly.

The bug I just fixed was an uncaught InvalidOperationException when attempting to open a StorageContainer after pulling the memory unit (which was picked as the storage device). The method that threw that exception was StorageDevice.EndOpenContainer. Sadly, the msdn documentation does not mention that.

I thought I would use EasyStorage as a reference. Unfortunately, even EasyStorage isn't 100% correct. For instance, see SaveDevice.OpenContainer: It does not catch InvalidOperationException. Although unlikely, this can happen if the user disconnects the storage device after BeginOpenContainer and before EndOpenContainer.

Anyway, I'll just go on and rely on testing to get it right... In the mean time, here are my findings regarding exceptions thrown by the StorageDevice API in XNA.

StorageDevice.BeginShowSelector

• GuideAlreadyVisibleException

StorageDevice.EndShowSelector

• None known

StorageDevice.BeginOpenContainer

• InvalidOperationException (from msdn)
• ArgumentNullException (from msdn)
• StorageDeviceNotConnectedException

StorageDevice.EndOpenContainer

• InvalidOperationException
• StorageDeviceNotConnectedException

StorageContainer.OpenFile

• StorageDeviceNotConnectedException
• FileNotFoundException

StorageContainer.CreateFile

• StorageDeviceNotConnectedException
• Possibly other exceptions, e.g. if no space is left on the device?

I have listed StorageDeviceNotConnectedException under most methods. Although it's never mentioned explicitly on msdn in the method documentation, it seems reasonable to expect it in any situation where the device might be accessed.

There are other failure scenarios associated to Stream and serialization which I'm not listing here. In particular, XML de-serialization of F# types (discriminated unions, records, tuples, lists...) will fail at run-time due to these types being immutable and lacking a default constructor.

Post-asteroid trauma

December and January have been hectic months for me.

December saw the publishing of a number of templates facilitating the use of F# with XNA:

• For the Xbox 360: library template, which has been downloaded 360 times. Does that mean there are 360 F# games coming to the Xbox? If you are working on a game, don't hesitate to let me know!
• For the Windows Phone 7 platform: library and full application  templates. Although I am not personally involved in WP7 development, I would love to hear about ongoing projects too!

I am not done yet with project templates. I am planning to make a full game template for the Xbox 360, and update all existing templates with the script for interactive development using fsi.

In January, the discovery of custom workflows and how they can dramatically increase the readability of one's code fascinated me. I think that's a feature that even functional programming sceptics should love.

My game, Asteroid Sharpshooter passed peer review and became available on the Xbox LIVE Marketplace. I doubt it will make me rich, but for a first try, it's not too bad.

What are February and the following months going to be like? First, I want to understand whether custom workflows are safe to use in the absence of tail call elimination. I'm still a bit confused... If they are safe, as I suspect they are, I will go ahead and make an XNA game starter kit, similar to the XNA Game State Management sample. It will include file IO and menus based on the Eventually workflow and cooperative multi-tasking.

At the end of February, Dream-Build-Play 2011 will open for registration, which will be the starting point for my next game.

I will also try to be more active in the XNA community and let people know C# is not the only language available in their toolbox. To this effect, I have registered on twitter. I'm deneuxj there. I'll be posting a few pre-paid codes for Asteroid Sharpshooter there, so stay tuned!

Is it possible to use F# to write and publish a game on the Xbox LIVE Marketplace?

Yes, it is.