VirtualDub Game Recording Guide

Windowed Mode List

VirtualDub Game Recording Guide

Benefits / Drawbacks
Windowed Mode
Notes / Troubleshooting
Sample Footage
Discussion thread @

2011-10-26  Links to sample footage have been fixed. Also, a few details in the guide have become outdated or inaccurate since its initial posting.

This guide is the result of my search for an alternative to game recorders such as Fraps, Xfire, or Gamecam. While those applications work well, I wanted something that could offer greater flexibility and customization for my game recording needs. This method, based on VirtualDub and other freely available software, provides almost total control 
over capture settings and system performance.

Read on for a summary analysis, then continue with the detailed guide to try this for yourself. I welcome opinions, tips, and feedback, so please feel free to contribute by visiting the discussion thread.

  • Capture games at virtually any resolution with realtime video resizing (during capture) to any resolution.
  • Capture at any framerate, without restricting the in-game framerate to the capture setting. If your rig can maintain 80+fps normally, it can probably sustain 50-60fps in-game while capturing 30fps video. If you capture to a Youtube-friendly 15fps, then in-game fps will increase because recording demands decrease. With Fraps, you would have to play the game at 15fps.
  • Tweaking game settings can directly improve capture performance. Often just turning down/off things like HDR, bloom, or post-effects can be enough to achieve a smooth playing/recording experience.
  • Capture using any compatible codec on your system. MJPEG is highly recommended, but you can compress directly to Xvid, Divx, Huffyuv, or any other codec you wish. Using the MJPEG settings described in the guide will yield near-lossless quality video files that are 4-5 times smaller than Fraps or Xfire videos. Of course, you can customize the compression level to your preferred quality/filesize ratio.
  • Capture audio with video, with optional realtime audio compression to any ACM codec. You can capture directly to Xvid+MP3 if you want.
  • Configurable stop conditions based on duration, file size, or free disk space.
  • Optional realtime video filtering/processing.
  • Near-universal game compatibility, independent of DirectX or OpenGL versions. If your computer can run the game, then you can probably record it. I have successfully recorded games as old as Planescape: Torment and Deus Ex.
  • Because this is based on screenrecording rather than DirectX hooking, these tools can also be used to record any other on-screen activity: create video tutorials, record live broadcasts, capture short clips from HD videos, etc.
  • Much more complicated than Fraps, Xfire, etc. Though most of the work is in the first-time setup and testing phases, you will need to select or change a few capture settings on a per-game basis for each recording session.
  • Depending on system configuration, games might have to be played in windowed mode to avoid video flickering caused by hardware overlays, though there are a couple of tweaks that might circumvent the issue. Almost all games support a windowed mode, and most run without problem. (The peformance data in the 'Benefits' section above is based upon recordings done using windowed mode. See the Sample Footage section for videos.)
  • No on-screen recording indicator.
  • No capture profiles or special video management features.
Be aware that this guide is provided as-is and you are responsibile for any changes you make to your computer. With that said, it's unlikely that the information here would cause damage or problems to your system.

Essential programs:
VirtualDub - versions 1.7.8-1.8.5 have been tested. I recommend the latest stable version.
VH Screen Capture Driver - to download click HERE.
HotkeyP - also download this configuration file and unzip it in the HotkeyP program directory.
PICVideo MJPEG Video Codec -
the current version of this codec is shareware; however, the old freeware version (used for this guide) can still be obtained via the Showshifter Demo. Simply install the demo, reboot, then uninstall it. The codec should still be on your system after uninstalling. If the uninstaller asks if you wish to keep the codec installed, choose 'Yes', of course.

Additional useful software:
LAME ACM MP3 codec - for realtime compression to MP3 audio within an .avi file.
ShellEnhancer - utility that allows for X-style window moving and resizing (Alt+Lclick anywhere on a window to move it), window position locking, and much more. Very useful for running games in windowed mode.
Huffyuv lossless video codec - scroll down to the 'Codecs' section to find it.
RivaTuner - enables anti-aliasing for games that don't feature the setting in-game.
Muvixer - application to preseve soundcard recording settings for games that force particular input (Battlefield 2, Team Fortress 2, etc.). Simple and very useful.


Open VirtualDub and enter capture mode by selecting File->Capture AVI. You should see this window (perhaps showing a preview):


Select Device->VHScrCap (DirectShow) to activate the VH Screen Capture Driver. The VirtualDub window should show a preview image of your desktop. If it does not, verify that you have either Overlay or Preview selected in the Video menu. (If neither option works, you might need to reinstall the VH
Screen Capture Driver). Later on, when you are ready to being capturing gameplay, be sure to turn off the preview by selecting Video->No Display. There's no need to waste CPU cycles on a preview that will not be visible.

To configure the VH
Screen Capture Driver, select Video->Capture filter. You should now see this dialog:


For fullscreen game recording, select the 'Track screen' option as shown. The Width and Height fields should auto-fill to your current resolution. With this setting the driver will automatically adjust its capture area to your current resolution, so if you play some fullscreen games at a lower or higher resolution you won't need to make any changes to this setting beforehand. Once finished, click the Settings button.

                                 tip: hover your mouse over dialog elements to view pop-up descriptions.

Here most of the options are self-explanatory. For now, start with a Frame rate of 30 and leave everything else unchecked. If you want to try using the resize options later on, then check the boxes for
Resize to output sizeUse linear resize in 32bit mode (better quality, especially for text), and Specify output size. Align video is a setting that will force any resolution you choose to auto-fill to the nearest power-of-4 dimensions (some video codecs are picky about this). I generally leave this checked because it does nothing unless I mess up my resolution settings with odd numbers or ratios--which has never happened (yet). Border color is the color used by 'Align video' to fill the video frame. Leave it black, unless you just love puce. Show Mouse does what it says. If you want to see the cursor, click the box. I have yet to use it. RTS players might want to turn this on, though I got a kick out of watching my Zero Hour units 'magically' perform my commands. Leave the Optimize settings off--they actually reduced performance in my tests.

Here is a list of some common resolutions you can try when using the Resizing options:

16:9 Aspect Ratio 16:10 Aspect Ratio 4:3 Aspect Ratio

When you have finished configuring the capture driver, click the X in the upper-right corner to return to VirtualDub. Go to Video->Set custom format [SHIFT-F] to open this dialog:


You need to make sure that the 'Use custom size' fields have the same resolution values you selected in the screen capture driver, and always double-check this setting after configuring the driver (conflicting settings can cause VirtualDub to error/crash). The 'Data format' field will automatically select 32-bit ARGB, since this is what the driver uses. I don't believe this can be changed--when I tried selecting a different format VirtualDub would always switch it back to 32-bit. Once your resolution values are correct, close the dialog.

To set the capture framerate in VirtualDub you can use the 'fps' button on the status bar:


Set the framerate to the same value you chose in the capture driver. If your framerate doesn't appear in the list--like 24fps--you can set it via the Capture->Settings menu option:


Enter your framerate here, but make sure the Wait for OK to capture option is not selected before you close the box. The framerate should now display correctly in VirtualDub's status bar.

Now go to Video->Compression [C]:


Select the PICVideo MJPEG Codec (as shown) and click the 'Configure' button. The codec settings dialog should look like one of these two screens:

picvideo01.gif          picvideo02.gif

While you can make quick adjustments using the slider, I recommend clicking the 'Advanced' button to expose all of the encoder's settings. The picture on the right shows the settings I use most often. I set the slider at 17 and then bump the Chrominance up to 15 just so it matches the Luminance. The human eye is more sensitive to Luminance, so you can get away with lower values for
Chrominance if you wish. I have found that the PICVideo codec offers great picture quality at very reasonable filesizes. Try different compression levels and see what you prefer.
Subsampling - I would leave at 4/2/2.
If you want to tweak this setting, this article provides good reference information.
2 Fields If More Than - leave this off.
Computer games generate progressive images--no need to deal with fields.
Swap Decompress Fields - off, same as above.
Force YUY2 Output - didn't notice any change or benefit when I tried it, so I recommend off. Experiment if you wish.
Codec Enabled - self explanatory. Make sure this is on.
Once you've chosen your settings, close the dialogs to get back to VirtualDub's main window.

If you want to capture audio in your recordings, select the Audio->Enable audio capture menu option. Next select the audio input from the menu shown here:

On my system the "What U Hear" input option works best. Other audio cards might have this labeled as "Stereo Mix". If you can't find an input like these, try the 'Wave' setting instead. After selecting the input, choose the audio format with the "Raw capture format" option or via the audio button on the main window's status bar:

vdubrawaudioformat.gif                       vdubaudiostatusbutton.gif

I generally use 48KHz, 16-bit, stereo audio since it provides great quality and is DVD-compliant. Note that some audio format options may only be selectable via the 'Raw capture format' menu option, as the screenshots above show for 96KHz.

You can choose various audio compressors with the Audio->Compression option:


I recommend the '<No compression (PCM)>' option for a few reasons. First, the disk space needed for audio is negligible compared to what you need for the video, so you won't get any significant space savings by compressing the audio in realtime. Second, compression will add some processing overhead on your CPU that could hinder game performance. Third, capturing uncompressed audio gives you a 'master' source that you can then compress to any other format without the quality reduction incurred by transcoding from one lossy format to another. If you do wish to compress the audio in realtime, I recommend using the 
LAME ACM MP3 codec--just make sure you match the MP3 encoder settings to the same frequency and channels you selected in VirtualDub's audio format menu. I have successfully tested capturing to MJPEG+MP3 as well as Xvid+MP3 .avi files, so any number of compressor combinations could be possible.

Once you have finished configuring your audio and video capture options, got to File->Set capture file [F2] to choose a directory and filename for your game recordings. You must choose a capture filename every time you open VirtualDub's capture window, even if you do not change any video or audio capture settings. If you have multiple hard drives I recommend capturing to a drive other than the one with your game installations and/or Windows OS. This will guarantee the best write performance possible and will prevent stressing a single drive with game loading, video recording, and managing swap space simultaneously. Also be sure to enable the Capture->Autoincrement filename after capture setting so that you can perform multiple captures during gameplay.

VirtualDub has customizable hotkeys available in the Capture->Preferences menu, but they do not function reliably when running a game. In order to use VirtualDub's capturing abilities without exiting your game, you need the global hotkey functionality of a program like HotkeyP.

Run HotkeyP and click the system tray icon
hotkeypicon.gif to open the main window:


HotkeyP is an advanced program. While it is logically designed and has an informative helpfile, it can be confusing to learn. To simplify this guide I have included a copy of the hotkey 
configuration file that I use for game recording, which you should already have unzipped in your HotkeyP program folder. To load the file, select the File->Open hotkeys menu option, navigate to the HotkeyP program folder and select vdub.htk. The main window should now show the following hotkeys:


VirtualDub requires 1 key to start recording and 1 key to stop (defaults are F5 and Esc, respectively). I have found the Numpad+ and Numpad- keys to work well in HotkeyP since I can hit them quickly with my thumb while gaming, but you can use any keys or combinations you want. Highlight one of the hotkey entries and click the Edit button to open this dialog:


Be sure both hotkeys have the correct path entered for VirtualDub. As you can see, my Program Files are on Drive F. Yours are probably different. To customize different hotkeys for recording, change the entry at the very top of this dialog window. I prefer the simplicity of a single keystroke, but you can configure just about any combination of keys using Ctrl, Alt, Shift, or even the Windows key.

Once HotkeyP is configured, you are ready to start recording. The recording process is actually rather simple: HotkeyP monitors your keyboard while you play a game, then sends VirtualDub 'start' and 'stop' recording commands when you press the associated keys, and VirtualDub records your videos to the folder you selected. Obviously there is no on-screen recording indicator, so you will need to mentally keep track of when you are recording (sometimes I randomly hit the 'stop' key, just in case I've forgotten that I started a recording earlier). When you are finished playing you'll have a collection of .avi files ready for playback or editing.

I recommend testing by recording your desktop first, then try some games. For HotkeyP to successfully send keystrokes to VirtualDub, the VirtualDub window must be open on your desktop and not minimized to the taskbar. Also remember to turn off VirtualDub's preview to save system resources. HotkeyP can be minimized to the system tray.

Windowed mode recording

Nearly all games render using hardware overlays/Direct3D surfaces. Because of this, general-purpose screen capturing applications like VirtualDub cannot successfully capture game images, but instead end up with a solid color image like black or purple (the color key used by the overlay). When I capture games in fullscreen on my system, the videos constantly flicker because the capture driver sometimes captures the overlay color instead of the game image, resulting in black frames peppered throughout the video. The first thing to do is try recording some of your games in fullscreen. If it works, awesome! Please post infomation about your results and system specs in the discussion thread. If your videos have flickering, read on.

There are a few ways to deal with this problem. With my old graphics card, I could successfully record most games in fullscreen just by enabling anti-aliasing. With other games, if I opened Media Player, played a video, then paused the video but left Media Player open, it would prevent the game from using the hardware overlay and I could thus record the game in fullscreen without issue. Games can obviously run without this feature--since it is not used in windowed mode--but I have not yet discovered how to universally disable hardware overlays so that games won't use them when running fullscreen. If anyone reading knows how to do this, please share the information!

If neither of the above tips works for you, then running your games in windowed mode will allow you to record game videos successfully. Almost every game can run in windowed mode by one means or another. Here is a list I have compiled of many popular games along with windowed mode instructions.

Scenario 1 - Games featuring a windowed mode that renders at fullscreen resolution with no window border. (best case)
Example games: Valve's Orange Box engine games (Team Fortress 2, Half-Life 2: Episode Two, Portal)

Games in this category essentially look and function like they are running fullscreen, even when windowed. For example, if you launch Team Fortress 2 with the command line parameters -windowed and -noborder, the game will run at maximum resolution, without a window border, perfectly aligned to the edges of your monitor. You can leave the capture driver set to 'Track screen' to record these games.

Scenario 2 -
Games featuring a windowed mode that renders at fullscreen resolution with a window border.
Example games: Lord of the Rings: The Battle for Middle-Earth series, some games when run with D3DWindower

For games like this, I use
ShellEnhancer's Alt+Lclick window dragging feature to center the game window so the window border is off-screen, allowing the game to fill 100% of the monitor as in Scenario 1. The 'Track screen' setting can be used here as well.

The only other option for these games is to run them windowed at less than your maximum screen resolution. In this case, refer to the instructions in Scenario 3 below.

**Note that running an RTS in windowed mode will likely disable the ability to pan the game camera by moving the mouse to a screen edge. Most RTS games also allow panning by arrow keys and/or click+drag, so the loss is minor.
**To disable the Taskbar's 'Always On Top' setting, right-click on it, choose 'Properties,' then un-check 'Keep the taskbar on top of other windows.'

Scenario 3 - Games
featuring a windowed mode that includes an always-visible window border on screen.
Example games: most games fall in this category

There are 2 methods available, depending on your workflow preference:

1. Capture fullscreen, crop in post.
  • No need to adjust the capture region each time you play, just leave the capture driver set to 'Track screen.'
  • Window border and excess desktop space is also captured and will require editing to remove (unless you want your desktop/other windows visible in the final video).
  • Slightly reduced capture performance since unnecessary screen space is recorded. Probably not significant unless you are running a game in a tiny window, in which case I suggest the next method.
  • Note: VirtualDub has a very effective cropping tool and is the best tool for basic cutting/encoding of .avi files.
2. Customize capture region to game window, no cropping in post (my preferred method).
  • Capture region must be set each time you play a new game or if you change the position of the game window.
  • No excess desktop or window borders are captured, so videos come out in a 'final format'--ready for posting or conversion to a different video format.
  • Reduced capture area means fewer resources needed for the recording process. This can improve performance, but not significantly unless you are capturing substantially less screen space than your full resolution.
  • Note: ShellEnhancer has 'Lock window position' and window auto-management features which you can use to retain custom window positioning on a per-game basis. The program works with many games without issue, but it will cause pre-Orange Box Source engine games to crash on launch (Half-Life 2, HL2: Episode One, CS: Source, etc.) Try terminating ShellEnhancer if you encounter problems with other games.
Example - Star Wars Battlefront 2

The game looks like this in windowed mode (set to my maximum resolution, 1440x900, in the game options menu):

Note: I use ShellEnhancer's auto-manage feature to lock the window position to X:0 Y:0 coordinates.

Notice the slightly visible window border at the top and bottom of the screen. Leaving the game running or paused, Alt+Tab to the VirtualDub window left open on the desktop, then access the capture driver via
Video->Capture filter:


Select the Region Tool as shown above. A crosshair cursor will appear as well as a dialog box showing realtime pixel coordinates feedback. Selecting your capture region is a 2-click process: select one corner, then the corner diagonally opposite:


The capture driver dialog window will auto-fill the values you selected:


The Width and Height values are most important. Make sure these are even numbers, preferably powers-of-4. You can manually correct them in this dialog if needed. If you want to use the realtime resizing features of the driver, click the Settings buttons and enter your chosen values.

Now you can close the driver dialog, but before you go back to the game and begin recording you should always Alt+Tab to VirtualDub and hit Shift+F to verify the resolution values in VirtualDub match exactly what you chose in the capture driver. VirtualDub normally auto-detects the driver's settings, but a mismatch will likely cause a crash that you won't notice until you have exited your game (and thus you will have no recordings).

Some games behave differently than others in windowed mode, but you can extrapolate the steps here to accomodate most any recording situation. I hope this guide proves useful, and don't forget to share your experiences, questions, or tips.

Notes / Troubleshooting

For reference, this guide was created and tested primarily on a machine with the following specs:
  • AMD Athlon64 X2 5600+ Dual Core CPU
  • EVGA Geforce 8800GTS 640MB SSC Edition
  • 2GB Corsair DDR2 800 RAM
  • Soundblaster Audigy2 ZS Platinum
  • Acer AL1916Wasd 19" Widescreen LCD, 1440x900 Native Resolution
  • Seagate 320GB SATA HD dedicated for capturing
  • Windows XP Pro SP2
Notes & Tips:
  • Be sure to close all other applications and terminate any unnecessary background processes, especially those that can directly conflict with the recording process: backup programs, desktop enhancers, anything that employs keyboard hooking, etc.)
  • No testing has been done with Windows Vista, so I don't know how well this guide will work on that OS. If you try this guide on Vista, please post feedback in the discussion thread. (**A post from VirtualDub's developer in this thread doesn't sound promising in regards to Vista compatibility.)
  • I have not tested any AMD/ATI cards, but I suspect they would work just as well as NVIDIA cards for game recording. Again, please provide feedback if you use one of these cards.
  • The ACE Mega CodecS Pack also has the PICVideo codec. I generally avoid codec packs like the plague, since they can easily wreak havoc on your system, so proceed with caution. This pack will supposedly allow selective installations. If so, I recommend installing only the PICVideo MJPEG codec. You might want to keep your favorite uninstall utility handy, just in case.
Codec comparison:
Encoding application- VirtualDub
Source video- 1280x800, 30fps, 0:30 game recording
VirtualDub controls processing thread priority. All encodes were done at the default 'Normal' setting, resulting in 65-70% CPU usage across the board.
19fps- Lagarith lossless
23fps- Xvid (realtime capture profile)
28fps- MS MPEG4 V2 (used by GameCam)
32fps- HUFFYUV lossless
37fps- Xfire lossless
54fps- PicVideo MJPEG

Sample Footage

Crysis - 1440x884, MJPEG, 0:14, 62MB
Call of Duty 4 - 1440x884, MJPEG, 0:43, 148MB
Battlefield 2 - 1280x800, MJPEG, 0:14, 47MB
Battlefield 2 - 640x400, XviD+MP3, 0:18, 2.5MB
Half-Life 2 - 1440x900, MJPEG, 0:12, 54MB
Team Fortress 2 - 1440x900, MJPEG, 0:05, 20MB
Star Wars Battlefront 2 - 800x500, MJPEG, 0:20, 50MB


Copyright 2008-2011