I must say this game has some of the worst jagged edges I've ever seen. There are no options for application applied Anti-Aliasing or Anisotropic Filtering. Normally this wouldn't be a problem as you could just force them through a profile in the Nvidia CP. However this game has a bug where if you do that, your HUD will disappear. There is an option to get it back in the in-game settings menu, however it is also bugged and will be grayed out.
So in order to get AA and AF to work with AA3, you need a third party program called nHancer.
http://www.nhancer.com/You'll need to create a profile for AA3. Do this with the Create new profile icon. Name it what you want. Select it in the profile list then below in the Executables section, click the icon to the right of the box and direct it to your AA3Game.exe. Next, click the Compatability tab and check the Anti-Aliasing compatability box. From the drop down menu choose 00000041: Unreal Engine 3. Next, open the Enhancements tab and check the Anti-Aliasing and Anisotropic Filtering boxes. Then choose the levels you want. I recommend Multisampling AA at whatever level you want. Obviously higher levels equate to less performance but better graphics. I also recommend 16x AF. Most modern graphics cards can handle 16x AF without much of a performance hit. When you're done, just close the program.
Until the game is patched, this is the only way of enabling AA and AF that I know of. Once they fix the no HUD bug and grayed out menu option we won't need this.
Expand the images below for comparison. All settings high in both, 0 AA and 0 AF in the first, 4x MSAA and 16x AF in the second.