Coding: Updating RendDx9 to use LAA D3DX9

[[R-DEV]Mats391]

BF2 has been compiled using the Large Address Aware (LAA) flag. That allows the process to use up to 4gb of memory. However RendDx9 does not use Direct3D in LAA mode, so using more than 2gb memory might cause crashes coming from Direct3D. This tutorial shows how to edit the shaders and Renddx9.dll in order to use LAA mode for Direct3D and use full 4gb of memory.

1. Shader preparation

To use the LAA flag we need to link RendDx9 to a newer d3dx9.dll. However those no longer support the shadermodel 1.3 and 1.4 that is used across the BF2 shaders. As such the first step is to prepare the shaders to use more modern shadermodel 2.0. For PR we went with shadermodel 2.0a. (Note: In theory SM3.0 could be possible, but it does not support enough input registers when used with DirectX 9).

1.1 Cleaning shaders

Before starting to convert shaders to the newer shadermodel, you should clean up a bit to cut down on the amount of code to adjust.

1. Remove all the unneeded effect files
2. Remove all the unreachable code inside of _FORCE_1_3_, _FORCE_1_4_ and #if 0 brachnes
3. Remove all the commented out shader code. Those are often things to reproduce bugs we have no clue about any more.

I also recommend to get rid of DICE weird datatypes (scalar, vector, matrix) and just replace them with the default ones (float, float3, float3x3 etc.). This will help a lot when asking for help or checking documentations.

1.2 Update existing HLSL shadermodel

Next you should change all existing HLSL shaders to use the newer shadermodel. This is mostly a find-replace job. Just as with the DICE datatype, I suggest to get rid of all the existing shadermodel definitions (e.g. PSMODEL or PS2_EXT) and instead use the default ps_2_0, ps_2_a, ps_2_b and vs_2_0, vs_2_a, vs_2_b instead. Again this makes things easier to read and share with people outside of BF2 world.
You also need to replace all the passes that use the name "point" to something like "point_". Point is a predefined control word in newer DirectX.

1.3 Updating inline ASM code

DICE used a lot of inline ASM code to reduce the amount of instructions in some shaders. This code is mostly written in shadermodel 1.3 or 1.4, so no longer supported for the newer DirectX. Updating this is the hardest part of the shader preparation. You have 2 options:

1. Update the ASM to your chosen shadermodel
2. Rewrite the ASM in HLSL and compile it with your chosen shadermodel

For PR we decided to rewrite these in HLSL. There is no straightforward way of doing it, you just have to read the documentation and do a bit of trail and error. Luckily for you, we already did this work for PR, so you can just have a look at what we did. (Note: You cannot simply use PR shaders_client.zip as we have a lot of PR specific parts in the shaders. You have to manually copy the translated HLSL that you are interested in. If you do use code of PR, please credit us as authors).

Helpful resources:

• https://docs.microsoft.com/en-us/win...-graphics-hlsl
• https://gamedev.stackexchange.com/

2. Updating RendDx9.dll

To use DirectX with LAA addresses, you need to create the effects with the D3DXFX_LARGEADDRESSAWARE flag. Here we edited the ShaderManager to add this flag in the constructor.
Since with LAA enabled you can no longer use names to address resources inside an effect and DICE uses this in some places. So these are the other things that need changing. Then there are some minor things that need fixing like taking screenshots in a different D3DFORMAT.
For PR we already did all this and you are free to use this as long as you credit us or in case it causes crashes, blame us

Explanation of the format:

[...] defines the file to edit
A: file offset of the bytes to change
O: Original bytes
N: New bytes
#: Comments

The code snippets in comments reference loaded addresses, not file offsets.

Spoiler for RendDx9.dll:

# Author: Mats
# This sets flags in shadermanager to include 0x200000 Large address aware

[RendDx9.dll]
A: 241B7
O: 38 5E 7C 74 07 83 8E 84 00 00 00 04
N: 81 8E 84 00 00 00 00 00 02 00 90 90

# RendDX9.dll+364C1 - 81 E2 00000200        - and edx,00020800
[RendDx9.dll]
A: 358C4
O: 08 00
N: 08 02

RendDX9.dll+3611F - 25 00000200           - and eax,00020800
[RendDx9.dll]
A: 35521
O: 08 00
N: 08 02

# Skipping DX9 calls with 0x0 as parameter handle. In non LAA this just return error, in LAA it crashes
# Replace at RendDX9.dll+1E715
# 85 C0                   - test eax, eax             ; Test for 0x0
# 74 37                   - je RendDX9.dll+1E77E      ; Skip DX call if 0x0
# 53                      - push ebx                  ; Push parameters
# 50                      - push eax
# EB 2F                   - jmp RendDX9.dll+1E74A     ; Jump Back to DX call
# 
# Replace at RendDX9.dll+1E748
# EB CB                   - jmp RendDX9.dll+1E715     ; Jump to Null-Check
# 
# Replace at RendDX9.dll+1E542
# 85 C0                   - test eax, eax             ; Test for 0x0
# 74 06                   - je RendDX9.dll+1E54C      ; Jump to pop
# 50                      - push eax                  ; Push parameter
# 8B 46 08                - mov eax, [esi+08]         ; Get last parameter
# EB B2                   - jmp RendDX9.dll+1E4FE     ; Jump Back to DX call
# 58                      - pop eax                   ; 2 parameter got pushed before
# 58                      - pop eax                   ; pop those into eax to discard
# EB B2                   - jmp RendDX9.dll+1E502
# 
# Replace at RendDX9.dll+1E4FA
# EB 46                   - jmp RendDX9.dll+1E542     ; Jump to Null-Check
# 90                      - NOP
# 90                      - NOP

[RendDx9.dll]
A: 1DB48
O: 53 50
N: EB CB

[RendDx9.dll]
A: 1DB13
O: CC CC CC CC CC CC CC CC
N: 85 C0 74 37 53 50 EB 2F

[RendDx9.dll]
A: 1D8FA
O: 50 8B 46 08
N: EB 46 90 90

[RendDx9.dll]
A: 1D942
O: CC CC CC CC CC CC CC CC CC CC CC CC CC CC
N: 85 C0 74 06 50 8B 46 08 EB B2 58 58 EB B2

# Passing handle instead of name into DX9 call
# Replace at RendDX9.dll+4C1D9
# E9 B6 E6 18 00          - jmp RendDX9.dll+1DA894    ; Jump to code cave to use correct parameters
#
# 52                      - push edx
# 8B 95 54 FF FF FF       - mov edx,[ebp-000000AC]    ; Get Handle instead of name
# 52                      - push edx
# E9 3D 19 E7 FF          - jmp RendDX9.dll+4C1DE     ; Back to DX call

[RendDx9.dll]
A: 4B5D9
O: 52 8B 55 F8 52
N: E9 B6 E6 18 00

[RendDx9.dll]
A: 1D9C94
O: 00 00 00 00 00 00 00 00 00 00 00 00 00
N: 52 8B 95 54 FF FF FF 52 E9 3D 19 E7 FF

# Changing screenshot D3DFORMAT from D3DFMT_A8R8G8B8 to D3DFMT_X8R8G8B8 as otherwise .png one has alpha in it
# This might reduce quality of screenshots
[RendDx9.dll]
A: 458B
O: 15
N: 16

After this just need to replace d3dx9_25.dll with d3dx9_43.dll (latest DirectX 9). BF2 also loads d3dx9_24.dll dynamically to fix some obscure bug. This should still get used. Using the d3dx9_43.dll here causes crashes and the d3dx9_24.dll does not seem to cause any further issues regarding high memory.


3. Updating TextureAtlasBuilder.dll and bf2.exe

These 2 are optional, but recommended. In TextureAtlasBuilder.dll you just need to change d3dx9_25.dll to d3dx9_43.dll. It wont change much, but reduces the amount of different d3dx9 getting loaded.
For bf2.exe I recommend to also apply this fix for loading minimap icons that have an address above the 2gb memory space.

Spoiler for bf2.exe:

# Author: Mats
# This fixes flags and other icons starting to become missing when high on memory

# Change
# From: prbf2_repo.exe+3746F2 - 0F8E 93010000         - jng prbf2_repo.exe+37488B
# To:   prbf2_repo.exe+3746F2 - 0F84 93010000         - je prbf2_repo.exe+37488B
[bf2.exe]
A: 373AF3
O: 8E
N: 84

4. Issues to be aware off

These changes are all part of current PR and thus have been tested by quite some people on different system. As such for the most part these can be considered stable. However one big issue are players using graphics mods like Reshade or overlays like Overwolf. These might no longer work with the updated shaders/d3dx9 and can cause crashes.