Only checking B + B_T <= 0 allows for F_T to be calculated as a negative value when T_T is only partially in the penalty zone. I can force the F_T calculation to only consider the part of the transaction in the penalty zone, though I haven't decided on the most elegant way to do this. An example fix might be:
# Fee Calculations
T_T = T_R # Use reference tx size
B_T = T_T / M_N # Increase from adding additional transaction to block
F_T = R_Base * (2 * B * B_T + B_T**2) # Additional fee required to overcome the increase in penalty, F_T = P_T
if B + B_T <= 0:
F_T = 0 # B + B_T > 0 for calculated F_T to be valid
if M_B < M_N and M_B + T_T > M_N: # If T_T jumps M_B into penalty zone
T_T = T_T - (M_N - M_B) # consider only portion of tx in penalty zone
B_T = T_T / M_N # Increase from adding additional transaction to block
F_T = R_Base * B_T**2 # take B = 0
f_I = 0.95 * R_Base * T_R / (M_L**2) # Minimum fee per byte, M_F = M_L
I'm still thinking about it.