If I login to my Teleworker MBG and configure my set to register with the SRC I have no problems with one way audio.
If I point my set to the PBX audio is not getting out to the teleworker set depending on the type of call.
To clarify "point my set to the PBX", do you mean from the configuration of the handset itself, or by choosing the "Configured ICP" for the extension in MBG?
If I call an extension on the PBX I have two way audio, If I call the embedded mail or call out on the SIP Trunk I have one way audio.
Are you using MBG SIP proxy for the SIP trunk? What type of MiVB is it? Bear in mind with vMiVB, the embedded voicemail media stream won't be coming from the IP where the ESM web interface lives, it will be coming from the MSL instance's IP. If you're using NuPoint then that's probably [don't know, never used it] yet another IP that will need to be allowed to talk to the MBG.
If everything works when configuring handsets to use the SRC as their "ICP" then I would be tempted to just leave it at that. TAP licenses are only used when calls are recorded.