Quest Techie: Fixing NVENC Issues Causing Link To Fail

You’re were all excited about Oculus Link — but when it released to beta in November 2019, it wouldn’t work for you. Or maybe you’re trying to use OBS to stream some Link action to your YouTube or Twitch channel — and get a cryptic error message. NVIDIA’s NVENC limitations may be the issue and I’ll help you fix them.

Image for post
Your card may be nerfed and you don’t even know it …

Nvidia NVENC (NVIDIA Encoder) is a feature in its graphics cards that performs video encoding, offloading this compute-intensive task from the CPU.

Welcome To The World Of Compression

Video compression is a very resource-hungry process and is used for a variety of use cases. Maybe you’re ripping a Blu ray movie to MP4 format. Maybe you’re live streaming gameplay from your PC to Twitch or YouTube.

Or maybe you’re trying to use Oculus Link with an NVIDIA graphics card. Maybe you wondered why not every card can support Link?

Thanks to the advanced GPUs we have today, we can hand off “mundane” tasks like video encoding to dedicated hardware on our video cards (Intel and AMD have similar “hardware encoders” built into some of their cards too) — leaving the rest of the GPU and CPU to run our favorite games or do other work.

Before we had this sort of hardware, any sort of video compression bogged the computer down to a stand-still.

The Oculus Link is no different. After all, you still need lots of CPU and GPU to play Asgard’s Wrath while it is being streamed to your Quest, right?

Understanding the Problem

Depending on what video card you have, there is a limit as to how many concurrent NVENC “video encoding” sessions you can have running at one time.

Effectively, most consumer grade video cards only allow one or two encoder sessions to be active at one time. Some older cards don’t have this feature at all which is why they are not compatible with Link.

This varies from card to card — but two appears to be the limit on video cards that are affordable by end users. NVIDIA sells “business class” GPUs that support more but they are a lot more expensive.

If you have only have one session available and something else is using it — Oculus Link won’t work.

Maybe you got Oculus Link to work once and now it won’t work? You might be using that single session somewhere else (like OBS or even NVIDIA’s own “overlay” will use a session). Using a ripping tool like DVDFab or Handbrake could be sucking down the NVENC session.

Once you know there is some sort of limit here, it suddenly makes sense why you may be having problems — intermittently or all the time.

So is there nothing that can be done? I didn’t say that …

Immediate Gratification — Disable the NVIDIA Overlay

If Link doesn’t work at all, you may go into NVIDIA GeForce Experience and shut off the In-Game Overlay feature.

  • Open up Geforce Experience
  • Click on the Cogwheel Icon in the Top Right
  • Go into the General Tab
  • Disable Ingame Overlay in the middle column
  • Reboot Windows (just to be sure)
Image for post
Disable the In-Game Overlay to free up a session …

Now, start up Oculus and plug your Quest back into your PC with the approved Link cable.

Does it work? AMAZING!

If not, read on. We’ll try to remove the limitations of NVENC sessions your GPU will perform and that will probably solve the problem.

The OBS Solution — Or Increasing NVENC Sessions

I ran into this issue while trying to live stream some Oculus Link game play.

Image for post
Ugh. What a USELESS error message.

After far too much troubleshooting, I figured out that as long as Oculus Link was running? I could NOT live stream (or record) in OBS.

Once I quit the Oculus app and tried to record something else, all worked as it always has.

I turned off my In-Game Overlay and rebooted, but still I had no love. OBS still wouldn’t record or stream while Link was running.

Note: If you have CPU power to spare you can always change your encoder from NVENC to X264 and you can then use OBS while you’re playing Link.

I posted this information on r/OculusQuest and got some information regarding this NVENC situation. Some good reading there.

Turns out that there is a conspiracy theory regarding NVIDIA and the number of concurrent NVENC sessions they allow — which has nothing to do with the power of the graphics card but rather an artificial limitation on the number of sessions (presumably in pursuit of making people buy more expensive cards).

Image for post
Someone tweeted it … it HAS to be true, right?

I did more searching … found this on StackOverflow … and THIS from SuperUser.com.

By the way, you can thank our friends in the Plex Community for helping uncover this rather unnatural limitation.

Let’s Do It — Increase My NVENC Sessions

As per usual, I’m not responsible if you do anything bad to your computer, etc. That being said, none of this is permanent and can easily be reverted. Follow the instructions carefully.

Note: This assumes Windows 10 and an NVIDIA graphics card that is known to have NVENC capabilities.

Right, now that we’re done with the disclaimer, here we go. By the way, this looks a lot more involved and scary than it actually is.

  1. Determine your operating system (32-bit or 64-bit).
  2. Determine your current NVIDIA driver version.
  3. Backup these file(s). Put them somewhere safe.
    For 64-Bit Windows:
    C:\WINDOWS\SysWOW64\nvencodeapi.dll
    C:\WINDOWS\system32\nvencodeapi64.dll
    For 32-Bit Windows
    C:\WINDOWS\system32\nvencodeapi.dll
  4. Download this patching tool and extract it somewhere handy.
    Note: You’ll need a RAR compatible compression tool to extract this. Try 7Zip if you don’t already have a tool.
  5. Go here to figure out what library patch you need.
  6. Find your card type (GeForce), your driver version, and x64 (64bit) or x86 (32bit). RIGHT CLICK (see how I put that in caps?) the library patch file and do a SAVE AS … Download it where you extracted the tool from step 4. See a sample of the table here.
    Note: It will have a weird extension (1337) — that’s ok.
  7. In the top box, choose the .1337 file you downloaded in step 6.
  8. You will be asked to find the file it is looking to patch. Navigate to C:\Windows\System32 and scroll down. Select the file.
  9. Now hit PATCH.

That’s it. You’re done.

You may want to reboot your PC for good measure — just to keep things clean.

If something goes wrong — you have the backup files you made. Replace the one you patched, reboot.

Test your scenario. If Link didn’t work before, does it work now? If you were unable to play Link and use OBS at the same time … can you now?

Did this work for you? Share your experiences below with my readers.

Written by

I write, blog, record and review anything that interests me — including humanity, parenting, gizmos & gadgets, video games and media.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store