@danielroe thank you very much for your response. Your solution works! I added a new server middleware as suggested by you. When I do npm run dev and cURL it, I can confirm the header is now set correctly.
However, at first, I ran into the issue again, so I checked the code you linked. It seems for error pages (such as a simple 404), the header is enforced to ensure a Nuxt error page is not shown in an iframe! Just wanted to note this here for future refrence in case someone else has the same issue.