Tried compatibility setting but to no effect. I've been wrestling with 3ds Max 2015 for a good part of the day. It's familiar enough that it shouldn't pose any trouble except (blushing here) I DLed NWMax version 0.8c00 here, unzipped it and dropped it in 3ds folder as per the instructions. When I ran 3ds Max I can't seem to find how to make the NWMax tool bar to display. NWNmax did add other functionality but the tool bar has remained petulantly illusive. Any help?
To use NWMax with Gmax, you MUST start Gmax using the new icon called NWMax, it will provide a green icon in your active programs list on the right hand section of your task bar. By this I mean, GMax must NOT be running when you start NWMax, it will automatically call up and run Gmax when you start NWMax, setting up some settings and an additional program called a 'listener' which will be that extra green icon. That util, reads the text files of gmax when you go to export, and properly exports the data as needed to be used by the game.
Anyway, assuming you have the above working correctly, the issues with video memory you are having is due to the fact that gmax was written to work with ANCIENT graphics drivers, as such, not much memory was available, and it simply will not recognize or use most of the ram of a newer video card or even the ram used by a built in Intel graphics chip. In other words, save often. Exit and re-load nwmax/gmax every hour or so to help alleviate that particular problem... otherwise, you just have to deal with it. Note that the problem persisted in 3dsmax up until version 9 or so, but by then, OTHER problems were showing up etc... basically trading one set of problems for another set with each upgrade. Version 9 was about the best 3ds version for NWN usage, but you can successfully get versions up to version 12 at least , working... after that, I have no idea how well any of the later versions of 3ds work with nwmax, etc...
Good luck.