Are you trying to create it in private or shared? Either way just click on the folder you want the new folder to be in, ie “Private Forms” or “Shared Forms” or a subfolder of those, and the button should become clickable. If you have one of your forms highlighted/selected the new folder button will be disabled. You have to select an existing folder to create a new folder.
If you are trying to create a new folder in the library, that has to be done in the System Admin console in the Documents section.