If you have the form instance stored on a field in the form, then yes you can use that in a ‘connect’ or ‘connect and search’ function. Using the ‘connect and search’ can then present details from the instance into the pop up window but not directly to show the actual instance in a new browser window. but if you perhaps write back the instance ID you could then use that in a ShowURL script where you can compose the link to that form instance and then open that into another browser window.
another option though is to look at the DataList object in a report, where using the ‘Meta-Link’ you can present links to the form in a data list column. Using a report may also be worth considering as you can utilise ‘filters’ on the report to enable a range of different selections to help you find the required instance