I'll try to supplement the answer some more.
Yes, you may end up harming performance, but it's worth considering the type of page / app you're creating.
If it's a simple page that can be used on mobile devices (3G or 4G) the weight is extremely important, since the app may be running on a slow-latency, high-latency network and possibly even on a device with very little ram resources, etc.
So leaving your heavy page will certainly spoil the experience in these cases, but if it is a page to be used primarily (or exclusively) on desktops (eg a web system) where people tend to spend a lot of time online in the tool , it's worth taking a little longer to load into the aperture and then have a smooth experience during use.
Things to take into consideration:
1) The more separate files, the greater the chance of being triggered multiple requests to your server which, with high latency, will cause slow loading. With http / 2 this can become irrelevant.
2) The larger your files, the longer it will take to load them, but if you set the headers correctly to cache, you will have users downloading the file only once and then receiving the 304 as a (not modified). This will greatly help the next time the user enters the page
3) Even if you choose to leave everything in a single file, take a look at the "weight" of your site (in kb itself) and try to take into account the profile of people who will use it. If they are mostly fixed broadband, a 1mb site is not so bad, but if it's a cell phone, it's pretty complicated.
I hope I have helped =)
