My opinion is the reason research took a drop because of the great society and the Vietnam war. Before that time, nearly every company had a Research and Development lab to produce better products. I feel that because in increased taxation, R&D departments were shut down and the research that took place was mostly at the government or school level. It couldn't match what was taking place at the private level so we stopped advancing at the previous rate.
As someone who has spent their whole career in R&D and is now close to retirement, from the inside it looked like it had more to do with R&D getting more expensive as much of the low hanging fruit had been picked, meaning lots of project failures and/or long development times. That combined with the emphasis on short term profitability si what happened IMO.
In terms of the lack of R&D jobs, more recently that was due to outsourcing much of what R&D the is both on shore and offshore to CROs (Contract Research Organizations)
And yes industry "collaborating" with academia, that has dirt cheap highly educated, though poorly paid labor in graduate students and post-docs, as well as the government supported basic research.
Companies take every opportunity to be as profitable as possible and the bean counters recognized these things were a LOT cheaper than in-house R&D, regardless of taxes, and thus give (at least in the short term) the biggest ROI.
-Karen
