TL;DR Plugin developers should use if(is_admin()) and get_current_screen() to determine the page and load these scripts and files conditionally. And the non-plugin developers, we can dequeue the scripts and css files easily. In short: Be an awesome, caring and courtesy developer, please.
Speed up a website by removing unused CSS from stylesheets. Use Chrome’s DevTools Coverage tool as a guide for what to remove.