This is a tricky question that comes up quite often. My general rule is to determine if the new function would be re-usable on other WordPress themes or a future revision to my site’s theme.
Does your site’s content require your custom functionality to always be there? An example where this would be the case is the use of custom shortcodes. A shortcode gets embedded in post content to add special features, such as embedding a photo gallery, video player, maps, and custom forms.
Registering a shortcode function in your theme’s functions.php has the potential to create future problems as upgrading or changing themes would wipe out these functions making all of your custom shortcodes in the post content not work. Using a plugin here would preserve your shortcodes no matter which theme you may be using. A plugin is also great if you manage multiple sites and want to use the same set of shortcodes everywhere for consistency.
If you’re creating some basic functionality that probably wouldn’t be carried over to another theme such as a simple function or hook it makes sense to embed that in functions.php. You can see plenty examples of this in WordPress’s default themes. Here the developers do things like add special classes to the body tag, change thumbnail sizes and register widgets. These types of functions probably wouldn’t be applicable to a future theme as it would need its own set of custom declarations.
You’ve probably read some articles which point to plugins causing performance issues with WordPress sites. Because of this we generally advise our clients to avoid installing third-party plugins other than a small subset of plugins we’ve properly vetted. Creating and using your own plugins however does not create a performance problem if it has been properly coded.