I recently asked a question about testing in game development – this is how I knew about it, by the way. The answers there pointed to some odd, specific drawbacks:
It's expensive if your code should be heavily coupled .
It is difficult when you need to be aware of different hardware platforms when you should be analyzing the output to the user and the code result only makes sense in a broader context .
UI and UX testing is very difficult . And in particular, automated testing can be more expensive. Be less effective than a set of very low cost (or free) beta testers .
The fourth point reminds me of my experience. I worked in a very lean, XP-oriented, Scrum-managed company where unit testing was highly recommended. However, in moving toward a leaner, less bureaucratic style, the company has neglected to build a QA team – we had no testers. So often customers found minor bugs in some systems, even with test coverage of> 95%. So I would add one more point:
– Automated testing can make you feel like quality assurance and testing are not important.
Also, I was thinking about documentation at the time and considered a hypothesis that might be valid (to a lesser extent) for test two. I just felt that code evolves so fast that it's quite difficult to create documentation that follows such a speed. Therefore, it is more valuable to spend time making code readable than writing heavy, easily outdated documentation. (Of course, this does not apply to APIs, but only to internal implementation.) The test suffers a bit from the same problem: Possibly also slow to write compared to the tested code. OTOH, it is a minor problem because the tests warn that they are outdated while your documentation remains mute unless you re-read them , very, very carefully .
Finally, a problem I sometimes find: automated tests can depend on tools, and those tools may be poorly written. I started a project with XUL some time ago and it is just painful to write unit tests for such a platform. I started another application using Objective-C, Cocoa and Xcode 3, and the test model was basically a set of workarounds.
I have had other experiences with the disadvantages of automated tests, but most of them are listed in other answers. Nevertheless, I am a vehement supporter of automated tests. This saved a lot of work and headaches and I always recommend it by default. I believe that these disadvantages are only details compared to the advantages of automated tests.