Why is good intelligence on the worst regimes so hard to come by these days?

Spook86 has posted an illuminating discussion at In From the Cold about the limitations of sources and methods used for intelligence gathering in this era:

The growing imbroligo over the Iran NIE is raising new concerns about the intelligence sources and methods used to produce the document--and its controversial judgments. [...]

While there are legitimate concerns about the personal biases and suspected political agendas of the study's primary authors, Americans should also worry about the quality and reliability of the information used in formulating the assessment. [...]

As with any National Intelligence Estimate, we assume the new Iran assessment makes use of the full array of intel sources and methods--SIGINT, HUMINT, IMINT, MASINT and even open-source reporting. But we also recognize that information from these same sources led to a dramatically different conclusion just two years ago. Moreover, the volume and quality of collection from these platforms has not improved dramatically--as far as we can tell. Technological refinements in our intel systems are offset by the adversary's own advances, and their attempts at denial and deception.

Consider the example of signals intelligence, or SIGINT. The National Security Agency (NSA) remains the preeminent SIGINT organization in the world. But agency veterans will tell you that the SIGINT environment has become increasingly challenging, thanks to the proliferation of fiber-optic technology and low-cost encryption devices. Phone calls and other communications that once bounced between relay towers are now routed over fiber-optic cable; intercepting them means tapping into the line, a difficult proposition in places like Iran, Syria or North Korea.

The problem is further compounded by wide availability of personal encryption devices. Complex cyphers that were once the exclusive property of governments and intelligence services can now be downloaded from the internet. Increased use of these systems and devices slows the decrypt of adversary communication--and the flow of information to decision-makers.

What about cell phones, you ask? They operate on a tower-based line-of-sight system. True, but intercepting those transmissions (usually) means getting inside hostile territory, further complicating the collection task.

And SIGINT isn't the only intel discipline facing such challenges. In the internet era, there are scores of websites that offer information on the orbits and potential collection "windows" for spy satellites. Today, an effective satellite warning program is just a few keystrokes away--and there's little we can do about it.

HUMINT? The Robb-Silberman Commission Report (released in 2005) deplored the state of our HUMINT capabilities, noting the meager haul on Saddam's WMD programs in the run-up to the war, and the wholesale lack of reporting on Iraqi leadership intentions. Similar problems are said to exist with the Iranian "target," which presents similar challenges. Clearly, there have been no major "breakthroughs" in traditional collection methods over the past two years. [..]

Read more for the former spook's discussion of the intelligence used for the latest NIE report on Iran.