{"id":1613,"date":"2007-10-30T15:39:39","date_gmt":"2007-10-30T15:39:39","guid":{"rendered":"http:\/\/rinf.com\/alt-news\/surveillance-big-brother\/att-invents-programming-language-for-mass-surveillance\/1613\/"},"modified":"2007-10-30T15:41:35","modified_gmt":"2007-10-30T15:41:35","slug":"att-invents-programming-language-for-mass-surveillance","status":"publish","type":"post","link":"http:\/\/rinf.com\/alt-news\/surveillance-big-brother\/att-invents-programming-language-for-mass-surveillance\/","title":{"rendered":"The Programming Language for Mass Surveillance"},"content":{"rendered":"

According to government documents studied by The New York Times<\/a>, the FBI asked several phone companies to analyze phone-call patterns of Americans using a technology called \u201ccommunities of interest\u201d. Verizon refused, saying that it didn\u2019t have any such technology. AT&T, famously, did not refuse.<\/span><\/span>What is the \u201ccommunities of interest\u201d technology? It\u2019s spelled out very clearly in a 2001 research paper from AT&T itself, entitled \u201cCommunities of Interest<\/a>\u201d (by C. Cortes, D. Pregibon, and C. Volinsky). They use high-tech data-mining algorithms to scan through the huge daily logs of every call made on the AT&T network; then they use sophisticated algorithms to analyze the connections between phone numbers: who is talking to whom? The paper literally uses the term \u201cGuilt by Association\u201d to describe what they\u2019re looking for: what phone numbers are in contact with other numbers that are in contact with the bad guys?<\/p>\n

When this research was done, back in the last century, the bad guys where people who wanted to rip off AT&T by making fraudulent credit-card calls. (Remember, back in the last century, intercontinental long-distance voice communication actually cost money!) But it\u2019s easy to see how the FBI could use this to chase down anyone who talked to anyone who talked to a terrorist. Or even to a \u201cterrorist.\u201d<\/p>\n

AT&T Invents Surveillance Programming Language<\/strong>\u00a0\u00a0<\/p>\n

By Ryan Singel<\/span><\/span><\/p>\n

<\/span><\/span><\/span><\/span>From the company that brought you the C programming language comes Hancock, a C variant developed by AT&T researchers to mine gigabytes of the company’s telephone and internet records for surveillance purposes.<\/p>\n

<\/p>\n

An AT&T research paper<\/a> published in 2001 and unearthed today by Andrew Appel at Freedom to Tinker<\/a> shows how the phone company uses Hancock-coded software to crunch through tens of millions of long distance phone records a night to draw up what AT&T calls “communities of interest” — i.e., calling circles that show who is talking to whom.<\/p>\n

The system was built in the late 1990s to develop marketing leads, and as a security tool to see if new customers called the same numbers as previously cut-off fraudsters — something the paper refers to as “guilt by association.”<\/p>\n

But it’s of interest to THREAT LEVEL because of recent revelations that the FBI has been requesting “communities of interest” records from phone companies under the USA PATRIOT Act without a warrant. Where the bureau got the idea that phone companies collect such data has, until now, been a mystery.<\/p>\n

According to a letter from Verizon to a congressional committee earlier this month, the FBI has been asking Verizon for “community of interest” records on some of its customers out to two generations — i.e., not just the people that communicated with an FBI target, but also those who talked to people who talked to an FBI target. Verizon, though, doesn’t create those records and couldn’t comply. Now it appears that AT&T invented the concept and the technology. It even owns a patent on some of its data mining methods<\/a>, issued to two of Hancock’s creators in 2002.<\/p>\n

Programs written in Hancock work by analyzing data as it flows into a data warehouse. That differentiates the language from traditional data-mining applications which tend to look for patterns in static databases. A 2004 paper published in ACM Transactions on Programming Languages and Systems<\/em> shows how Hancock code can sift calling card records, long distance calls, IP addresses and internet traffic dumps, and even track the physical movements of mobile phone customers as their signal moves from cell site to cell site.\u00a0<\/p>\n

With Hancock, “analysts could store sufficiently precise information to enable new applications previously thought to be infeasible,” the program authors wrote. AT&T uses Hancock code to sift 9 GB of telephone traffic data a night, according to the paper.<\/p>\n

The good news for budding data miners is that Hancock’s source code and binaries<\/a> (now up to version 2.0) are available free to noncommercial users from an AT&T Research website.<\/p>\n

The instruction manual<\/a> (.pdf) is also free, and old-timers will appreciate its spare Kernighan & Ritchie style. The manual even includes a few sample programs in the style of K&R’s Hello World, but coded specifically to handle data collected by AT&T’s phone and internet switches. This one reads in a dump of internet headers, computes what IP addresses were visited, makes a record and prints them out, in less than 40 lines of code.<\/p>\n

\n
#include \"ipRec.hh\" \r\n#include \"ihash.h\" \r\n\r\nhash_table *ofInterest; \r\n\r\nint inSet (ipPacket_t * p) \r\n{ \r\n if (hash_get (ofInterest, p->source.hash_value) == 1) \r\n  return 1; \r\n if (hash_get (ofInterest, p->dest.hash_value) == 1) \r\n  return 1; \r\n return 0; \r\n} \r\nvoid sig_main<\/strong> (ipAddr_s addrs < l:>, \r\n{ \r\n \/* code to set up hash table *\/ \r\n ofInterest = hash_empty (); \r\n iterate<\/strong> \r\n  (over<\/strong> addrs) { \r\n  event<\/strong> (ipAddr_t * addr) { \r\n \u00a0 \u00a0if (hash_insert (ofInterest, addr->hash_value, 1) < 0) \r\n  } \r\n } \r\n \/* code to select packets *\/ \r\n iterate<\/strong> \r\n  (over<\/strong> packets \r\n \u00a0 filteredby inSet) \r\n { \r\n  event<\/strong> (ipPacket_t * p) \r\n  { \r\n \u00a0 \u00a0printPacketInfo (p); \r\n  } \r\n }; \r\n}<\/pre>\n<\/blockquote>\n
Another sample program included in the manual shows how a Hancock program could create historical maps of a person’s travels by recording nightly what cell phone towers a person’s phone had used or pinged throughout a day.<\/p>\n
AT&T is currently defending itself in federal court from allegations that it installed, on behalf of the NSA, secret internet spying rooms in its domestic internet switching facilities. AT&T and Verizon are also accused of giving the NSA access to billions of Americans’ phone records, in order to data-mine them to spot suspected terrorists, and presumably to identify targets for warrantless wiretapping.<\/p>\n","protected":false},"excerpt":{"rendered":"
According to government documents studied by The New York Times, the FBI asked several phone companies to analyze phone-call patterns of Americans using a technology called \u201ccommunities of interest\u201d. Verizon refused, saying that it didn\u2019t have any such technology. AT&T, famously, did not refuse.What is the \u201ccommunities of interest\u201d technology? It\u2019s spelled out very clearly […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1614],"tags":[30],"class_list":{"0":"post-1613","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-surveillance-big-brother","7":"tag-big-brother"},"_links":{"self":[{"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/posts\/1613","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/comments?post=1613"}],"version-history":[{"count":0,"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/posts\/1613\/revisions"}],"wp:attachment":[{"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/media?parent=1613"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/categories?post=1613"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/tags?post=1613"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}