{"id":53942,"date":"2013-07-29T02:44:13","date_gmt":"2013-07-29T01:44:13","guid":{"rendered":"http:\/\/rinf.com\/alt-news\/breaking-news\/big-data-dynamo-how-giant-tech-firms-help-the-government-to-spy-on-americans\/53942\/"},"modified":"2013-07-29T02:44:13","modified_gmt":"2013-07-29T01:44:13","slug":"big-data-dynamo-how-giant-tech-firms-help-the-government-to-spy-on-americans","status":"publish","type":"post","link":"http:\/\/rinf.com\/alt-news\/breaking-news\/big-data-dynamo-how-giant-tech-firms-help-the-government-to-spy-on-americans\/","title":{"rendered":"\u2018Big Data\u2019 Dynamo: How Giant Tech Firms Help the Government to Spy on Americans"},"content":{"rendered":"

As the secret state continues trawling the electronic communications of hundreds of millions of Americans, lusting after what securocrats euphemistically call \u201cactionable intelligence,\u201d a notional tipping point that transforms a \u201cgood\u201d citizen into a \u201ccriminal\u201d suspect, the role played by telecommunications and technology firms cannot be emphasized enough.<\/em><\/p>\n

Ever since former NSA contractor Edward Snowden began leaking secrets to media outlets about government surveillance programs, one fact stands out: The zero probability<\/em> these privacy-killing projects would be practical without close (and very profitable) \u201carrangements\u201d made with phone companies, internet service providers and other technology giants.
\"\"<\/a><\/p>\n

Indeed, a top secret NSA Inspector General\u2019s report published by The Guardian<\/a><\/em>, revealed that the agency \u201cmaintains relationships with over 100 US companies,\u201d adding that the US has the \u201chome field advantage as the primary hub for worldwide telecommunications.\u201d<\/p>\n

Similarly, the British fiber optic cable tapping program, TEMPORA<\/a>, referred to telcos and ISPs involved in the spying as \u201cintercept partners.\u201d The names of the firms were considered so sensitive that GCHQ \u201cwent to great lengths\u201d to keep their identities hidden, fearing exposure \u201cwould cause \u2018high-level political fallout\u2019.\u201d<\/p>\n

With new privacy threats looming on the horizon, including what CNET<\/a> described as ongoing efforts by the FBI and NSA \u201cto obtain the master encryption keys that Internet companies use to shield millions of users\u2019 private Web communications from eavesdropping,\u201d along with new government demands<\/a> that ISPs and cell phone carriers \u201cdivulge users\u2019 stored passwords,\u201d can we trust these firms?<\/p>\n

And with Microsoft<\/a> and other tech giants, collaborating closely with \u201cUS intelligence services to allow users\u2019 communications to be intercepted, including helping the National Security Agency to circumvent the company\u2019s own encryption,\u201d can we afford to?<\/p>\n

Hiding in Plain Sight<\/strong><\/p>\n

Ever since retired union technician Mark Klein blew the lid off AT&T\u2019s secret surveillance pact with the US government in 2006, we know user privacy is not<\/em> part of that firm\u2019s business model.<\/p>\n

The technical source for the Electronic Frontier Foundation\u2019s lawsuit, Hepting v. AT&T<\/a><\/em> and the author of Wiring Up the Big Brother Machine<\/a><\/em>, Klein was the first to publicly expose how NSA was \u201cvacuuming up everything flowing in the Internet stream: e-mail, web browsing, Voice-Over-Internet phone calls, pictures, streaming video, you name it.\u201d<\/p>\n

We also know from reporting by USA Today<\/a><\/em>, that the agency \u201chas been secretly collecting the phone call records of tens of millions of Americans\u201d and had amassed \u201cthe largest database ever assembled in the world.\u201d<\/p>\n

Three of those data-slurping programs, UPSTREAM, PRISM and X-KEYSCORE, shunt domestic and global communications collected from fiber optic cables, the servers of Apple, Google, Microsoft and Yahoo, along with telephone data (including metadata, call content and location) grabbed from AT&T, Sprint and Verizon into NSA-controlled databases.<\/p>\n

But however large, a database is only useful to an organization, whether its a corporation or a spy agency, if the oceans of data collected can be searched and extracted in meaningful ways.<\/p>\n

To the growing list of spooky acronyms and code-named black programs revealed by Edward Snowden, what other<\/em> projects, including those in the public domain, are hiding in plain sight?<\/p>\n

Add Google\u2019s BigTable<\/a> and Yahoo\u2019s Hadoop<\/a> to that list. Both are massive storage and retrieval systems designed to crunch ultra-large data sets and were developed as a practical means to overcome \u201cbig data\u201d conundrums.<\/p>\n

According to the Mountain View behemoth, \u201cBigTable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers.\u201d Along with web indexing, Google Earth and Google Finance, BigTable performs \u201cbulk processing\u201d for \u201creal-time data serving.\u201d<\/p>\n

Down the road in Sunnyvale, Yahoo developed Hadoop as \u201can open source Java framework for processing and querying vast amounts of data on large clusters of commodity hardware.\u201d According to Yahoo, Hadoop has become \u201cthe industry de facto<\/em> framework for big data processing.\u201d Like Google\u2019s offering, Hadoop enable applications to work with thousands of computers and petabytes of data simultaneously.<\/p>\n

Prominent corporate clients using these applications include Amazon, AOL, eBay, Facebook, IBM, Microsoft and Twitter, among many others.<\/p>\n

\u2018Big Data\u2019 Dynamo<\/strong><\/p>\n

Who might also<\/em> have a compelling interest in cataloging and searching through very large data sets, away from prying eyes, and at granular levels to boot? It should be clear following Snowden\u2019s disclosures, what\u2019s good for commerce is also a highly-prized commodity among global eavesdroppers.<\/p>\n

Despite benefits for medical and scientific researchers sifting through mountains of data, as Ars Technica<\/a><\/em> pointed out BigTable and Hadoop \u201clacked compartmentalized security\u201d vital to spy shops, so \u201cin 2008, NSA set out to create a better version of BigTable, called Accumulo.\u201d<\/p>\n

Developed by agency specialists, it was eventually handed off to the \u201cnon-profit\u201d Apache Software Foundation. Touted as an open software platform, Accumulo<\/a> is described in Apache literature as \u201ca robust, scalable, high performance data storage and retrieval system.\u201d<\/p>\n

\u201cThe platform allows for compartmentalization of segments of big data storage through an approach called cell-level security. The security level of each cell within an Accumulo table can be set independently, hiding it from users who don\u2019t have a need to know: whole sections of data tables can be hidden from view in such a way that users (and applications) without clearance would never know they weren\u2019t there,\u201d Ars Technica<\/a><\/em> explained.<\/p>\n

The tech site Gigaom<\/a><\/em> noted, Accumulo is the \u201ctechnological linchpin to everything the NSA is doing from a data-analysis perspective,\u201d enabling agency analysts to \u201cgenerate near real-time reports from specific patterns in data,\u201d Ars<\/em> averred.<\/p>\n

\u201cFor instance, the system could look for specific words or addressees in e-mail messages that come from a range of IP addresses; or, it could look for phone numbers that are two degrees of separation from a target\u2019s phone number. Then it can spit those chosen e-mails or phone numbers into another database, where NSA workers could peruse it at their leisure.\u201d<\/p>\n

(Since that Ars<\/em> piece appeared, we have since learned that NSA is now conducting what is described as \u201cthree-hop analysis,\u201d that is, three degrees of separation<\/em> from a target\u2019s email or phone number. This data dragnet \u201ccould allow the government to mine the records of 2.5 million Americans when investigating one suspected terrorist,\u201d the Associated Press<\/a><\/em> observed).<\/p>\n

\u201cIn other words,\u201d Ars<\/em> explained, \u201cAccumulo allows the NSA to do what Google does with your e-mails and Web searches\u2014only with everything that flows across the Internet, or with every phone call you make.\u201d<\/p>\n

Armed with a \u201cdual-use\u201d program like Accumulo, the dirty business of assembling a user\u2019s political profile, or shuttling the names of \u201csuspect\u201d Americans into a national security index, is as now easy as downloading a song from iTunes!<\/p>\n

And it isn\u2019t only Silicon Valley giants cashing-in on the \u201cpublic-private\u201d spy game.<\/p>\n

Just as the CIA-funded Palantir<\/a>, a firm currently valued at $8 billion and exposed two years ago as a \u201cpartner\u201d in a Bank of America-brokered scheme to bring down WikiLeaks<\/a>, profited from CIA interest in its social mapping Graph<\/a> application, so too, the NSA spin-off Sqrrl<\/a>, launched in 2012 with agency blessings, stands to make a killing off software its corporate officers helped develop for NSA.<\/p>\n

Co-founded by nine-year agency veteran Adam Fuchs, Sqrrl sells commercial versions of Accumulo and has partnered-up with Amazon, Dell, MapR and Northrop Grumman. According to published reports, like other start-ups with an intelligence angle, Sqrrl is hoping to hook-up with CIA\u2019s venture capital arm In-Q-Tel<\/a>.<\/p>\n

Its obvious why the application is of acute interest to American spy shops. Fuchs told Gigaom<\/em> that Accumulo operates \u201cat thousands-of-nodes scale\u201d within NSA data centers.<\/p>\n

\u201cThere are multiple instances each storing tens of petabytes (1 petabyte equals 1,000 terabytes or 1 million gigabytes) of data and it\u2019s the backend of the agency\u2019s most widely used analytical capabilities.\u201d<\/p>\n

Accumulo\u2019s analytical functions work because of its ability to perform lightning-quick searches called \u201cgraph analysis,\u201d a method for uncovering unique relationships between people hidden within vast oceans of data.<\/p>\n

According to Forbes<\/a><\/em>, \u201cwe know that the NSA has successfully tested Accumulo\u2019s graph analysis capabilities on some huge data sets\u2014in one case on a 1200 node Accumulo cluster with over a petabyte of data and 70 trillion edges.\u201d<\/p>\n

Considering, as Wired<\/a><\/em> reported, that \u201con an average day, Google accounts for about 25 percent of all consumer internet traffic running through North American ISPs,\u201d and the Mountain View firm allowed the FBI and NSA to tap directly into their central servers as The Washington Post<\/a><\/em> disclosed, the negative impact on civil rights and political liberties when systems designed for the Pentagon are monetized, should be evident.<\/p>\n

Once fully commercialized, how much more intrusive will employers, marketing firms, insurance companies or local and state police with mountains of data only a mouse click away, become?<\/p>\n

Global Panopticon<\/strong><\/p>\n

The sheer scope of NSA programs such as UPSTREAM, PRISM or X-KEYSCORE, exposed by the Brazilian daily, O Globo<\/a><\/em> should give pause.<\/p>\n

A crude illustration (at the top of this post), shows that all data collected in X-KEYSCORE \u201csessions\u201d are processed in petabyte scale batches captured from \u201cweb-based searches\u201d that can be \u201cretrospectively\u201d queried to locate and profile a \u201ctarget.\u201d<\/p>\n

This requires enormous processing power; a problem the agency may<\/em> have solved with Accumulo or similar applications.<\/p>\n

Once collected, data is separated into digestible fragments (phone numbers, email addresses and log ins), then reassembled at lightning speeds for searchable queries in graphic form. Information gathered in the hopper includes not only metadata tables, but the \u201cfull log,\u201d including what spooks call Digital Network Intelligence, i.e., user content.<\/p>\n

And while it may not yet<\/em> be practical for NSA to collect and store each single packet flowing through the pipes, the agency is already<\/em> collecting and storing vast reams of data intercepted from our phone records, IP addresses, emails, web searches and visits, and is doing so in much the same way that Amazon, eBay, Google and Yahoo does.<\/p>\n

As the volume of global communications increase each year at near exponential levels, data storage and processing pose distinct problems.<\/p>\n

Indeed, Cisco Systems forecast in their 2012 Visual Networking Index<\/a> that global IP traffic will grow three-fold over the next five years and will carry up to 4 exabytes of data per day, for an annual rate of 1.4 zettabytes by 2017.<\/p>\n

This does much to explain why NSA is building a $2 billion Utah Data Center with 22 acres of digital storage space that can hold up to 5 zettabytes of data and expanding already existing centers at Fort Gordon, Lackland Air Force Base, NSA Hawaii and at the agency\u2019s Fort Meade headquarters.<\/p>\n

Additionally, NSA is feverishly working to bring supercomputers online \u201cthat can execute a quadrillion operations a second\u201d at the Multiprogram Research facility in Oak Ridge, Tennessee where enriched uranium for nuclear weapons is manufactured, as James Bamford disclosed last year in Wired<\/a><\/em>.<\/p>\n

As the secret state sinks tens of billions of dollars into various big data digital programs, and carries out research on next-gen cyberweapons more destructive than Flame or Stuxnet, as those supercomputers come online the cost of cracking encrypted passwords and communications will continue to fall.<\/p>\n

Stanford University computer scientist David Mazi\u00c3\u00a8res told CNET that mastering encrypted communications would \u201cinclude an order to extract them from the server or network when the user logs in\u2014which has been done before\u2014or installing a keylogger at the client.\u201d<\/p>\n

This is precisely<\/em> what Microsoft has already done with its SkyDrive cloud storage service \u201cwhich now has 250 million users worldwide\u201d and exabytes of data ready to be pilfered, as The Guardian<\/em> disclosed.<\/p>\n

One document \u201cstated that NSA already had pre-encryption access to Outlook email. \u2018For Prism collection against Hotmail, Live, and Outlook.com emails will be unaffected because Prism collects this data prior to encryption\u2019.\u201d<\/p>\n

Call the \u201cwrong\u201d person or click a dodgy link and you might just be the lucky winner of a one-way trip to indefinite military detention under NDAA<\/a>, or worse.<\/p>\n

What should also be clear since revelations about NSA surveillance programs began spilling out last month, is not a single ruling class sector in the United States\u2014including corporations, the media, nor any branch of the US government\u2014has the least interest in defending democratic rights or rolling-back America\u2019s emerging police state.<\/p>\n

Republished from: Global Research<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

As the secret state continues trawling the electronic communications of hundreds of millions of Americans, lusting after what securocrats euphemistically call \u201cactionable intelligence,\u201d a notional tipping point that transforms a \u201cgood\u201d citizen into a \u201ccriminal\u201d suspect, the role played by telecommunications and technology firms cannot be emphasized enough. Ever since former NSA contractor Edward Snowden […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[487],"tags":[],"class_list":{"0":"post-53942","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-breaking-news"},"_links":{"self":[{"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/posts\/53942","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/comments?post=53942"}],"version-history":[{"count":0,"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/posts\/53942\/revisions"}],"wp:attachment":[{"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/media?parent=53942"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/categories?post=53942"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/rinf.com\/alt-news\/wp-json\/wp\/v2\/tags?post=53942"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}