Posted on Jun\u00a020,\u00a02013<\/h4>\n\n<\/p>\nBy Bill Boyarsky<\/p>\n
One of the most disturbing aspects of the National Security Agency surveillance scandal is the way government has reportedly worked with private companies such as Yahoo, Facebook and Google. Those companies have disputed allowing the government direct access to their servers, as was originally reported by The Washington Post and The Guardian. The technology firms, Google foremost among them, have even pressed the government to be more transparent. However it was with their help that the NSA has devised a system to sort the massive amounts of data swept up by its snooping.<\/p>\n
The system, called Hadoop, was first disclosed by The Wall Street Journal and in depth last week by Salon. It is a way of storing, processing, classifying and analyzing billions of phone numbers, emails, texts, addresses, motor vehicle registrations, births, deaths, purchases and all of the other data we generate in our daily lives, speeding the information into the computers of such enterprises as Google, Facebook, Yahoo, as well as into the NSA database. The question to be answered is just what is being done with the data.<\/p>\n
\u201cHadoop\u2019s importance to how we live our lives today is hard to overstate,\u201d Andrew Leonard wrote in Salon on Friday. \u201cBy making it economically feasible to extract meaning from the massive streams of data that increasingly define our online existence, Hadoop effectively enabled the surveillance state. And not just in the narrowest, Big Brother, government-is-watching-everyone-all-the-time sense of that term. Hadoop is equally critical to private sector<\/em> corporate surveillance. Facebook, Twitter, Yahoo, Amazon, Netflix\u2013just about every big player that gathers the trillions of data \u2018events\u2019 generated by our everyday online actions employs Hadoop as a part of their arsenal of Big Data-crunching tools. Hadoop is everywhere\u2013as one programmer told me, \u2018it\u2019s taken over the world.\u2019\u00a0\u201d<\/p>\nI was introduced to Big Data during the last presidential campaign by Sasha Issenberg\u2019s articles in Slate and in his book \u201cThe Victory Lab.\u201d Issenberg wrote how a new generation of political campaigners was analyzing all this information to target voters, direct specific messages at them and make sure they got to the polls on Election Day. The process helped President Obama win re-election.<\/p>\n
Their data came from sources as simple as voter rolls and birth certificates, but it also drew on untold amounts of more sophisticated information compiled by firms that accumulate all kinds of particulars on individuals. One such company was Acxiom, which, Issenberg wrote, has gathered millions of details from Lands\u2019 End and other retailers, financial institutions such as Charles Schwab, auto dealers and magazine publishers and distributors. Much of the data included phone numbers and addresses.<\/p>\n
If it was an invasion of privacy, at first it didn\u2019t matter. There was just too much information. The accumulators could not keep track of what they had. Political campaigns weren\u2019t the only ones with the problem. Unmanageable data was pouring into Facebook, Google, Yahoo, other businesses\u2013and the Central Intelligence Agency and the National Security Agency.<\/p>\n
That\u2019s where Hadoop came in.<\/p>\n
I learned about its development in a series called \u201cThe History of Hadoop\u201d by Derrick Harris on the technical business website GigaOM.<\/p>\n
Harris reported that in 2002, Doug Cutting, an Internet search expert, and Mike Cafarella, a University of Washington graduate student, began work on an improved search engine. (Hadoop was named after Cutting\u2019s son\u2019s stuffed elephant.) At the time, Harris wrote, there were about a billion Web pages compared with what Wired magazine estimates as a trillion at present. <\/p>\n
Yahoo worked with Cutting and Cafarella as did Google and others. It was open source development; anyone with an idea and ability could jump in. The NSA and the CIA were among those that did. As a result, Leonard wrote, \u201cThe spooks and the social media titans and the online commerce goliaths\u201d have collaborated to track our behavior in \u201cfantastically intimate ways. …\u201d <\/p>\n
So our privacy was shattered long before Edward Snowden\u2019s revelations about snooping by the NSA, for which he did work while employed by private contractor Booz Allen Hamilton.<\/p>\n
Snowden was courageous in disclosing what he knew about the NSA and its PRISM project. But the outrage sparked by his leak has been centered on the NSA and the CIA. Nobody complains about the social media on which we volunteer every passing thought and record our life\u2019s events, great and small. Nor do we refrain from buying goods from Amazon.<\/p>\n
A blogger saw the danger in the collaboration of such powerful and pervasive commercial enterprises with the much more powerful government security organizations. This is how it was put it in a post on the Center for Digital Democracy\u2019s website June 11:<\/p>\n
\u201cThe dramatic global expansion of pervasive data collection by Google, Facebook, Yahoo and the digital marketing industry has created a commercially oriented surveillance system. But it\u2019s no surprise that such immediate access to our lives\u2013including rich details gathered by tracking our mobile device use (including location) and communications with friends (social media), as well as reams of first and third party data about our finances, health, personal and professional interests make it fertile territory for government collection.\u201d<\/p>\n
The huge data-gathering businesses are co-conspirators with the NSA and the CIA in the development and expansion of the surveillance state. They worked together to produce the Hadoop system that made their surveillance sweeps of the Internet possible. Now these businesses are trying to distance themselves from their government allies. I hope they don\u2019t get away with it.\n<\/p>\n
![\"\"](\"http:\/\/www.truthdig.com\/images\/eartothegrounduploads\/AP317260802424-facebook-sign-300.jpg\")
<\/p>\n
AP\/Marcio Jose Sanchez<\/p>\n
The Facebook \u201clike\u201d symbol is on display on a sign outside the company\u2019s headquarters in Menlo Park, Calif.\n<\/p>\n
<\/span>\n<\/div>\n
By Bill Boyarsky<\/p>\n
One of the most disturbing aspects of the National Security Agency surveillance scandal is the way government has reportedly worked with private companies such as Yahoo, Facebook and Google. Those companies have disputed allowing the government direct access to their servers, as was originally reported by The Washington Post and The Guardian. The technology firms, Google foremost among them, have even pressed the government to be more transparent. However it was with their help that the NSA has devised a system to sort the massive amounts of data swept up by its snooping.<\/p>\n
The system, called Hadoop, was first disclosed by The Wall Street Journal and in depth last week by Salon. It is a way of storing, processing, classifying and analyzing billions of phone numbers, emails, texts, addresses, motor vehicle registrations, births, deaths, purchases and all of the other data we generate in our daily lives, speeding the information into the computers of such enterprises as Google, Facebook, Yahoo, as well as into the NSA database. The question to be answered is just what is being done with the data.<\/p>\n
\u201cHadoop\u2019s importance to how we live our lives today is hard to overstate,\u201d Andrew Leonard wrote in Salon on Friday. \u201cBy making it economically feasible to extract meaning from the massive streams of data that increasingly define our online existence, Hadoop effectively enabled the surveillance state. And not just in the narrowest, Big Brother, government-is-watching-everyone-all-the-time sense of that term. Hadoop is equally critical to private sector<\/em> corporate surveillance. Facebook, Twitter, Yahoo, Amazon, Netflix\u2013just about every big player that gathers the trillions of data \u2018events\u2019 generated by our everyday online actions employs Hadoop as a part of their arsenal of Big Data-crunching tools. Hadoop is everywhere\u2013as one programmer told me, \u2018it\u2019s taken over the world.\u2019\u00a0\u201d<\/p>\n I was introduced to Big Data during the last presidential campaign by Sasha Issenberg\u2019s articles in Slate and in his book \u201cThe Victory Lab.\u201d Issenberg wrote how a new generation of political campaigners was analyzing all this information to target voters, direct specific messages at them and make sure they got to the polls on Election Day. The process helped President Obama win re-election.<\/p>\n Their data came from sources as simple as voter rolls and birth certificates, but it also drew on untold amounts of more sophisticated information compiled by firms that accumulate all kinds of particulars on individuals. One such company was Acxiom, which, Issenberg wrote, has gathered millions of details from Lands\u2019 End and other retailers, financial institutions such as Charles Schwab, auto dealers and magazine publishers and distributors. Much of the data included phone numbers and addresses.<\/p>\n If it was an invasion of privacy, at first it didn\u2019t matter. There was just too much information. The accumulators could not keep track of what they had. Political campaigns weren\u2019t the only ones with the problem. Unmanageable data was pouring into Facebook, Google, Yahoo, other businesses\u2013and the Central Intelligence Agency and the National Security Agency.<\/p>\n That\u2019s where Hadoop came in.<\/p>\n I learned about its development in a series called \u201cThe History of Hadoop\u201d by Derrick Harris on the technical business website GigaOM.<\/p>\n Harris reported that in 2002, Doug Cutting, an Internet search expert, and Mike Cafarella, a University of Washington graduate student, began work on an improved search engine. (Hadoop was named after Cutting\u2019s son\u2019s stuffed elephant.) At the time, Harris wrote, there were about a billion Web pages compared with what Wired magazine estimates as a trillion at present. <\/p>\n Yahoo worked with Cutting and Cafarella as did Google and others. It was open source development; anyone with an idea and ability could jump in. The NSA and the CIA were among those that did. As a result, Leonard wrote, \u201cThe spooks and the social media titans and the online commerce goliaths\u201d have collaborated to track our behavior in \u201cfantastically intimate ways. …\u201d <\/p>\n So our privacy was shattered long before Edward Snowden\u2019s revelations about snooping by the NSA, for which he did work while employed by private contractor Booz Allen Hamilton.<\/p>\n Snowden was courageous in disclosing what he knew about the NSA and its PRISM project. But the outrage sparked by his leak has been centered on the NSA and the CIA. Nobody complains about the social media on which we volunteer every passing thought and record our life\u2019s events, great and small. Nor do we refrain from buying goods from Amazon.<\/p>\n A blogger saw the danger in the collaboration of such powerful and pervasive commercial enterprises with the much more powerful government security organizations. This is how it was put it in a post on the Center for Digital Democracy\u2019s website June 11:<\/p>\n \u201cThe dramatic global expansion of pervasive data collection by Google, Facebook, Yahoo and the digital marketing industry has created a commercially oriented surveillance system. But it\u2019s no surprise that such immediate access to our lives\u2013including rich details gathered by tracking our mobile device use (including location) and communications with friends (social media), as well as reams of first and third party data about our finances, health, personal and professional interests make it fertile territory for government collection.\u201d<\/p>\n The huge data-gathering businesses are co-conspirators with the NSA and the CIA in the development and expansion of the surveillance state. They worked together to produce the Hadoop system that made their surveillance sweeps of the Internet possible. Now these businesses are trying to distance themselves from their government allies. I hope they don\u2019t get away with it.\n<\/p>\n AP\/Marcio Jose Sanchez<\/p>\n The Facebook \u201clike\u201d symbol is on display on a sign outside the company\u2019s headquarters in Menlo Park, Calif.\n<\/p>\n <\/span>\n<\/div>\n<\/p>\n