Data Mining, The Internet, and Counter Intelligence

We can, and do, talk about Data Privacy and Ownership until we're blue in the face; we also talk about how seriously screwed up some of the things the National Security Agency did were, but we never really harp on the obvious fact:  All the security in the world doesn't do a damn bit of good if you give your information away; one ambitious JSON project over on Github called Looking Glass is capitalizing on just that very fact.

In fact, as of this publication, they have data mined over 139,361 resumes belonging to military and civilian officials within our nation's Intelligence, Surveillance, and Reconnaissance (ISR) fields.  Those handy little endorsements have been used by job seekers to categorize them into fields (e.g. "Security Clearance" or "ISR") where data miners have been more than willing to scoop that  information up.

 # Get all profiles within numhops of original(s)
  def getRelatedProfiles
    @numhops.times do |hop_count| { |profile| profile[:degree] == hop_count }.each do |item|
        downloadRelated(item, hop_count) if item[:related_people]
   # Add a score to each profile based on the # of times it appears in "people also viewed"
  def relScore(data)
    profile_scores = fullProfileList(data)
    # Get degree and calculate score for each profile
    data.each do |data_item|
      if data_item[:related_people]
        data_item[:related_people].each do |person|
          profile_scores = addPointsToProfile(profile_scores, data_item, person)

Unfortunately for those individuals who don't post these types of things to their profile (kudos), it doesn't matter much.  In a rather elegant (and well commented (kudos)) fashion, the web crawler rather quickly goes through all of the people most like those who tagged themselves with these keywords.

Essentially, it turns LinkedIn's most powerful asset, People You May Know, against those who arguably have the most to lose from the wrong people knowing about them.

And while the average American, let alone the average German, may not have a lot of heartburn with more transparency in who works within one of the most opaque industries on the planet, this is a huge example of the importance of good security in our every-day lives.


It doesn't matter how many books we read cautioning us against the insightful and insurmountably detailed power of "Big Data" if we throw it all out the window just to find a job or to participate in some new meme.  And while this data isn't particularly damning to these individuals (in fact, some of the information could be very beneficial to helping us understand government spending, our surveillance programs, and the like), it could be combined with other very public information (like an unsecured Facebook page), to identify and locate people who probably would very much rather remain anonymous.