We live in a world where flu outbreaks are predicted faster and more accurately by analysing Google search results rather than by doctors or clinicians, where traffic jams are better judged by crunching data from cellphone signals rather than from direct reports from people on the ground, and where your shopping habits might reveal that you’re pregnant before anyone else in your family knows.

This is the power of big data. It is defined not by the sheer volume of information, but by what that large volume enables us to do that similar smaller volume of data wouldn’t. For example, Google’s flu trends will hardly work if the amount of queries made per second were not in the thousands. Another aspect of big data is that using it means shedding our obsession for causality and embracing correlations. The important thing is to know what rather than the why.

Take the example of how Google’s page ranking system works. The computer algorithm at the heart of Google search that is analysing data from all across the web is not trying to understand what the websites say or mean, so much as it is trying to correlate what people want when they type something in the search query. More queries followed by more clicks on relevant sites will better the algorithm that ranks pages, helping it to make better predictions which links will work best. Now add to it information like a person’s search history, location, time of the day, etc. and Google is able to give near perfect search results.

The book also rightly argues that despite data’s ubiquitous use today, the revolution has only just begun. A lot of the information in the world still remains locked or wasted. Consider the example of electrocardiography (ECG). When a patient undergoes ECG, hundreds of data points are collected every second but most of it gets thrown away. Instead the capability of easy storage (thus never needing to throw away any data) can be used to make better predictions of the patient’s health in the future. Datafication, which is recording everything possible, can unlock information around us. Things which might seem uninteresting could, in combination with other data, reveal insights that we could not have guessed before

Big data’s use does not paint a uniformly rosy picture. Governments are desperately trying to control more and more data from their citizens’ lives under the guise of security concerns. But, just like private companies do, the data can easily be employed by governments for uses that citizens would not approve of, if they were asked to give consent. The story of Minority Report could come true. In it the government develops a system that is used to predict the future occurrence of crime and make arrests in time to stop it. These sort of uses are still science fiction, but not for long. They risk taking away from humanity its most dear capacity—to act on “free will”.

It is this dual-edged sword of big data that makes Messrs Cukier and Mayer-Schonberger’s book timely and important. Written beautifully and convincingly, it makes for a great read. Where I don’t agree with the book is that big data “will transforming how we live, work and think”. I think it already has.

Less than a decade ago, it was easy to recognise a book. It was anything that could be printed, bound and put on shelf of a library or a store. Now, though, things have gotten messy.  There are ebooks, Kindle singles, Atavist originals, Matter stories, and the list goes on.

In many parts of the world, digital has become the primary platform for the written word. The advantages are plenty and this trend towards digital is no surprise. But it disrupts how ideas get shared, and sharing ideas was the main reason for books to come into existence.

While it was with the classical definition of a book that I began my #100bookschallenge, the main reason behind taking up the challenge was to be able to learn about the greatest ideas out there. These are increasingly being communicated not just in books. A lot of the ideas are long conversations that have been running on a blog, or those that appear in longform writing/journalism like the New Yorker or The Economist’s special reports.

Thus I’m revising the definition of a (non-fiction) book that can count towards my challenge of 100 books. Apart from the classical definition, all pieces of writing that will fulfill all the conditions below can be counted towards my target:

• Longform writing that has a clear-defined message or explores a topic in a significant amount of detail or has a central theme.
• Has been written by a single author (‘classical’ books may have more than one author).
• Is more than 10,000 words long as a single piece.
• A series of blog posts won’t count if at least one of them is not close to 10,000 words long and explains the main idea of the series.
• The writing should be so dense (full of ideas) that I cannot stop myself from writing a review of what I read.
• (UPDATE) The work should be not just newsworthy ie it should still relevant and worth reading after, say, many months or sometimes years.

As to why just 10,000 words? Because it’s not too short and it feels like the right length to have a comprehensive look at a topic. I’m open to revising my definition, so please feel free to make suggestions.

R: What kind of risks do you think we overlook most in day to day life?

T: The answer to your question is in the following: 7000 Americans die every day, many, many of preventable causes. What we talk about is usually the sensational. Do the math: they die from lack of stressors (activity), corn syrup, cigarettes, etc. So the real risks/killers are discernible; they map to the risks for your life.

R: Is your freedom the only source to go on fighting with such fervor?

T: I have always been fighting… But my freedom gives me more moral obligations, make me feel more guilt for not shouting fraud when I see it.

R: What kind of system would you set up in order to promote anti-fragility?

T: Rule: any company that would cause a national emergency requiring a bailout should it fail should be classified BAILABLE-OUT and employees should not be allowed to earn more than civil servants. That would force companies to 1) be small, 2) not leech off the taxpayer.

R: What is the most important skill or trait a human being can have in the modern world?

T: A sense of honor. It puts you above everything else.

R: What is one thing that a recent college graduate can due to be Antifragile?

T: Get passing grades and follow voraciously your curiosity on the side instead of competing in school. In the end what matters is your curiosity, nothing else. And read nothing that doesn’t interest you but interests someone else.

R: You’ve talked a lot about financial issues and health issues. You have touched on the environment, but not said much about energy use.

T: The problem is the nonlinearity of harm. We have too many people on the planet, with too much concentration of pollutants. And these people are converging to the same habits.. We are not supposed to be eating the same thing. Any concentration harms.

R: What can the average joe do to make sure “skin in the game” is enforced on those in power?

T: Decentralization is where we start. Vote for that and for people promoting it.

R: What have you been reading recently?

R: You can check out his amazon reviews if you haven’t seen that yet. Link

R: How many books in your library have you not read?

T: Actually, only 40% partially read.

R: Can you begin to be antifragile while being poor or you should first make some money and plan ahead?

T: The poor is more antifragile than the rich: less to lose, both economically and psychologically.

R: As an engineer and technologist, I’m exposed to a lot of neophilia. Do you have any suggestions for heuristics besides reading the classics as an inoculation against neophilia?

T: Yes, use the Lindy effect as a testing rule… that is, look for solutions from simpler technologies.

The longer a technology has been around, the longer it’s likely to stay around.

R: according to your principles, how would you deal with the obesity epidemic hitting the U.S.?

T: The general problem is that we are not made to control our environment, and we are designed for a degree of variability: in energy, temperature, food composition, sleep duration, exercise (by Jensen’s inequality). Depriving anyone of variations is silly. So we need to force periods of starvation/fasts , sleep deprivation , protein deprivation, etc. Religions force shabbats, fasts, etc. but we are no longer under the sway of religions… The solution is rules…

It can be very troubling to realise that inside my own head I maybe tucking away truths that I am already aware of!

Now Eli Dourado has an essay in The Umlaut that discusses a paper by Tyler Cowen and Robin Hanson that grapples with the issue of self-deception.  Here’s the relevant bit:

Self-favoring priors, they note, can help to serve other functions besides arriving at the truth. People who “irrationally” believe in themselves are often more successful than those who do not. Because pursuit of the truth is often irrelevant in evolutionary competition, humans have an evolved tendency to hold self-favoring priors and self-deceive about the existence of these priors in ourselves, even though we frequently observe them in others.

Self-deception is in some ways a more serious problem than mere lack of intelligence. It is embarrassing to be caught in a logical contradiction, as a stupid person might be, because it is often impossible to deny. But when accused of disagreeing due to a self-favoring prior, such as having an inflated opinion of one’s own judgment, people can and do simply deny the accusation.

So how can we cope with self-deception?

Cowen and Hanson argue that we should be on the lookout for people who are “meta-rational,” honest truth-seekers who choose opinions as if they understand the problem of disagreement and self-deception. According to the theory of disagreement, meta-rational people will not have disagreements among themselves caused by faith in their own superior knowledge or reasoning ability. The fact that disagreement remains widespread suggests that most people are not meta-rational, or—what seems less likely—that meta-rational people cannot distinguish one another.

We can try to identify meta-rational people through their cognitive and conversational styles. Someone who is really seeking the truth should be eager to collect new information through listening rather than speaking, construe opposing perspectives in their most favorable light, and offer information of which the other parties are not aware, instead of simply repeating arguments the other side has already heard.

What sells in the name of journalism today is written to appease one side or the other. Opinions are important, but what is more important is facts that help you arrive at it and the process you use to form them. A good journalist should be a meta-rational person. He should also be able to find other meta-rational people, so that the world is made aware of facts and arguments that help take things a step forward.

The problem, of course, is that meta-rational people care less about success but more about seeking the truth/understanding the world. Their voices are present but harder to find. They get far less attention than they deserve.

You can stop reading now, if you like. For, because we are on the subject of good journalism and meta-rational people, I’d like to discuss an example of exactly the opposite. Aleks Eror writes in VICE that the Chinese are engineering genius babies. This is a great story, if it were true. Sadly, he bases this claim on the word of Geoffrey Miller, an American academic, who wrote a heavily biased article about it in January. What’s worse is that it Miller’s argument had been criticised and debunked (sort-of) immediately after. But Eror chose to ignore that and rehash the fear-mongering story.

The central point of the argument is that the Chinese have an institute where they are doing genome sequencing of the world’s smartest people to find the genes that makes us smart. This is not a new story. Eugenics in some shape or form has existed for a long time. The problem with Eror’s piece is that he doesn’t run by an expert the claims made by Miller.

A good science journalist should always do that because the complex nature of the subjects we usually deal with. Experts may not be the perfect meta-rational candidate that we’d like, but they are close enough. They bring an external perspective, and I’ve found that invaluable in my own work.

A quick, interesting read for those who like reading websites like Quora (and later having arguments in a pub). This is a book about setting the facts right. David Bradley attempts to use science and rational thinking to clear some age-old myths (like “cats are smarter than dogs“) and some modern ones (like “everyone is connected to the other through six connections“).

This “book” should really exist as well-referenced posts on a single website. Some of it exists on sciencebase.com, Bradley’s blog. There are others like snopes.com too. Perhaps what Bradley has done with the book is refine his explanations.

I somewhat agree with what Brian Clegg has to say about it: If you liken popular science books to food, Deceived Wisdom is simply not meaty enough to make it a three course meal. It is, however, a top notch box of chocolates – and who doesn’t like that?