The Other "Silicon" Places

Somebody recently pointed me to an MIT Technology Review article written by Vivek Wadwha asserting that "Silicon Valley can't be copied".

I have been thinking about this topic a lot lately, because I frequently wonder if I really am getting enough added value from living in Silicon Valley to justify the cost, and if I'm honest, I no longer can say that I am.

I too used to believe Silicon Valley couldn't be copied, and have repeated the same assertions about critical mass many times before. However, I have become increasingly disillusioned with SV, both with the shortsightedness and bureaucracy within "big tech", and (maybe I'm getting old, but) also with how millenials are running the startup scene.

"Silicon Valley" has moved further north over time, from the actual valley of San Jose / Santa Clara, to Palo Alto / Mountain View, to SF itself. Now even if a company isn't based in SF, almost all of the single employees live in SF and commute out to work. (Almost all the single Googlers I know take the bus down from SF to Mountain View, which adds 3 hours of commute to their day.) It's an oxymoron to call San Francisco "Silicon Valley", but that's where all the young startups are all setting up shop these days, because kids want to live where the other kids live.

The Great Compensator is of course the cost of housing, and this may ultimately break the SV monopoly. The cost of housing in SF has gone from laughable to well past the pain threshold. And once you're no longer 22, and you're no longer willing to live in a closet or survive off of ramen, you might start to care about lifestyle. (Apologies for the BuzzFeed link.)

A lot of Wannabe Silicon Valleys, with much better quality of living than SF at lower cost, seem to be finally gaining traction. You need enough people acutely interested in making an area a tech hub for a long enough period of time, and then you need some sort of development incentives and/or re-branding to kick off what will hopefully turn into a runaway process of tech creativity and investment confidence. The clever re-branding of NYC as "Silicon Alley" and Wellington as "Silicon Welly" seems to have had an effect on the burgeoning of tech in those areas. A lot of new startups reportedly moved to Kansas City and Provo when Google Fiber was announced in those cities. etc.

One of the many new tech hubs is "Silicon Slopes" (centered around Lehi UT), which has an unusually high number of CS and EE grads, and a large amount of new investment money. The mortgage payment on a brand new 5 bedroom house with a large yard in UT is less than my rent payment for a studio apartment in a garage attic in Palo Alto. Gas, groceries and restaurant meals are just over half the price in UT compared to CA, and some of the best skiing, hiking and rock climbing in the world falls within 30-60 minute radius of any urban center in UT.

Ultimately I think a lot of people in these wannabe tech hubs are quietly ignoring the assertion that you can't duplicate SV, and just getting on with it. In some cases, maybe they're building something different and better.

Boston (the city that I called home for seven years) still has a lot of unrealized potential, especially in biotech -- and Boston/Cambridge has a lot more soul than SF. But Boston will always face an uphill battle due to weather and old, cramped housing, which gives rise to the transience of the non-local "population overlay" (people that are just there for school or jobs).

But in the end, maybe location doesn't matter at all: there's always the "digital nomad" lifestyle, where you can actually save money by traveling the world while working from your laptop.


What Artificial Intelligence is not

An lot of people seem to think that the recent successes in deep learning indicate we're closer to building intelligence. However, just because we can now build algorithms that solve more complex, less-well-structured problems doesn't mean that today's algorithms are necessarily more intelligent than older machine learning algorithms.

What is required to build an intelligent system? We can in fact build highly functional learning algorithms -- algorithms that improve given more instances of what we want them to recognize. But machine learning is simply the training of arbitrary function approximators, where "function" is defined in the mathematical sense as mapping from a domain (a set of inputs) to a range (a set of outputs corresponding to each input). Functions are mathematical objects; they are not intelligent. Pattern matching, which is the creation of a function that maps regions of some pattern space to a corresponding output, is therefore not the creation of intelligence: "dumb" algorithms can match complex patterns. Inference, by itself, is not intelligence either -- is a Bayesian equation "intelligent", or is the construction of an algorithm that computes a Bayesian equation intelligent, even if the equation itself is not? Solving logical constraints, is also, by itself, not intelligence -- is Prolog sentient? Intelligent beings are constantly modeling and predicting the world around them, comparing observed to expected, but prediction is not, by itself intelligence: recurrent neural nets are just neural nets that can hold internal state while they match a time series (i.e. the domain is the time series, the range is a variable of interest).

Robust machine learning systems are also able to generalize, and thereby absorb common sources of noise and modes of variation. Jean-Loup Gailly (creator of gzip and a chess expert) made an observation along these lines in a comment on Slashdot many years ago: that the brain hierarchically compresses knowledge into higher and higher levels of abstraction, breaking old abstractions and forming new higher-level abstractions when new knowledge is introduced, in order to achieve a higher compression ratio after incorporating the new knowledge. A beginner chess player exhaustively searches through the neighborhood of all moves around the current board position, but an expert player rarely does this -- they are thinking at a much higher level, about patterns and strategy. In effect, they have compressed a large amount of information into a relatively small number of higher-order concepts or generalizations.

All of these capabilities -- learning, pattern matching, inference, constraint satisfaction, prediction and generalization -- seem to be present in intelligent organisms, so it would be reasonable to assume that they are necessary capabilities for a system to be intelligent. However, the reverse is not necessarily true: we have no evidence that any of these capabilities, or the collection of all of them together, is sufficient for a system to become intelligent. It's a cop-out to say that maybe intelligence will spontaneously emerge if we build a big enough, complex enough system that has all of these elements, i.e. that once we have a computer that is as fast as the human brain, that it will function like the human brain. We can't conclude something about emergent phenomena without understanding how and why the emergent behavior or property arises. And just because we don't understand an opaque machine learning classifier, in spite of it doing something powerful and useful to humans, doesn't necessarily mean that the classifier is intelligent.

The elephant in the room of AI futurism is that we can't predict when we'll be able to build intelligence, because we don't know what we're building. To quote the Dread Pirate Roberts, "Anyone who tells you differently is trying to sell you something." In fact, I'd love to hear a good definition of what it means to "understand" something. I wrestled with this for a couple of years while building NLP and AI systems at Google. I could never quite come up with a good enough definition to code towards, and I haven't seen a good definition anywhere else either. Dictionary definitions like "to perceive the intended meaning of" start tying you in knots: "perceive" is circular when used in a definition for "understand"; "intended" gets into issues of Theory of Mind (which are probably inescapable anyway when discussing language, since language is about using terms that create a semantic model in somebody else's mind based on the one you have in your own). But most importantly, now you have two problems, because you have to define the word "meaning" too.

(Speaking of semantic models, check out these incredible demos by the Gallant Lab where they map the regions of the cortex that are firing while subjects are watching a movie, and correlate those regions to the WordNet English word topology.)

Another issue is how we'll know once we've built intelligence. The Turing test, unfortunately, is not going to provide the answer. If we eventually create algorithms that do better than expert humans, they'll fail the Turing test, because they'll do noticeably better than a human would at the same task. Therefore, to pass the Turing test, a superhuman intelligence would need to behave with Artificial Stupidity. This shows that the Turing test is not a test of the intelligence of the tested agent at all; it is a test of the deceivability of the observer.

Ultimately, machine learning is progressing in huge leaps and bounds, but we have made little to no progress with AI, and I don't think we will make any further progress until we understand what intelligence really is.

Update: From a comment by Melissa Basileo: "It is hard for us (perhaps impossible) to think outside of our own species' context, which is scary when I think about it as it pertains to AI. An AI would have no contextual basis for emotion, would not have the same internal motivators that we have (food, shelter, sex), which means it is very likely that if an AI ever was created, we might not be able to recognize it because it is so out of context with our own species - or biological life as we know it even."


Interstellar is full of plot black holes

Saw #Interstellar last night. Don't go see it on a domed IMAX screen, despite the recommendations saying it was made for IMAX: everything is warped by the screen and you'll miss half the action because you can't crane your neck around to see opposite sides of the screen fast enough.

[spoiler warning] In spite of all the efforts to stay true to relativistic physics, the movie had some major gaping "black holes" in its plot, due to major violations of the laws of physics. I love sci-fi and try to suspend disbelief when I watch sci-fi movies, but when a movie's big claim to originality is that they "got the physics right" by employing a world-famous physicist to write equations that they then fed to their rendering team, the bar is raised!

Some things the movie got wrong, in no particular order:
  • A wormhole was supposed to be created by some alien intelligence "near Saturn". So was the wormhole orbiting the sun in the same orbit as Saturn? Or was it orbiting Saturn? Either option would require the wormhole to have mass.
  • If you descended deeply enough into the gravity well of a black hole (such that one hour for you was equivalent to seven years for an external observer), there is absolutely no way the combustion-based thrusters on a small ship would have any chance of ever lifting you out of the gravity well (oblig. xkcd). Look at the massive 3-stage rocket that was needed to launch that same lander craft out of earth's gravity well at the beginning of the movie. And yet somehow the lander was able to get back up to the mothership that was orbiting the black hole just that much further out from the orbit of the planet that time was not as severely distorted. Note that it takes 60 times more fuel to get from 100km of altitude (the edges of space) to 250km of altitude (the orbit of the International Space Station) -- and in the movie, we're talking about a gravity well many zillion times deeper than even the one depicted for Saturn in that xkcd comic.
  • Speaking of which, the mothership would have a much slower orbit around the black hole than the planet that they went in to land on, due to Kepler's second and third laws (dramatically exacerbated by the strong gravity around a black hole). So by the time they "popped back out" of the planet's orbit to return to the mothership, they would be far, far away from where the mothership's orbit would be at that point. (The movie "Gravity" made this same mistake multiple times in the name of "good Hollywood action".)
  • If the main character really did end up getting pulled into a black hole, and made it out alive, he would not meet his own daughter, even as an old woman -- she would be long, long gone by the time he somehow got back out again due to this whole time dilation thing, which they already experienced moving just barely inside the "steep part" of the time dilation curve. Falling "all the way into the black hole" would cause an even bigger time dilation than what they experienced on the first planet they visited.
  • You can't fall "all the way into a black hole" anyway, it would take an infinite amount of time to reach the event horizon from the point of view of the guy falling in, and an infinite number of infinite amounts of time from the point of view of an external observer.
  • If you were extremely patient and the life support in your space suit somehow managed to support you until you did reach the event horizon of a black hole, and tidal forces didn't already tear you apart, you would probably be torn into subatomic particles as (from the point of view of an external observer) you entered the photon sphere, which is the radius at which light orbits the black hole. (Technically the photon sphere is 50% further out than the event horizon in some frame of reference.) This is a giant supercollider that continually collects light and matter and sucks it into an orbital "firewall" around the black hole, and then eventually spews jets of energy outwards from the two orbital poles of the black hole at energies so unimaginable that structures beyond galactic scale are created.
  • Even if the black hole you fell into somehow conveniently delivered you back to the earth-side of an unconnected wormhole, way out by Saturn, there's approximately zero chance that a spaceship would be able to randomly find you floating out in the vastness of space if they weren't looking for you. (The movie "Gravity" also made this faux pas numerous times: "Oh look, there's the Chinese space station off in the distance!") Or, as Douglas Adams put it, "Space is big. You just won't believe how vastly, hugely, mind- bogglingly big it is. I mean, you may think it's a long way down the road to the chemist's, but that's just peanuts to space."
  • Even in a black hole, time travel is not possible in any direction other than forwards (by means of time dilation) -- and gravity waves don't violate the arrow of time, so they can't be used to send information back in time.
  • And love, although powerful, probably isn't a powerful enough force to make time travel possible either. Sorry, Anne Hathaway.


Understanding the "full stack" of life

Some non-programmer friends were talking among themselves about computer programming a couple of days ago, and they were going back and forth about how they didn't understand programming but it must be complicated. One of them said, "Yeah, I just thought it was just kind of part of the design that made apps on your phone work the way they do."

For the first time I caught a vision of what technology would look like without a background in computer science or software engineering, and, strangely enough, it totally blew my mind. I had never really thought about what it would be like to have a technology integrated into every part of my life, accepting it on face value without seeking to understand it down to its fundamental axioms -- the algorithms, the logic circuits, the bits, the transistors. I know that technology is not something that gets everybody excited, but I don't understand how people can operate only in the realm of high-level abstraction. I cannot function that way, I have an immense drive to understand everything around me -- everything I see, use, interact with and experience.

I then realized though that dealing only in high-level abstractions is exactly what humans have done for millennia with biology: treating human beings and other organisms as high-level, abstract entities, without understanding (or without in most cases seeking to understand) their axioms and composition. We did lack the observational tools until recently to introspect our biology at microscopic and molecular levels, but throughout history, as is true today, most of society has not even concerned itself with seeking deeper understanding. There is immense value in dealing in high-level abstractions, but to truly understand a system, you really have to understand "the full stack".

Now we're finally starting to understand the axioms of biology, and what we are seeing and understanding is incredible. A friend sent me photos today of his single-cell fertilized IVF embryos. A fertilized egg is the most amazing thing in the world: it is the very first domino, set in motion, in the most amazing and intricate self-stabilizing, order-creating chemical reaction known in the universe. Think of all the quintillions of steps and interdependencies in this reaction that end up forming an adult human... all packaged up into only about 10B bits of information, within a biological information processing machine. And even more amazing than the biology itself is our ability to introspect and understand our own very composition. Think about how incredible that is to be a being that is not only intelligent and supremely intricate and complex, but a being that is able to understand its own structure and functioning -- eventually -- down to the axiomatic level. If that is not the very definition of transcendent potential and even nascent divinity, I do not know what is.

I ran into the XVIVO founder David Bolinsky for the second or third time a few days ago, and was reminded of the amazing video Inner Life of the Cell that he produced a few years ago:


The Turing test and Artificial Stupidity

Every news outlet is currently covering the story that a chatbot pretending to be a 13-year old Ukranian boy has deceived 33% of human judges into thinking it is a human, thereby "passing the Turing test for the first time".

There are so many problems with the Turing test (even with the numerous refinements to it that many have proposed) that I don't know if it will ever tell us anything useful. The creators of the above chatbot hinted that part of their success in convincing the judges was that “his age ... makes it perfectly reasonable that he doesn’t know everything” -- in other words, to make a believable bot, you can't give your bot super-human knowledge or capabilities, even if this is technically possible to do (e.g. computers can multiply large numbers almost instantly). Limiting computational power to appear human-like is known as "artificial stupidity". The need for artificial stupidity to pass the Turing test illustrates one of the deepest issues with the test, and one that cannot be fixed by simply tweaking the rules: the Turing test is a test of human dupe-ability, not of machine intelligence.

I'm pretty sure we'll start seeing several claims per year that a bot has "passed the Turing test", followed by a flurry of discussion about what was actually tested and whether the result is believable or even meaningful, until it becomes so cliche'd to say that your bot passed the Turing test that nobody with a halfway decent AI would actually *want* to claim that their AI passed a test of this form.

Hopefully we see the day when the Turing test is inverted, and we realize we need a test to establish that someone is a "genuine human" and not a bot ;-)  But until then, we still have a heck of a lot of work to do!


How to synchronize collaborative music performances with Google+ Hangouts

Someone just asked me via email how she can synchronize her singing over a Google+ Hangout with a musician on the other end, when it seemed like there was a time delay that was tripping them up. She wanted to know how to eliminate the time delay, or if Google was planning to eliminate it at some point.  Here is my reply, posted here in case it helps somebody else.


Great question! It was my mother's birthday last week, and the family got together in four different venues over Google Hangout to sing "When I'm 64" to her for her birthday :-)  I called it before we even started the hangout: the latency (delay) would cause us to all keep slowing down to let each other catch up, then realize that everybody was getting further behind, so we all needed to speed up and skip ahead, and then we would slow down again, etc.  Sure enough, every few seconds we seemed to have singing synchronization issues. It made the whole thing a lot funnier, but it wouldn't work for your situation at all!

In the general case, this is not solvable for the same reasons that Einstein said that all simultaneity is relative: when it takes a non-zero amount of time to send information from point A to point B, and back again to point A, it's impossible for both point A and point B to agree on a global concept of "now". You simply cannot reduce the latency to zero for network connections, and much less for running a complex streamed application like a Hangout over the network, and the further apart you are in the world, the greater the expected latency.

The way that this has been solved in the past (e.g. by that massive virtual orchestra / virtual choir project that has been run over YouTube a couple of times before) was to pre-record the music, and have each singer play the sound in their headphones while singing / playing. Then they each separately recorded their videos and sent them to someone who mixed them down into a single track, offline, after they had all finished recording their separate tracks.  i.e. they simply avoided the problem entirely by not performing simultaneously :-)

If I were you, I would simply experiment with performing simultaneously, maybe you can practice having one of you (the one on the recording end) singing exactly on time with the other person, and the other person playing/singing at exactly 2x the one-way delay time ahead of the other person. The trick would be to have the performer that is playing ahead (not on the recording end) set the tempo and basically pay no attention to the person on the recording end (i.e. don't try to slow down to let them catch up). As long as the person on the recording end is on-time and keeps up with the one that is leading the piece, nobody will know about the synchronization issues.

If you're not recording locally, but rather broadcasting the Hangout live, you both need to split the delay equally, so that each of you sings/plays at exactly 1x the one-way time delay ahead of the other person (or ahead of what you hear coming out of your speakers). Actually, you probably need to play 0.5x the latency ahead of what you hear coming out of your speakers, because each connection is routed through Google's servers and then back out to the other person, and it's from Google's servers that the two different video signals are mixed and then broadcast out to the rest of the world.

I hope this makes sense. There's really no way around this for live hangouts though! (But you might be able to make it work for recordings.)


More on leaving academia

The problem with being a grad student is that the burden of an unfinished thesis or dissertation, or paper deadlines, or assignment deadlines, never goes away. You never feel quite free to walk away from it on nights and weekends. It reminds me of the poem "Something Left Undone":

Labor with what zeal we will, 
Something still remains undone, 
Something uncompleted still 
Waits the rising of the sun. 

By the bedside, on the stair, 
At the threshhold, near the gates, 
With its menace or its prayer, 
Like a mendicant it waits; 

Waits, and will not go away; 
Waits, and will not be gainsaid; 
By the cares of yesterday 
Each to-day is heavier made; 

Till at length the burden seems 
Greater than our strength can bear, 
Heavy as the weight of dreams 
Pressing on us everywhere. 

And we stand from day to day, 
Like the dwarfs of times gone by, 
Who, as Northern legends say, 
On their shoulders held the sky. 

--Henry Wadsworth Longfellow

Your career trajectory in academia depends upon building your reputation (through your publication record, service rendered to the Ivory Tower, and awards received). It's a game some people can play well, but it's a hard game to keep playing year after year until your previous successes carry you on to your next successes with very little personal effort beyond showing up to as many meetings and speaking engagements as you can on your completely overbooked calendar while trying to stay sane.

Ultimately I got out of academia because I realized that I had just spent seven years just on my PhD and postdoc, which was 8% of my expected lifespan or something, and the work didn't feel like it justified 8% of my life. Life is precious.

However, academia is in my blood, in my DNA. I'll be back, I'm sure.


How to change timezone settings on Gmail, Google Calendar etc.

Every time I go back and forth between the East and West coast of the US, I have to google how to change timezones on Google products, because (1) the settings are hard to find, and (2) you have to do no fewer than SIX different things to avoid all sorts of weird timezone-related glitches:
  1. Update the timezone setting for your computer's system clock -- this is important because SOME but not all Google products use the system timezone (e.g. Gmail reads the system time zone, as do some but not all features in Google Calendar). The option to change the timezone can be found by right-clicking on the clock in the system tray and choosing a Preferences or Time/Date Settings option or similar. (In Fedora Linux, you can also type "system-config-date" as the superuser in a console.) Make sure "System clock uses UTC" is checked so that Daylight Savings time is handled correctly.
  2. Make sure you change the time zone on ALL devices that are logged into your Google account (phones, tablets, laptops, desktops etc.), or you'll have problems. Every time a device refreshes the Gmail webview, for example, it changes the displayed Gmail time zone for all connected clients -- so if you forget you're logged into your Gmail account on a second laptop, and that laptop's timezone settings are wrong, you'll have ongoing issues with multiple devices fighting to change the timezone back and forth between two settings.
  3. Restart Chrome (or whichever browser you use) -- Chrome doesn't always pick up the timezone change of your system clock until it has restarted, even if you refresh or log out of your Google accounts and log back in with the browser still open -- i.e. Gmail may continue to display message timestamps in the old timezone until the browser is restarted, regardless of the timezone setting on your Google account.
  4. In Google Calendar, you have to manually change the display timezone. Go to the gear menu near top right, choose Settings, then under the default tab (General), change the setting "Your current time zone".
  5. If you use Google Voice, it has its own timezone setting too: Go to the gear menu at top right, choose Settings, click the Account tab, and change the Time Zone setting there.
  6. Google Drive now has its own timezone setting too, although it is unset by default (which, I assume, means it uses the system timezone? Or maybe the Google account timezone, described below?): Go to the gear menu at top right, choose Settings, and in the General tab you'll see Time Zone. (I guess if it's unset, leave it unset, hopefully it will use one of the other correctly-set time zones.)
  7. (The really hard one to find): Go to https://security.google.com/settings/security/contactinfo , and under "Other emails", click Edit, which will probably take you to the more obscure-looking URL https://accounts.google.com/b/0/EditUserInfo?hl=en . (You can go directly to that second link, but it looks like the sort of URL that won't be around forever, so I provided the navigation directions.) Here you'll find a timezone setting that, if wrong, can mess with timezone display in strange places in Google products (e.g. the red line showing current time on Google calendar may appear at the wrong time, or events are shown at the wrong time, or popup reminders pop up at the wrong time, but not all of these -- I forget exactly what weird effects were caused by this problem exactly). This setting says it's optional, so you can probably select the blank line from the dropdown list, effectively unsetting an account-specific timezone. I'm hoping then that one of the two timezone settings above will be used instead, but I need to test this. 
(Probably Google apps should all read the browser's timezone, and the browser should watch for updates of the system clock and timezone. Hopefully this gets fixed someday...)

I hope this saves somebody some frustration!


On the nouveaux érudits and anti-theism

Since the rise of a few prominent and vocal atheists about five years ago, public anti-theist sentiment has increased dramatically. The nouveaux érudits are successfully introducing into the public subconscious the idea that theism as a liability. These days, everybody I know that believes in God in some way goes about it relatively quietly. The public scoffing at religion is unfortunate, because removing religion from public discourse robs humanity of an important aspect of human cultural identity and diversity. Just like in the natural world, diversity is at the root of the beauty of humanity.

Atheism is as much a belief system as any religion I can think of. Both theism and atheism is a belief and a choice. Unfortunately neither theism nor atheism constitutes a falsifiable hypothesis based on tangible, observable evidence, so we should all simply agree to coexist happily while respecting others' beliefs, since arguing, belittling, attacking etc. doesn't actually do anybody any good. I don't understand how much effort some people put into tearing down others' beliefs.

That applies to both theists and atheists, incidentally -- there is no justification for, or usefulness in, fighting with a person or trying to coerce them to believe the way you do in the name of either "saving their soul" or "protecting them from religion". Share what you're excited about by all means, if somebody wants to listen, but respect people's agency, and once they have heard what you claim as fact, let them figure out for themselves whether or not what you are saying is true. And you can teach people what you believe without tearing down what they believe. Or as the Qu'ran rather ironically says, "Let there be no compulsion in religion -- truth stands out clear from error."

Back to my original point though -- it is sad that the religious people I know generally don't feel that they can present this as part of their public identity anymore.

(In response to Nils Hitze's post on Google+ wondering why most people he follows are atheist. The above is also posted to my G+ feed, so there may be comments there too.)


Results before Methods??

Why do so many journals require you to put results before methods? That's so backwards, and it is maddening to have to write a paper to conform to. You can't explain what you found before you explain how you found it. This also betrays the mindset of many biologists that I know at least (since this pattern is ubiquitous in biology and biomedical journals), that you can safely ignore the contextual assumptions embedded in the methods when you interpret the results.


On leaving academia, and wanting to create "Google X" without the Google part

[ tl;dr version: I'm leaving academia after many years; have big life decisions to make; need a dose of perspective. What's a good next step? What's the likelihood of success of creating some sort of standalone Google X type lab with a few brilliant people and getting it funded in the current climate? ]

So I'm finally biting the bullet and leaving academia, after a nasty realization that the powers that be (the hand that was supposed to feed me, NSF, cough) -- in conversations right to the top of the hierarchy -- have insufficient technical understanding to tell sound ideas from the rest, and insufficient foresight to take a risk on funding potentially revolutionary ideas when there's an evolutionary idea from a good friend of theirs also submitted in the same round of proposals that lets them check off their keyword boxes.

As a result, I have started gingerly digging through some job listings on both coasts. When you've been in academia as long as I have (and had it drummed into you that "you will never amount to anything in this world, or accomplish anything much, if you leave academia -- and you'll never come back if you leave"), it's hard to look forward to being a code monkey. I have, however, worked as a software engineer several times, for several different companies, so it's not like I have never had a "real job", but I figured I should ask for a dose of perspective. Tell me that academia has been lying to me all these years :-)

Really, I know that the perception that academia is the be-all and end-all of innovation and world-saving is for the most part false or at least myopic, and nobody outside academia really sees it that way, even though some cool research comes out of all the major institutions each year. Academia is certainly not the be-all and end-all of wealth creation. Peter Diamandis (whom I know from Singularity University) once expressed complete disbelief that I would even be considering a career in academia -- "You'll amount to 1/10th of your potential if you stay in academia."). Deep down I know he's right, even if I still have a strong urge to at least keep my foot in the door.

Anyway, I also have a strong entrepreneurial streak (I have a doc I have been compiling over the last few years, that now consists of hundreds of different ideas, some of which might even succeed to some degree if the execution is good) but I also have a strong dislike for business operations (although I have been involved in a small startup before). There seems to be a lot of funding out there right now, and a resurgence of new ideas, as well as more risk-taking than the industry has seen for a decade or more. I think this might be the right time to try to jump into entrepreneurship rather than working for the man.

So, I guess the questions I have for anybody that stumbles across this post are:

(1) Does anybody have experience with leaving academia and getting back into it after having accomplished something useful in industry? Is it even worth trying to keep that option open if my greatest interests are all heavily research-oriented (and when I would go crazy with mundane coding), or are there all-round better alternatives? Does anybody have experience working somewhere like PARC or one of the Intel research labs?

(2) How hard would it be to create a new research lab with a few top-notch guys, as a standalone "skunkworks" type lab but based outside of any organization or company with deep pockets, and get it funded today? (i.e. something like the Google X lab, but run outside of Google -- this would obviously be a big investment risk if the focus is even partly on blue-skies projects.)  Is working with the type of employee that is frequently attracted to that sort of working environment (i.e. top 1% of engineers, theorists, inventive types etc. with all the crazy ideas and the audacity to think that they might be able to make them work) like herding cats? Has anybody worked at Google X or in a similar moonshot research lab?

I guess I'm just trying to figure out where I can make the biggest difference. Or maybe even thinking that way is a delusional after-effect of standing on the Ivory Tower :-)

PS in case anybody is hiring...  http://resume.lukehutch.com/


The Multicore Dilemma (in the big data era) is worse than you think

The arrival of the big data era almost exactly coincided with a plateau in per-core CPU speeds in 2004 and the beginning of the multicore era, and yet, the National Research Council found as recently as 2010, "There is no known alternative for sustaining growth in computing performance; however, no compelling programming paradigms for general parallel systems have yet emerged." [Fuller & Millett (Eds.), The Future of Computing Performance: Game Over or Next Level? Committee on Sustaining Growth in Computing Performance, National Research Council, The National Academies Press, 2011]. Additionally, as every field of scientific research and every major business endeavor begins to generate more data than can be easily analyzed on a single CPU, it has become evident that this "multicore dilemma" is impeding progress across all of science, and across all fields of business. The multicore dilemma, therefore, presents the biggest near-term challenge to progress.
Processor performance from 1986 to 2008 as measured by the benchmark suite SPECint2000 and consensus targets from the International Technology Roadmap for Semiconductors for 2009 to 2020. The vertical scale is logarithmic, where a straight line trend indicates exponential growth. A sharp change in the growth rate can be seen around 2004. Before 2004, processor performance was growing by a factor of about 100 per decade; since 2004, processor performance (in GHz) has been growing and is forecasted to grow by a factor of only about 2 per decade. Source: [Fuller & Millett 2011]

The coining of the term "big data" is a reflection of the fact that the exponential increase in data generated per year, which has been going on for many decades, finally crossed some scale where humans sat up and started paying attention. This phenomenon occurred across a wide range of fields over a short period of time, including at the leading edge of most fields of scientific research, where many scientists (who have no training in parallel and distributed systems) are now having to spend a significant proportion of their time trying to figure out how to parallelize their analysis code. The difficulty of parallelizing code is impeding progress in all major fields of science, and the seriousness of this problem cannot be overstated.

DNA Sequencing costs: Data from the NHGRI Large-Scale Genome Sequencing Program

Ultimately, no matter what kinds of parallel computing paradigms may be created, if they require conscious engineering to use, they will incur cognitive overhead for programmers, and significant programmer knowledge and expertise, which is not possessed by the large number of non-skilled programmers (e.g. medical researchers) who must write parallel code just to get their job done. Long-term, the only sustainable approach to parallelization is the one that requires close to zero programmer effort: “Write once, parallelize anywhere”.

Figuring out what to parallelize, and how to parallelize it, is not a problem we should be assigning to humans to solve. The lead authors of the NRC report further observe that “The intellectual keystone of this endeavor is rethinking programming models so that programmers can express application parallelism naturally. This will let parallel software be developed for diverse systems rather than specific configurations, and let system software deal with balancing computation and minimizing communication among multiple computational units.” [Fuller & Millett 2011] Without finding a better model for automatic parallelization, some great challenges lie ahead in the big data era.

The multicore dilemma meets big data: incidental casualties

There are several important ramifications of the confluence of "the end of Moore's Law" (as it is traditionally interpreted) and the beginning of the big data era:

  1. We are facing an impending software train-wreck. Concurrency introduces whole new classes of potential bugs, and the human brain is notoriously bad at writing safe parallel code. As per-core speeds plateau, we are not just headed towards a plateau in software speed, but an era of significantly buggier software.
  2. We are facing an impending software morass. Parallel code, using current parallelization paradigms, is not just harder to write, it is more complex, and therefore harder to understand. It is very easy to get mired in a spaghetti of parallel code. All current platforms, libraries and techniques for parallel processing introduce significant issues:
  1. Parallel programming requires additional programmer effort. The data dependencies inherent in a problem must be determined manually by a programmer in order to design a parallelization strategy, and the implementation must be shoehorned into some parallelization paradigm that is appropriate for a specific target architecture.
  2. Parallel programming makes debugging, maintaining and extending code significantly more difficult. Parallel programming produces boilerplate code, increases the likelihood of introducing hard-to-trace bugs, and increases the complexity of program structure.
  3. Parallel programming can result in longer development times. For large-scale projects, reining in the added complexity of parallelization can be intractable, either leading to the failure of the project or the abandoning of parallelization.
  4. Parallel programming can incur significant extra cost. Programmers who have a solid grasp of parallel programming are rare and expensive.
  1. We urgently need automatic parallelization for “mere mortal” programmers. We urgently need a programming language for the masses that does not require a background in concurrency or parallel and distributed systems. Increasingly, programmers of all skill levels, and from all fields, are being required to parallelize their code. Many programmers with little or no formal CS background (e.g. biomedical researchers) are writing code to target parallel architectures, for example using a series of duct-taped Bash scripts to launch R and MatLab jobs in a work queue on an NIH-funded compute cluster. It will soon no longer be possible to do biology without doing computational biology, or physics without doing computational physics, and dataset sizes in many fields of science long ago outstripped what may be quickly computed on one processor. The NRC report gave the following advice: "Recommendation: Invest in research and development of programming methods that will enable efficient use of parallel systems not only by parallel systems experts but also by typical programmers" [Fuller & Millett 2011]
  1. After the big data era will be “the messy data era”. Data types are becoming increasingly diverse, increasingly varied in numbers and types of attributes, increasingly sparse, increasingly cross-linked with other disparate datasets, and increasingly social in nature. We need a smarter compiler that will automatically parallelize our code and then get out of the way, relieving us of the cognitive burden of figuring out how to parallelize the code, and allowing us to focus on the more important issues of the specific problem we are trying to gain insight into.

The solution to the multicore dilemma: “the sufficiently smart compiler”

It is clear that the multicore dilemma is one of the most pressing issues directly or indirectly impacting all of scientific research today, and that a robust solution must be found as quickly as possible. Paul Graham recently wrote:

"Bring Back Moore's Law: The last 10 years have reminded us what Moore's Law actually says. Till about 2002 you could safely misinterpret it as promising that clock speeds would double every 18 months. Actually what it says is that circuit densities will double every 18 months. It used to seem pedantic to point that out. Not any more. Intel can no longer give us faster CPUs, just more of them.

"This Moore's Law is not as good as the old one. Moore's Law used to mean that if your software was slow, all you had to do was wait, and the inexorable progress of hardware would solve your problems. Now if your software is slow you have to rewrite it to do more things in parallel, which is a lot more work than waiting.

"It would be great if a startup could give us something of the old Moore's Law back, by writing software that could make a large number of CPUs look to the developer like one very fast CPU. There are several ways to approach this problem. The most ambitious is to try to do it automatically: to write a compiler that will parallelize our code for us. There's a name for this compiler, the sufficiently smart compiler, and it is a byword for impossibility. But is it really impossible? Is there no configuration of the bits in memory of a present day computer that is this compiler? If you really think so, you should try to prove it, because that would be an interesting result. And if it's not impossible but simply very hard, it might be worth trying to write it. The expected value would be high even if the chance of succeeding was low."

[Paul Graham, Frighteningly Ambitious Startup Ideas #6, March 2012]

It has furthermore been observed that “exploiting large-scale parallel hardware will be essential for improving an application’s performance or its capabilities in terms of execution speed and power consumption. The challenge for compiler research is how to enable the exploitation of the [processing] power of the target machine, including its parallelism, without undue programmer effort.” [Mary Hall, David Padua, and Keshav Pingali, 2009, Compiler research: The next 50 years, Communications of the AC 52(2): 60-67.]

The “sufficiently smart compiler” cannot actually exist (for modern imperative programming languages)

Functional programming languages are implicitly parallel (automatically parallelizable), because calls to pure functions have no side effects. However, “mere mortal” programmers are not able to be productive in functional programming languages, or are unwilling to use them.

Many efforts to automatically parallelize imperative programming languages have been attempted with varying success, but with imperative languages, the compiler must be overly cautious in applying parallelization strategies, because it is not just difficult but actually uncomputable in the general case to safely parallelize imperative code, because the data dependency graph of an imperative program (the graph of which values are used to compute which other values) can’t be known until you actually run the code. Static analyzers can often only guess at the origin or identity of a specific value without actually running the code. The sufficiently smart compiler cannot actually exist for modern imperative programming languages.

We need a new programming model that allows programmers to work in the maximal implicitly parallel subset of the imperative paradigm while allowing the compiler to safely automatically parallelize the code.


Darwin was (half) wrong

Slashdot ran a story, "The science of handedness". I'm pretty sick of reading this sort of thoughtlessness describing evolutionary biases. If you're going to say that an adaptation gives a reproductive or predatory advantage, then fine, you're talking Darwinian evolution -- survival of the fittest -- and that's pretty trivially easy to show, even in a lab. If you're going to say, "Everybody in chummy societies had the same handedness so they could share tools", then please tell me how that weak-sauce tiny (or effectively zero-magnitude) biological fitness bias is supposed to have produced a genotypic change to an entire species within the known anthropological lifetime of the species. Remember that Darwinism requires that for your random trait variation to survive and thrive, at a minimum you have to pass your genes on while somebody else does not. So, I have to wonder if the authors "believe in" Darwinism or if they do not.

This gets at what I think is a much bigger issue: fundamental to Darwinism is not just survival of the fittest, but also randomness. I think that true biological evolution -- what's actually happening in the real world -- is not Darwinism, because it is very non-random. It is inconceivable that the complexities of the human organism, or any other for that matter, could have occurred by chance via a random walk through the state-space of possible genetic mutations (many of which could easily give rise to non-viability) in the number of generations since the major forks in the tree of life. There just isn't enough time, enough generations. There isn't sufficient evidence of non-viability, through miscarriage etc., for the worst mutations to die out -- and there isn't sufficient evidence that most traits that are said to evolve through "survival of the fittest" actually gave the possessor of that attribute an actual survival advantage, a reproductive advantage or an advantage as a predator, at the expense of those that did not possess that attribute.

What's the alternative? Even setting religious issues completely aside, personally I think that built into every feed-forward mechanism in biology, crossing back across vast numbers of levels of emergent complexity, are corresponding feed-BACK mechanisms (actually, back-propagation mechanisms, to use the machine learning terminology) such that a system's biology -- and even its genome -- can "learn" in response to environmental stimuli. Everything we have come to understand about learning and optimization from the field of machine learning supports the hypothesis that to learn anything, at any appreciable rate, you must introduce feedback loops that back-propagate the error between expected and observed in some way such that the model can be updated to reduce the error for future predictions. In other words, mutations (and epigenetics) are very NON-random, driven by the environment and by life-experiences and even by the conscious choices of the host organism. This is much less about Lamarckism (although epigenetics are proving Lamarck was pretty much right) and much more about *directed* evolution (i.e. evolution being a biological learning and optimization problem).

In summary, I claim that Darwin was (half) wrong: evolution is about fitness, but optimizing for a given fitness function is not necessarily a random walk.


How to legally work in the US as a student

Somebody asked me how to gain work authorization in the US as a student. I have lived here for 11 years on student (F-1) and working (H1-B) visas. Here is the quick summary I sent back about what I have learned:


During your time as a student on an F-1 visa, you can work 20 hours a week but it has to be on-campus. The best sort of job is a research assistantship or teaching assistantship, since the college will often also pay your tuition.

As far as working off-campus: the easiest thing is to apply for one-year CPT (Curricular Practical Training) during your degree, and/or one-year OPT (Optional Practical Training) after graduation. If you are doing a STEM degree (Science, Technology, Engineering, Mathematics), then you can apply for a 17-month extension once your 12-month OPT is finished. CPT and OPT give you an actual EAD (Employment Authorization Document - a card) which gives you employment authorization, so now you can work more than 20 hours a week and you can work off-campus. You can even work multiple jobs, one or more of which can be self-employment.

After OPT, the easiest thing to do is get an H1B to work at a specific company. (Or marry a US citizen.) Generally a company will sponsor you for two 3-year H1B visas before sponsoring you for a green card. You can't apply for a green card directly from an F-1 visa, it is a "non-immigrant intent" visa. You can apply directly from an H1B, it is a "dual-intent visa".

Another option is to apply for an O-1 visa ("alien of extraordinary abilities") if you have some major award or accomplishment that is equivalent to national recognition in the US. However an O-1 has non-immigrant intent. If you ever want a green card, the better option is an EB-1 visa, which is dual-intent. You can then apply for a green card on the National Interest Waiver program, which allows extraordinary aliens to get fast-tracked to a green card if you can prove that it is in the United States' interest to do so. The NIW qualification can be proved by a string of high-profile publications in top journals, among other things.

There's also the green card "diversity lottery", which you should apply for every single year, since it doesn't actually count against you as an attempt to gain citizenship when they ask you at the border if you have ever sought citizenship on your F-1 "non-immigrant intent" visa (you don't have to declare dv lottery attempts): https://www.dvlottery.state.gov/ (Beware of all other sites than this one, some sites will charge you $10 to apply, this one is free and is the official site.) Applications open later in the year. Your chances at getting a diversity lottery green card range from quite high to vanishingly small depending on what country you are from.

There is also a startup founder visa that has been proposed, and the bill is, going through the system right now. You have to employ a certain number of Americans, and bring in a certain amount of funding within the first 1-2 year period for the visa to get renewed.

Here is info on the startup visa act. Everybody should consider supporting this.


The last link also talks about the possibility of working on an EB visas. Notably, EB-5 is a category for people investing a lot of money in the US. This can get you in the door if you can afford it.


Update: JBQ posted the following comments on my G+ post that links to this blog post:

Chances are, if you'd qualify for an O-1, you'd also be EB1 in the green card process and that'd probably be a very easy path.

Also, there's no benefit in waiting to apply for a green card when starting a new H-1B job. It just delays the priority date and therefore the waiting time.

Finally, the total experience doesn't matter when converting an H-1B to a green card, what matters is the experience when getting hired. Switching companies can be beneficial, as there can be enough experience to move from EB3 to EB2, while keeping the priority date. It's best to do that with more than a year left before the 6-year line, and it's best to do that with a priority date far enough in the past.

IIRC there are no specific requirements to apply for a green card.

As it was for me, the process starts with a certification by DOLETA that there are no citizens or residents to fill the job (similar to an H-1B, but a bit more thorough). Applying for this also sets the priority date.

Once that's done (a few weeks), the next step is to apply for the visa petition (proving that the employee is qualified and that the employer can pay them). That's also similar to an H-1B. That can take a few months IIRC. This is the I-140. Don't wait long as the labor certification is only valid for a limited period.

Once that's done, the next step is to wait for an available visa. There are 3 waiting lists per country of citizenship, based on the skill level. The waiting lists are shorter for the categories with the highest skills. Each list is represented by a cutoff date, and if your priority date is earlier than the cutoff date for the category you're in you're eligible for the next step.

Finally, once a spot is available, applying for an I-485 adjustment of status turns the H-1B into a green card. That also takes a few months IIRC.


Intolerance: criticizing what somebody *is*, not they ideas they believe in

Somebody just asked the following question to the csail-related mailing list about the appropriateness of political slurs during public talks:

On Wed, Mar 14, 2012 at 9:19 PM, IJ wrote:
I was wondering what people in CSAIL think about speakers including gratuitous political insults in their talks.

I was at a talk last week at HMS about systems-based analysis of disease. The speaker, Joseph Loscalzo (Chairman of Medicine at the Brigham), said that before Hippocrates people thought illness was caused by evil spirits. He then added that this view is shared by Republicans.

Coming from a background in industry where one often encounters very nice, very intelligent people of all political leanings, I found it shocking that the speaker would be so unprofessional as to insult people who, for whatever reason, have a political affiliation different from his own. Still worse was his subsequent joke that there might "even" be some Republicans in the audience, with its presumption that all or almost all of his audience must share his political views. I thought he would next suggest that if we spot one of these Republicans we might examine him or her as an interesting specimen!

As discussed in this New York Times article, and borne out by my own acquaintances, academia can be a hostile environment to people who are not liberal Democrats (www.nytimes.com/2011/02/08/science/08tier.html). The researcher mentions how these non-liberals remind him of closeted gay students in the 1980's, how they "hid their feelings when colleagues made political small talk and jokes predicated on the assumption that everyone was a liberal." I know of medical students who were justifiably afraid that they would be discriminated against if their political affiliations were "outed".

I hope that in CSAIL we would not tolerate remarks like these that create a hostile environment for any of our members, students, or guests, whether they are women, gays, or even Republicans.

Does CSAIL have a policy on this?

Should it?

RMS jumped in with the following:

On Thu, Mar 15, 2012 at 12:22 AM, Richard Stallman wrote:
People are responsible for their opinions; criticizing and even
condemning political opinions is a normal part of political discourse.

Here's my take on it:

There is a difference between criticizing a theory that somebody holds, and something that a person *is*. The former is the foundation of academic discourse, and theories must be able to withstand scrutiny to be of value. The latter -- criticizing what somebody is, and either hasn't chosen to be (in the case of phenotypic attributes -- skin color, gender etc.) or has chosen to be, by culture or agency (religion, political orientation etc.), is intolerance. In the context of the original poster's situation, it is OK to criticize a theory about how the government should be run, and to subject that theory to academic discourse about relative merits or lack thereof. It's not OK to poke fun at somebody's *identity* as a person that follows a given ideology ("one of those people", in label-speak).

I really don't like the word "intolerance" however: the grand irony is that much of the time that this word is used, it is used to superciliously indicate that another person's views are quaint, and not broad enough to include one's own views. Whether or not that is true, unless one party is being harmed, tolerance must be extended in both directions, or a claim of intolerance is plain hypocrisy.

The word "intolerance" seems to have therefore lost a lot of its real meaning, because it is often used in this self-serving way. Is there a less worn-out word available for use than "intolerance"? (I guess this is why policies about this generally refer to harassment, not intolerance, because they deal with cases where one party is in fact harmed?)