2010-12-31

Why we all need to learn Chinese

I had to write a short essay about an emerging issue in US-China relations and what I think should be done about it. I figured I'd post it here too. Short story: we all need to learn Chinese.

--

The next 10 years will see the beginning of the end of the US' "free lunch".  As the standard of living continues to improve in China and as economic reform and access to information continues to spur growth, wages and prices will rise, which will cause an increase in the cost of goods manufactured in China, and much of this cost increase will be passed on to the American consumer. The increase in the cost of consumer goods may make continuing to import goods from China unsustainable for some industries. Unfortunately the US has already lost (or never developed) the ability to manufacture certain goods and materials in quantity, and has long relied on cheap manufacturing in Chinese factories. Chinese economic growth is therefore likely to cause tensions between China and the US.

Meanwhile China has started investing heavily in outsourcing cheap manufacturing to Africa and other developing regions, so it is likely that China will emerge as the next super-consumer country, and with an emerging middle-class and much greater purchasing power than the US (and maintaining trillions of dollars of US debt), the rise of China will likely drag the US into economic doldrums.

The traditional business and economic approaches to address this problem will all of course be pursued (investing in emerging Chinese markets, exporting Western brands to China and/or developing multinational business conglomerates).  However I think to truly stay relevant, the US needs to focus on teaching Chinese language and culture to every school student the way that every Chinese school student is taught English language and culture, and the US government needs to focus on setting up an extensive network of student exchange programs with China and other Chinese-speaking countries.  By exposing school children to Chinese language and culture, the next generation of business leaders, political leaders, scientists and engineers will be enabled to work alongside Chinese counterparts rather than simply competing against them while the economics of scale turn in China's favor.

2010-12-14

My quick analysis of the leaked Gawker passwords

I just got an email from Gawker Media stating that their login details on Lifehacker, Gizmodo etc. had been compromised and a database of 1.3M usernames and passwords was being distributed via Bittorrent. Naturally I went and found the database and downloaded it. I extracted the subset of passwords from the file that have already been cracked, and uniquified and generated counts. You can download the list at the end of this post.

I have written about the dangers of using one password on multiple accounts before, and when I used to work at a company where I had access to a massive password database, I was shocked to discover how many people use really weak passwords -- like a first name or a number like 123456, or the word "password".

The leaked Gawker data contains the following explanatory text (along with a ton of leaked private chat logs between Gawker executives, and other juicy stuff):


  After gaining access to gawkers MySQL database we stumble upon a huge
  table containing ~1,500,000 users. After a few days of dumping we
  decided that 1.3 million was enough.


  Gawker uses a really outdated hashing algorithm known as DES (Data Encryption Standard).
  Because DES has a maximum of 8chars using a password like "abcdefgh1234" only the
  first 8 characters "abcdefgh" are encrypted and stored in the database. If your
  password is longer than 8 characters you only need to enter the first 8 characters
  to log in! 


  YA DONT SAY!! :D?


  Because of this we were only able to recover the first 8 characters of someones password!
  If the password is 8 characters long there's a good chance that it migt be longer
  than 8 characters! But still, there's 1000's of people using 1 - 8 character passwords
  for us to have some fun with! 


  We managed to crack ~200,000 hashes, if you want the rest of them cracking
  DO IT YOUR ****ING SELF! >:3 (censored)

So ~200,000 hashes were cracked out of 1.3M by de-hashing (actually 188281 hashes were cracked, producing 91688 unique passwords). I assume that the 189k passwords that were cracked are somewhat representative of the rest of the database.

I ran some basic statistics on the password database because I was interested in seeing the distribution of password usage.  Here is a plot of the usage count (out of 189k cracked passwords total) for the top 50 passwords:


Here is the same plot with a log Y axis and with the rank of all cracked passwords shown on the X axis:



Basically the top 5 or so passwords are used by a ridiculously high proportion of users, and the top few thousand passwords are very common and therefore very easy to guess using a dictionary attack.

Here are the top 50 passwords, with their rank and count out of 189k:


Rank    Count    Password
   1     3057    123456
   2     1955    password
   3     1119    12345678
   4      661    lifehack
   5      418    qwerty
   6      333    abc123
   7      311    111111
   8      300    monkey
   9      273    consumer
  10      253    12345
  11      247    letmein
  12      241    trustno1
  13      233    dragon
  14      213    baseball
  15      208    superman
  16      202    iloveyou
  17      202    1234567
  18      199    gizmodo
  19      196    sunshine
  20      194    1234
  21      187    princess
  22      184    starwars
  23      179    whatever
  24      175    shadow
  25      158    000000
  26      157    cheese
  27      156    123123
  28      149    nintendo
  29      149    football
  30      148    computer
  31      141    ****you (censored)
  32      135    654321
  33      134    blahblah
  34      132    passw0rd
  35      132    master
  36      126    soccer
  37      124    michael
  38      120    666666
  39      118    jennifer
  40      115    gawker
  41      114    Password
  42      114    jordan
  43      113    pokemon
  44      113    pepper
  45      113    michelle
  46      113    killer
  47      111    welcome
  48      111    batman
  49      109    kotaku
  50      109    internet


This gives an insight into the password-setting habits (and, if you read through more of the list, the mentality) of a large proportion of the Internet-using population.

A lot of people use numerical passwords -- here are the top 50 numerical passwords. Check out the password at rank 221:


Rank   Count    Password
   1    3057    123456
   3    1119    12345678
   7     311    111111
  10     253    12345
  17     202    1234567
  20     194    1234
  25     158    000000
  27     156    123123
  32     135    654321
  38     120    666666
  74      82    123321
 100      72    123
 101      72    121212
 137      63    159753
 163      56    88888888
 164      56    11235813
 186      53    7777777
 202      50    555555
 221      48    8675309
 236      47    98765432
 237      47    11111111
 243      46    696969
 253      45    112233
 267      43    00000000
 272      42    1111
 286      41    123654
 318      39    222222
 344      37    131313
 430      32    0000
 501      29    987654
 502      29    55555
 537      28    12341234
 608      25    102030
 643      24    147258
 645      24    101010
 684      23    888888
 685      23    159357
 741      22    789456
 742      22    11223344
 743      22    007007
 799      21    12312312
 869      20    99999999
 871      20    147852
 872      20    1212
 873      20    11111
 874      20    09876543
1105      17    0123456
1218      16    151515
1219      16    123789
1221      16    112358


DOWNLOAD LINK: Curious to see the passwords of all 189,000 users? Here's the whole password list with counts for each password.  It's a .tsv file (tab-separated values), you can load it into a spreadsheet or text editor. (This file doesn't contain names or usernames, just the password info.  If you want usernames you'll have to go get them yourself.)

UPDATES:

  1. Ranks and accidentally-stripped leading zeroes fixed.
  2. Highlighting one of my replies to a comment: if you have to even ask if your password is on this list, it's probably not secure enough! 

2010-08-09

Someone stole your password on facebook or your email account? What to do about it -- and why it's worse than you think

I get emails or facebook messages "from friends" every couple of weeks that are spam, and that the friend clearly did not intend to send.  It is obvious that the password on the friend's email account or facebook account got stolen.  This is potentially a lot more serious than it seems -- this can make old-school identity theft look like childplay:  If somebody steals your email password they have everything about you.  They can not only trawl through all your email to gather information about your life for impersonating you, blackmailing you or other nefarious purposes, but they can also send password reset requests to any other website you have registered with, and steal your passwords on all those sites too.

For this reason, your email password is gold, and you should protect it more than any other password: it should be harder to guess than other passwords, longer, not be based on dictionary words etc., and you shouldn't use your email account password on any other website!  Why?  Because when you register on another site XYZ.com, and XYZ.com asks you to register with your email address and a password, and if you use your email account password as your password on XYZ.com too, then if that website's owners are evil or if somebody hacks their site, they have both your email address and your email account password.

There are three main way people steal your email passwords:

(1) Dictionary attack. I had access to a database of thousands of user login credentials in one job I worked at, and more than 60% of passwords were just a first name!  Scammers/spammers just go through a dictionary and submit every word in the dictionary to a website as passwords with common or publicly-visible usernames until they get in.  This is becoming less common because of security precautions like having to type in a Captcha when you get the password wrong, but still -- if you have a simple name or word as your password, fix it!  Use letters, numbers, upper and lowercase, and punctuation.  Make it at least 8 characters long.  Write down passwords somewhere secure and memorize your main email account password.

(2) Keylogging.  This is becoming the most common method for stealing passwords today: most viruses that infect your computer will install a keylogger that sends every keystroke you type -- login names, passwords, credit card numbers, love letters -- to a computer in China or Russia.  And a substantial number of computers you have probably used in the last few years have had one of these viruses on them.  No I'm not kidding.  See below for how to clean up your computer if it's infected, and general guidelines of how to avoid this issue.

(3) You used your email account password on other sites too, and those sites were either malicious or got hacked. It was recently revealed that around 75% of people reuse their email password on social networking sites like Facebook.  Don't ever, ever use your email account password to register anywhere else.

Q: My password got stolen.  What do I do?

Step 1: If you can log into your email account and the spammers literally just started sending out messages from it, log in immediately and change the password.  You might have to change the password again later, because your computer may still be infected with a keylogger (see point (2) above) and they might be able to watch you change your password.  But change it now to halt them in their tracks.  If you can't log into your email account because they changed your password on you (it does happen), then try sending a password reset request, if your account is connected with another email account that you can access, or contact your email provider (or facebook, if it's your facebook account) to notify them of the problem.

Step 2: Assess the damage.  Look in your Sent folder to see what they sent and to whom.  Send out a "Sorry -- don't click on this link, I didn't send it" message or something -- lots of the links sent out by spammers go to sites that try to infect more people's computers with viruses.

Step 3: Get all the critical updates for your computer and update your antivirus software.  Follow all the steps in my other blog post about speeding up your computer.  Following those steps will not only speed up your computer but should guarantee that any viruses on your computer are killed and that you are safe from viruses in the future.  In the end you should (a) have all the critical Microsoft updates on your computer, (b) have replaced your antivirus software (Norton/Symantec/McAfee/Kapersky/ClamAV/etc.) with Microsoft Security Essentials (it's a lot better at catching viruses than the others, and it's free forever so you don't have to pay to stay up to date so your computer catches the latest viruses), and (c) you should have Google Chrome installed as your browser and you should never use Internet Explorer again, it is one of the biggest reasons computers get infected with viruses, because it is so insecure.

Step 4: With your new bulletproof computer, go to any other site that you used the same password on as the email password that was stolen, and change your passwords there too.  Then probably go back and change your main email account password again, just in case you had a keylogger on your computer that got eliminated in Step 3.  Here's a general strategy for selecting passwords:
  • Password Level 1: Have a throwaway, "don't care" password that you can use for all those sites that ask you to register that you wouldn't mind someone stealing your password for -- like when you have to register on a bulletin board to ask a question about your car, and you never plan to go back there again.  Reuse this password on other sites only when you don't care if somebody who has access to one of those sites with a stolen password has access to all of them -- because if your account gets hacked on that site, it can get hacked on other sites too.
  • Password Level 2: Have a second level of password that you use for sites that you do care about somebody breaking into -- for example any e-commerce site like Amazon.com that saves your credit card details.  Use a different password for each site that stores credit card details!  Remember that sites that save credit card details are especially targeted by hackers/crackers.  Ways of generating unique passwords for these sites include just writing each password down, coming up with a tricky way of taking a base password and combining some letters from the domain name of the site into the password to make a non-predictable unique password for each site, or using something like LastPass to generate and save the passwords for you.
  • Password Level 3: You should have a more secure password for your bank and your bank only.  Each bank should have its own password.  You should also request extra authentication methods from your bank if it is available -- e.g. PayPal has a keyfob that generates a pseudo-random number when you push a button (a new number every minute) that you have to type in right after your password.  This verifies you have the physical device, and protects you from password attacks.
  • Password level 4: Your main email account password on gmail, yahoo, hotmail etc.  This must be secure, unique, and unguessable.  Note that I put the security requirements for this password even higher than the requirements for your bank password.  Don't underestimate what somebody can do with your email account password.

    Step 5: In future, don't log into your main email account on any computer that is *not* running Microsoft Security Essentials, and don't ever use Internet Explorer to do it -- no matter how badly you need the Internet.  I never, under any circumstances, ever log into my email account on a Windows computer that is not my own (but I'm more paranoid than most, I just deal with the fallout all the time when people come to me to fix their virus-ridden computers, and I know that most people have not followed the steps above for guaranteeing their computers are virus-free).  Getting a cellphone that you can read your email on helps with this, it lets you check email and reply even when you're away from the safety haven of your own computer.  In particular, avoid Internet cafes especially while traveling -- a large majority of computers in Internet cafes are infected with some sort of spyware that will send your passwords and credit card numbers who-knows-where.

    Good luck :-)

    2010-08-02

    Taking OpenCourseWare to North Korea

    A blog post I wrote for ChosonExchange.org, cross-posted from there.  I'm going with Choson Exchange to Pyongyang in September -- my second trip to North Korea -- and I'm heading up their OpenCourseWare strategy.

    ----

    Choson Exchange to Share Creative Commons Licensed Materials from the World's Best Universities With North Korea 

    Choson Exchange is committed to providing educational materials from the world's best educational institutions to North Korean students free of charge. This goal is made possible through the OpenCourseWare (OCW) initiative, in which dozens of top universities all around the world have chosen to post a large number of course materials such as lecture videos, lecture notes, handouts and assignments on the Internet under the Creative Commons open access license. This license gives people all over the world the ability to obtain a top-quality education for free, and gives professors the ability to legally reuse these materials and incorporate them into their own teaching.

    Several other sources of top-quality educational materials are also available under Creative Commons licenses, such as lectures on a wide array of topics in mathematics, economics and finance from the Khan Academy, full high-quality textbooks on WikiBooks.org and encyclopedic content on Wikipedia.org. Recently, WikiBooks and Wikipedia added the ability to select sets of articles and have them assembled into a PDF format e-book for downloading, or these books can be easily printed, bound and shipped with a few mouse clicks through a company called Pedia Press. This provides an easy method for creation of high-quality printed textbooks or e-books that meet the content and pedagogical requirements of our North Korean colleagues.

    Choson Exchange has been invited to present Open CourseWare content and e-books at the Pyongyang International Science and Technology Book Fair (PISTBF) in September 2010. The initial content that we will take to North Korea includes both OCW and Wikipedia/WikiBooks-sourced material in the subject areas of business, economics and finance; basic sciences such as physics, chemistry and biology; medicine, including first aid, physiology and gynaecology; computer science and engineering. We plan to bring both electronic copies of lecture videos and lecture notes as well as printed copies of some WikiBooks to use in exhibitions in Pyongyang and training programs.

    The quality of many of the materials available through Creative Commons sources is very high. However no educational program can stand on the strength of the educational materials alone, there is a lot of structure and that has to be put in place for an educational program to succeed. For this reason, Choson Exchange is also committed to helping create and support the teaching infrastructure necessary to effectively kickstart training courses incorporating open content. To accomplish this, foreign advisors who are expert in each teaching area are being recruited to assist in helping their North Korean counterparts get up to speed with teaching the new academic material. We are confident this is the fastest way to improve the quality of education, and that improving education will improve quality of life and the level of wellbeing of the country.

    Finally, North Korea is unprecedented in its culture and rich history. As we work with our North Korean colleagues to bring the highest quality Creative Commons academic materials from the best educational institutions to North Korea and help them to build programs that employ these resources, we would also like to work with them, if they choose, to contribute North Korean literature, cultural and academic course materials back into the body of Open CourseWare, so that the world can learn about the North Korean story directly from North Koreans themselves. This will add to the richness of the cultural tapestry that is the Creative Commons.

    Posted by Luke Hutchison, Director of Educational Technologies for Choson Exchange

    2010-07-23

    Why Harvard Students are successful; dealing with Harvard Karma Envy

    I just tweeted:

    @MetaLev: The Social Network http://goo.gl/rarh looks semi-interesting, Zuck well-casted, but actors look more like Hollywood than Harvard

    I immediately got the following response:

    @positiveneuro: I think our different reactions to the social network are so interesting! I'm electrified by it! You're so humdrum about it! I wonder if it's because Harvard is not a land of mystique for you--it's your daily. For better or worse, the whole meme of Harvard seduces me.

    (I take classes at Harvard and attend MIT.)  I posted the following reply, which I thought was worth reposting here, because I think I learned something by tweeting it:

    @MetaLev: Harvard seduced me too until I went to take classes there, and realized everyone is pretty normal.  Normal but with an edge that I would best describe as a tendency to choose to thrive. They're a bit more enthusiastic and excited about possibilities and opportunities than the average person. That is all. Although that's a big "all", it has remarkable consequences. I hope this is teachable, I want my kids to take this approach to life.

    and, because karma envy is relevant to any discussion about Harvard, here's some more of the conversation:


    @positiveneuro: That's really insightful re:differences about Harvard kids, Luke. That's GOT to be a conditioned attitude. A friend's comment to me: "I think you reacted so strongly to the preview because you want to be a mark zuckerberg"

    @MetaLev: Yeah -- karma envy :-) Be your own MZ.


    @positiveneuro: I can't remember if we've discussed this principle from the bhagavad gita or not: it is better to live your own destiny imperfectly than to live someone else's destiny perfectly

    @MetaLev: Hmm. You could also say it's better to live your own destiny perfectly than someone else's imperfectly.  You probably won't be the next Mark Zuckerberg, you'd be imperfect at it if you tried. But he will never be the perfect you either.

    2010-07-02

    Post updated, see new link below

    This post has been superceded by a newer post illustrating the Android vs iPhone 4 signal-strength-to-bars mapping, please visit that post instead.

    Signal-strength-to-bars mapping on iPhone 4

    Apple released the statement today that they were "stunned to find" (could they be any more dramatic?) that the formula used to map signal strength to bars on the iPhone 4 is "totally wrong".  They state they will fix the problem by reducing the number of bars displayed for a given signal strength.

    "Our formula, in many instances, mistakenly displays 2 more bars than it should for a given signal strength.  For example, we sometimes display 4 bars when we should be displaying as few as 2 bars. Users observing a drop of several bars when they grip their iPhone in a certain way are most likely in an area with very weak signal strength, but they don’t know it because we are erroneously displaying 4 or 5 bars. Their big drop in bars is because their high bars (sic) were never real in the first place (sic again)."

    HOWEVER I claim that they will increase the number of bars for weak signals, and only decrease the number of bars for medium-strength signals.  Here's the reasoning:

    • The open letter implies several times that when the number of bars is low, the reading doesn't reflect the actual operability of the phone, i.e. that the low end of the bar scale is too low: "iPhone 4 can drop 4 or 5 bars when tightly held...This is a far bigger drop than normal, and as a result some have accused the iPhone 4 of having a faulty antenna design.  At the same time, we continue to read articles and receive hundreds of emails from users saying that iPhone 4 reception is better than the iPhone 3GS. They are delighted. This matches our own experience and testing...The iPhone 4's performance is the best we've ever shipped".
    • They say "To fix this, we are adopting AT&T’s recently recommended formula for calculating how many bars to display for a given signal strength", but it's impossible to believe that they'll just apply AT&T's formula without tweaking it if you also believe their claims that the iPhone 4 really is better at operating on a weak signal than previous phones -- because bars are a subjective measure of the cellphone's ability to connect over the cell network.  These two claims simply don't match up.  They'll jack up the number of bars compared to the AT&T recommendation if it really is true the iPhone 4 can get the same quality of connection on a weaker signal than previous iPhones.
    • There would be a public outcry if they just drop the signal threshold for each of the bars (a drop of 4->2 could now be 2->1 or 2->0 if they are just displaying "two more bars than we should"), or even if they just dropped the signal strength threshold of the high-numbered bars -- "I got the iOS update and now my phone has a weaker signal than ever!".  The whole problem is that people believe bars and don't understand the science or math behind the bars.
    • Apple's magic fix for previous iPhone signal strength problems was supposedly an inflation of the number of displayed bars.  (It keeps people happier.)

    The great irony of all this is that it appears Apple has been caught red-handed trying to inflate their signal strength in the first place...

    2010-06-26

    Is your computer slow? Fix your own computer.

    I have fixed about six computers in the last two weeks, and just like 95% of the other computer help I give people these days, it came down to this one problem:

    "Why is my computer so slow?"
    "My computer seems to have a virus, how do I fix it?"

    One thing I have noticed is that most people just live with slow computers because they don't know where to turn for help, or they don't know that things could be better.  I figured it was time to write this blog post -- slow computers are fixable, and in fact you can fix it yourself.  The following steps should fix 95% of your computer slowness issues, and your computer will probably feel 2x-5x faster when you are done.  Please spread the word, you don't need to suffer in silence anymore :-)

    (Note that this blog post applies to the Windows operating system, but you should be aware that there are other alternatives that don't get slower the longer you use them, including Mac OS, Linux and soon Google Chrome OS.)


    HOW TO FIX YOUR OWN SLOW WINDOWS COMPUTER, AND ELIMINATE VIRUSES:


    Step 1: Fix DNS settings (if necessary)

    Some viruses redirect Web traffic to malicious sites, so you can't actually download or update antivirus software, etc. to fix your computer. If antivirus sites are inaccessible, and/or you get suspicious popups or fake sites when trying to visit well-known domains, follow the advice here and/or here.


    Step 2: Disable unnecessary startup processes

    Explanation: One of the biggest reasons your computer is slow is probably that when you switch your computer on, it loads all sorts of programs into RAM (memory) that are not needed.  (A lot of these programs put icons down in the System Tray at the bottom left of the screen, but not all of them do.)  When you run out of RAM, your computer uses the hard drive as extra storage (it "swaps out to disk"), and the hard drive is on the order of 1000 times slower than RAM.  So the first thing you should do is stop all the unnecessary programs from starting when you boot, and you'll be less likely to run out of RAM.

    How to do it: Download, install and run a startup manager.You will then be presented with a list of programs that are run when your computer starts up.  In most cases, you won't break anything really badly if you uncheck all of them -- but try to understand what they each are before you uncheck.  If you don't know, you can always uncheck it and come back and re-check it again if you notice something doesn't work right.  A few cases to get you started:
    • If you disable anything that says "Synaptics" or "SynTPE" then your touchpad might not have the full functionality, e.g. this program detects movement on the right hand side of the touchpad and causes the current window to scroll without you having to drag the scrollbar.  You probably want to leave this checked.  Also anything that talks about hotkeys handles the Fn+F4 key combinations etc., leave that checked if you need those functions.
    • If you sync an iPod with your computer, you probably want to leave the iPod stuff checked.
    • If you see something about HP printing and you have an HP printer, you can uncheck this and you will still be able to print, but you might not get notified on your computer when your ink is running low (not a big deal).
    • If you uncheck something to do with your digital camera, then your photo editing software might not pop up when you plug in your camera, but you will get the standard Windows photo import window instead, and you can still start your photo editing software manually, so again it's not a big deal if you uncheck it.
    • The programs you really want to kill is anything that says "QuickStart" or similar -- Acrobat reader, OpenOffice and other programs have quickstart options.  They may be bringing your computer to its knees by filling up your memory.
    • There are usually programs that start up that put advanced control panels for your graphics card and/or sound into the system tray.  However you can usually disable these too and your sound and graphics will continue to work fine.
    • In general, if you don't know what it is or why you need it, you probably won't miss it -- uncheck it!  Some people would think this is extreme and generally bad advice -- but honestly, your computer won't be in worse shape than it was before when it was unusably slow :-)  And again, you are not likely to break anything serious, but if something doesn't work properly after this step (e.g. your webcam doesn't work right), you can just try switching some of these programs back on again (by re-running CodeStuff Starter) and rebooting until you figure out what needed to be switched on.

    Now reboot your computer and hopefully your computer will feel faster already.

    Step 3: Get all Microsoft updates

    Explanation: Microsoft pushes out critical and non-critical updates that fix bugs on your computer, speed your computer up, and make your computer less vulnerable to viruses and spyware.  (Viruses and spyware are another big reason why computers can slow down.)  In particular you need all Service Pack downloads, and you should update Internet Explorer to version 8, as this fixes some critical parts of Windows -- but you should get all the critical and recommended updates if you don't have them already.

    How to do it: Open Internet Explorer and go to http://update.microsoft.com/ .  You may be asked if you want to update from Windows Update to Microsoft Update.  If you have the option, definitely install Microsoft Update, it is better than the older Windows Update because it keeps not just Windows but also Office up to date.  Next follow directions to check for the latest software updates, and download and install all critical and recommended updates.  You can also manually select a few optional updates here but you don't need them.  You probably have to reboot after installing updates.  Once you have rebooted, go back to http://update.microsoft.com/ and make sure there aren't new recommended updates listed that were there before.  If there are new updates, then rinse and repeat.  (Sometimes you can't install all updates at one time, some of them have to be performed in separate steps.)

    Step 4: Replace your antivirus software with Microsoft Security Essentials

    Explanation: Antivirus software has to wedge itself into all sorts of critical parts of your operating system to stop viruses in their tracks.  As a result it can sometimes cause more problems than it prevents.  Also you have to pay $60-100 per year for antivirus updates, otherwise your computer is still at risk from being infected by the newest viruses.  Microsoft recently realized they need to start cleaning up the messes they created by selling an operating system full of security holes, and they released a product called Microsoft Security Essentials, which seems to be one of the few products they have gotten really right -- it's solid, fast, and best of all free forever: they will keep sending you updates for free and you never need to pay the Symantec / Norton / McAfee yearly tax again.

    How to do it:
    1. Go to Start -> Settings -> Control Panel, and find the Control Panel program that lets you uninstall programs.  You might have to click on "Classic View" on the left hand side of the Control Panel to find it easily.  It might be called "Add/remove programs" or "Programs" or something else, they keep changing the name of it in different Windows versions, and I can never remember what it's called on each Windows version (I'm a Linux user) -- but when you find it it will have a list of all the programs you have installed.  Uninstall anything that has "Symantec", "Norton", "McAfee", "ClamAV", "Kapersky" or similar in the name.  Also uninstall anything that contains keywords like "Antivirus", "Spyware", "Spybot", "Internet security" etc. in the name.  It's important to uninstall these before installing Microsoft Security Essentials, and don't worry, you'll only have it uninstalled briefly.  You'll probably need to reboot after installing one or each of these.
    2. After rebooting, go to http://www.microsoft.com/security_essentials/ and download and install Microsoft Security Essentials.  It's pretty easy to install, and when it is done it will ask if you want to download the latest virus definitions and run a scan -- choose Yes.
    Step 5: Install the Google Chrome Web browser and never use Internet Explorer to browse the Web again

    Explanation: Something most people don't know is that 95% of virus infections today come from using Internet Explorer.  You should only ever use it for Microsoft Update and for nothing else.  Something a lot of people also don't know is that they have a choice when it comes to web browsers.  Internet Explorer is a terrible program, Firefox is better, safer and faster, and Google Chrome is like Fort Knox as far as security and it's faster than greased lightning.  Using Google Chrome will therefore protect you online and it will make your computer feel even faster.

    How to do it: Go to http://google.com/chrome and download and install Google Chrome, and then go find any Internet Explorer icons on your desktop or in your Quick Launch tray (bottom left, by Start) and delete them so you don't accidentally use IE again.  Use Chrome for all your browsing, I promise your computer will feel much, much faster.

    THAT'S IT!  Enjoy your new turbo-charged computer.

    2010-06-20

    Clustering large datasets

    On the MIT-internal "csail-related" mailing list someone recently asked for software to help him perform matrix multiplications of 10^6 x 10^6-sized matrices.  Ron Rivest quite correctly replied that to multiply matrices this size, even for a single multiplication you would probably need about 4 years of compute time -- because there are a trillion entries in matrices this size.  I posted the following reply, which I am re-posting here partly for my own reference because it contains a lot of the points I have learned in various work clustering huge datasets, and partly in the hope that somebody else will find it useful.

    --

    Ron's analysis is correct, unless your matrix is very sparse -- in which case sparse matrix methods may make this problem entirely tractable, and any of the linear algebra toolkits implement efficient sparse matrix methods that you can use. The main problem you'll have is fitting it all in memory -- you'll need to look into matrix blocking techniques (dividing the big matrix into submatrices, and figuring out the correct way to multiply the subblocks to get the full result). There's some great discussion about keeping subblocks in CPU cache in the following paper that may help you figure out how to keep your much larger subblocks in main memory as long as possible: http://homepages.cae.wisc.edu/~ece539/software/matrix/mbpv11.ps The difference between swapping subblocks in and out at the right time vs. the wrong time could make several orders of magnitude in difference in how long it takes to compute your result. There are also some parallel solutions to multiplying large matrices that will run on large clusters and trade off time swapping subblocks in and out of memory for data communication overhead between nodes.

    A similar problem to matrix multiplication problem you describe is encountered in data clustering, given the "N Choose 2" or O(N^2) scaling of the number of pairwise distances in a dataset. It is intractable to use all-pairs distances with even the simplest clustering algorithms in large datasets, for example hierarchical agglomerative / bottom-up clustering applied to more than tens of thousands of points. Depending on the exact nature of the problem, you may be able to exploit spatial coherence within your problem space -- e.g. for agglomerative clustering, you use only the nearest neighbors when joining clusters, so you can often reduce the complexity of clustering problems using a smart data structure like a kd-tree that gives approximately O(log(N)) time per nearest neighbor lookup. However the kd-tree algorithm quickly degrades to ~O(N) performance per lookup in high-dimensional spaces because of the curse of dimensionality, so you may need to do dimensionality reduction first anyway to make the kd-tree useful. (You'd also have to reframe your matrix multiplication problem into a format where using a kd-tree to find nearest neighbors in your vectorspace helps you compute a fast, close approximation to your desired solution.)

    Another approach used to cluster datasets with millions of points (and therefore trillions of pairs of points) is to pick a few exemplar points and cluster these instead to generate a sample approximation of cluster assignments for the full dataset. For example you could randomly choose one point out of every thousand, and cluster these into your K target clusters (= O(N^2 / 1000^2) time to cluster 1/1000th of the points), then go back through your full dataset and find the closest exemplar point to each original data point in order to compute the cluster labeling (= O(N^2/1000), although you can often skip this step entirely if you just use the exemplars). The largish constants in the time complexity can make this approach tractable for larger datasets than you could otherwise work with. The exemplar method I just described is incidentally half an iteration of the k-means/k-medians algorithm applied to a set of exemplar points. You can go further by completing the full first iteration of k-medians by going back and updating your selection of exemplar points using the medians of the newly-labeled clusters, and then depending on how much compute time you can afford, you could run multiple whole iterations of this exemplar-modified k-medians algorithm (or run until convergence) to get better exemplars -- though even the first half-iteration may be sufficient to get you a useful result. As far as how to phrase your matrix multiplication problem in this framework, depending on the problem you may be able to find representative row/column vectors this way and then just multiply the representative vectors together to get a product that in some sense is a low-dimensional approximation of the full matrix product.

    Ultimately whether you use (PCA-based) dimensionality reduction or k-means / k-medians, you're doing approximately the same thing, and this is why preprocessing with PCA can often help k-means to converge faster : quoting from http://en.wikipedia.org/wiki/Principal_component_analysis#Relation_between_PCA_and_K-means_clustering :

    It has been shown recently (2007) [12] [13] that the relaxed solution of K-means clustering, specified by the cluster indicators, is given by the PCA principal components, and the PCA subspace spanned by the principal directions is identical to the cluster centroid subspace specified by the between-class scatter matrix. Thus PCA automatically projects to the subspace where the global solution of K-means clustering lie, and thus facilitate K-means clustering to find near-optimal solutions.

    2010-05-12

    An "energy bubble" theory for the formation of ball lightning

    And now for something completely different: I just read that some scientists believe ball lightning is a phenomenon caused by transcranial magnetic stimulation (TMS).  This doesn't seem consistent with the huge amount of observational evidence that ball lightning is a real physical phenomenon.  I want to offer a different hypothesis.

    Wikipedia cites a review from 1972 that established the following commonly-claimed properties of ball lightning:

    • They frequently appear almost simultaneously with cloud-to-ground lightning discharge
    • They are generally spherical or pear-shaped with fuzzy edges
    • Their diameters range from 1-100 cm, most commonly 10-20 cm
    • Their brightness corresponds to roughly that of a domestic lamp, so they can be seen clearly in daylight
    • A wide range of colors have been observed, red, orange and yellow being the commonest
    • The lifetime of each event is from 1 second to over a minute with the brightness remaining fairly constant during that time
    • They tend to move, most often in a horizontal direction at a few metres per second, but may also move vertically, remain stationary or wander erratically.
    • Many are described as having rotational motion
    • It is rare that observers report the sensation of heat, although in some cases the disappearance of the ball is accompanied by the liberation of heat
    • Some display an affinity for metal objects and may move along conductors such as wires or metal fences
    • Some appear within buildings passing through closed doors and windows
    • Some have appeared within metal aircraft and have entered and left without causing damage
    • The disappearance of a ball is generally rapid and may be either silent or explosive
    • Odors resembling ozone, burning sulfur, or nitrogen oxides are often reported

    This all seems consistent not with an energy ball but rather with an energy bubble: a highly-charged outer surface layer of ionized particles and a highly oppositely-charged inner layer of ionized particles separated by a highly non-conductive layer -- like a bubble-shaped or topologically-closed capacitor.  The attractive forces of the outer layer towards the inner layer act as a "surface tension" that gives the bubble its spherical shape but also highly compresses the central air cavity of the bubble, until the electrostatic attractive force pulling the membrane complex inward is equal to the reactive force of the air pressure inside the bubble pushing outward.

    This model explains the following:

    • The spherical shape of the lightning ball.  (In the case of pear-shaped ball lightning, as shown in the "UFO or something?" video below, it is simply two bubbles stuck to each other with a central dividing surface, as frequently observed when blowing soap bubbles.)  A surface tension gradient is required to induce the Maraghoni effect for bubble formation, which would be present for an electric field surrounding a plasma.  This gradient would also explain the fuzzy appearance of lightning balls.
    • The tendency of the bubble to be attracted to some objects and surfaces and approach or glide along other surfaces.  This can be explained by the difference in electric field potential of the two bubble surfaces relative to the charge of the object or surface in question.
    • The reports of popping or even explosion when the lightning ball destabilizes.  This is the bubble popping, releasing the potentially huge interior air pressure, and causing a powerful arc due to electrical breakdown between the two bubble surfaces
    • The reports of sulfurous smell after the ball pops.  This is the production of ozone from the electrical discharge between the surfaces.
    Two questions remain: how would such a bubble form, and what could form the insulating layer?

    It turns out that induced current in a fully ionized gas becomes unstable or breaks down in a DC electric field of sufficient strength, i.e. above a critical threshold [Nonlinear Electrical Conductivity of a Fully Ionized Gas, Kun-Mu Chen, 1962].  Thus it is conceivably possible to wedge a layer of fully ionized gas (plasma, with net charge close to zero) between a layer of positive ions and a layer of negative ions, if done with sufficient speed and with sufficient induced field strength that current flow through the insulating layer is eliminated before the charges have a chance to equalize.

    As far as bubble formation, a lightning bolt would need to topologically enclose a pocket of air to trigger the phenomenon, though I don't know if it would also require a particular mixture of gasses, air moisture content or other conditions to start with.   I hypothesize you might be able to create ball lightning by blowing bubbles in front of a powerful enough van de Graff generator.  Can anybody test this?

    NOTE: People have created plasmoids in a microwave by using smoke clouds emanating from a just-extinguished match.  These are balls of ionized gas, and are not the same as the energy bubbles as I am describing here, but they also differ in properties from ball lightning, such as the fact that they disappear within 30ms of the microwave source being switched off.

    UPDATE: I emailed Antonio Pavão (see the video below) to ask him his opinion of this theory, and he said: Thank you for sending your BL energy bubble model for my appreciation. It is an interesting model, but I would suggest a quantitite treatment in order to prove the existence of the "highly-charged outer surface layer of ionized particles and a highly oppositely-charged inner layer of ionized particles separated by a highly non-conductive layer". Recently V. Bichkov has proposed a “quasi-solid-vitrified” cover layer around the BL, but I´m still not sure about his results. The existence of this layer is also claimed in the paper “On phenomenon of light radiation from miniature balls immersed in water” from Torchigin V. P., Torchigin A. V., Institute of Informatics Problems, Russian Academy of Sciences, Nakhimovsky prospect 36/1, 119278, Moscow, Russia. Our model considers a metal core surrounded by an atmosphere of silicon oxides.


    VIDEOS OF BALL LIGHTNING

    A video of ball lightning following some power lines:


    A video of some guys that created ball lightning by vaporizing silicon in the lab, Antonio Pavão and Gerson Paiva of Universidade Federal de Pernambuco in Brazil:



    "UFO or something":



    Switching to Scala

    I heard about Scala a while back and got excited about it, but it wasn't until I started reading Programming in Scala by Martin Odersky (creator of the language) et al. that I realized what an absolutely brilliant language it is.  I'm not even halfway through this book and my take on it so far is that I have wasted an immense amount of time in the half my programming career that I have focused predominantly on Java.

    The exciting thing is that Scala produces standard (and valid) JVM bytecodes, which (I believe) can be compiled to Javascript by GWT, work fine with Google App Engine, and can be cross-compiled to Dalvik bytecodes for use on Android.  I'm excited to try Scala with each of those platforms.

    At the same time as being really excited about Scala, I wanted to document here my initial growing pains with using the language.  I am finding Scala to be awesome in concept, but a little frustrating in practice, at least initially. The reasons for that include:

    • Poor IDE support compared to Java -- no Shift-Alt-R for rename in Eclipse, for example, and compilation takes a few seconds longer than Java which can be a bit frustrating when you don't know the language well and you're having to try a zillion things to get something to work.  I know there's a daemon version of the compiler that shortens compilation from cold to warm startup time, but I think the latest versions of Eclipse uses it and the Netbeans IDE purports to have moved to it in the latest release -- but it still feels like it takes as long as a cold startup of the compiler (8 secs for a Hello World program), which is unacceptable in my opinion for incremental hacking.  (Update, see below)
    • There is a slight impedance mismatch between Java and Scala libraries, e.g. Scala implements its own collections framework, and things can get a little confusing if you try to mix Scala code with Java code that heavily relies on collections. (Of course you can always call all the Java methods to handle collections, but then you can't rely on Scala's nice syntax for working with the contents of collections, so the code looks different.) Best to stick with 100% one or the other probably. (Update, see below.)
    • I'm starting to discover a couple of tiny warts in Scala that are due to trying too hard to map onto the Java/JVM world. I had heard they exist, but now have experienced a couple of them. Nothing major and the benefits of the language will probably outweigh this in the long run, but it was something interesting to note.
    • Thinking in functional style screws with your brain -- it feels really really good, like a hard workout, but it hurts like a hard workout too. I have already adopted lots of functional paradigms in my own programming from previous forays into the FP world (in Lisp, Scheme and Haskell) and I'm a much better programmer for it, but I still need to change my algorithm design paradigms significantly to be really good at using Scala the way it potentially can be used. The lack of FP features is one of my biggest complaints about Java, so this is all welcome frustration.
    • The syntax is close enough to Java but subtly different enough that the syntax screws with your brain too if you have done a lot of Java. I haven't learned to "see" the name:Type parameter syntax without thinking about it yet, for example, so my subconscious keeps thinking of name as Type and Type as name, and I just get confused until my conscious brain talks my subconscious out of it :-) There are a lot of subtle differences between the two languages that have this effect.
    • The language encompasses a huge number of older-but-not-mainstream as well as new (as in modern) language features, absorbing the best parts of a number of languages, including ML, Scheme, Haskell, Erlang and C#, mixed into a Java-like syntax in a (surprisingly) mostly self-consistent way. However to fit all that into the language, one of the most remarkable things that Scala accomplishes is that it deconstructs the Java syntax, builds several layers of abstraction and generalization below them, and then re-builds the syntax on top of these generic layers so that the syntax is not a special case, but rather the application of a general prinicple. For example, a method call a.b(c) and an operator expression (x - y) can only be called in those ways in Java, but in Scala you could write x.-(y) or (a b c) to achieve the same effects, because infix operators are just method calls, and you can define methods using symbols. This deconstruct-generalize-reconstruct pattern is pervasive in the difference between Java and Scala, affecting everything from type inference to the handling of built-in types (int etc.) to even control flow: you can define your own control flow operators (for, while, if, try-catch etc.) -- they are just functions operating on blocks.

    All in all despite the frustrations with inadequate IDE support and some minor Java/Scala impedance mismatch warts, I think my main problem with Scala right now is just unfamiliarity. It takes me 10 times longer to write anything in Scala -- but I write sometimes 10 times less code, so maybe it's worth it. As I get proficient in the language I'm sure I'll become more productive. I just hope I can learn to switch back and forth without too many brain thinkos.  (Similarly, I switched to Dvorak years ago and going back to qwerty just confuses me now :-) )

    UPDATE: Peter 'kovac' Hausel (pk11 on Twitter) pointed out:

    Thanks Peter!

    2010-05-09

    The Internet is the nervous system of the human superorganism

    This video shows the impact of Wikipedia, OLPC and the Internet on schoolkids in remote Peru.  So inspiring!

    Web: La Selva from Web on Vimeo.


    Chris Ball worked on the OLPC Wikipedia activity, and said, "I think working on the Wikipedia activity may be the most important thing I've ever done".

    2010-04-29

    My experience with laser vision correction (PRK)

    Just got PRK laser vision correction

    I decided to get Lasik/PRK after realizing that the amount I have spent on glasses over the years added up to the cost of a Lasik operation.  I also owned a Canon 5D Mark II digital camera -- an awesome light-capturing machine -- and I realized the camera was worth as much as a Lasik operation, but that when I was focused on taking photos of beautiful scenery I totally missed the moment.  It is much more important to see with my eyes than through a camera lens.  So I sold the camera to pay for my laser vision correction operation.

    I decided to write up my experiences in some detail so others have more information in one place than I was able to find when I was investigating getting this operation.

    What is PRK?

    PRK is like Lasik but rather than cutting, separating and peeling back a flap on the front of the cornea and then ablating (removing tissue) underneath the flap as with normal Lasik, with PRK they brush off the 6-cell-thick layer off the front of the cornea and then directly laser the front of the corneal protein.  It's more painful and has a longer recovery time but is an older, often more trusted technique and has numerous advantages.

    I chose PRK over flap-based Lasik for the following reasons:
    • There are fewer problems with dry eyes with PRK compared to Lasik in the 6 months to 1 year after the operation, because there is less nerve damage done with PRK (there is supposedly less nerve damage with PRK than with flap-based Lasik, though I don't know the details).
    • There is less chance of dislodging the flap later in life with PRK, because there is no flap -- with Lasik, the structural integrity of the cornea is never again quite what it was, and even running into a tree branch can sometimes dislodge it.  It's not super-common for this to happen but if it happens it can give you serious vision problems, so the cost of a problem is high.
    • There is less chance of infection with PRK (you can get an infection under the flap with Lasik, which is relatively rare but can be pretty bad).
    • They ablate/remove less of the cornea with PRK than with Lasik, so if you need a touchup later on, you have more corneal tissue to work with.  Touchup operations are free within 6 months with my provider.
    • UPDATE: @GlennHagele stated on Twitter (I read this after my operation): "The first 3 weeks you will wish you had Lasik and then every day thereafter you will be glad you had PRK. http://USAEyes.org "
    The downsides to having any form of laser vision correction at all include:
    • At about age 44 you start to experience presbyopia -- the inability of the eye to focus outside of a limited range -- and by the early 50s this process of deterioration in focusing ability is pretty much complete.  Laser vision correction corrects your vision to see to infinity with the eye's focusing muscles completely relaxed, and you'll eventually need reading glasses to see up close.  I'm 34 though and although I don't need glasses when reading books, I wear them 100% of the time anyway, and I do need them already when using a computer screen.  The way I figure it, I'll get 10 good years of use out of my eyes without needing glasses for anything, and then I'll just need them when reading a computer screen again (which I already do) or a book.
      • The main downsides I see with presbyopia are that I will have to carry around reading glasses with my cellphone, otherwise I won't be able to read text on the cellphone screen (and that I won't be able to see my wife up close!)
    • Increased risk of chronic dry eye problems (I already had some issues with this, PRK/Lasik would make this worse, especially non-PRK Lasik)
    • Risk of haze / halos / starburst patterns around lights at night: your pupil is more dilated at night so the chance of having light refracted from both ablated and unablated regions of the cornea is higher, causing possible halos and other artifacts.  The chance of these side effects is higher with strong prescriptions, -8 diopters or worse.  My prescription is closer to -3 so I don't expect to have problems with this.
    The downsides to PRK vs. Lasik
    • Recovery time (both in terms of pain and quality of vision) is much longer -- 2-6 weeks for PRK rather than 1 day for flap Lasik.
    • Important: you need to take Vitamin C and wear good sunglasses when outside for at least 6 months after the op, especially in the summertime (CONSIDER GETTING THE OPERATION IN THE LATE FALL) -- the new cells that regrow over the cornea are very susceptible to scarring in UV light.
    One big upside of getting laser vision correction
    • I can finally wear REAL SUNGLASSES and still see!  Nobody that hasn't worn glasses for years could appreciate how great this is :-)  I bought polarized prescription sunglasses before and they can be really expensive.  First thing I did after the operation when I could get out and about again was to go look at sunglasses (since I'll need them anyway).

    Consultations are usually FREE

    Because some people are not good candidates for Lasik/PRK, most places that perform laser surgeries will give you a consultation (actually as many visits as necessary) for FREE before you get the actual surgery, to determine whether or not you're a good candidate.  You pay NOTHING until the day you get the surgery.  Personally I went in for about seven visits before my surgery because I had dry eye issues that needed to be treated with drops and monitored to make sure I could qualify for Lasik/PRK.  In the end it was determined I could get either, but PRK would give me fewer ongoing issues with dry eyes, so this was one determining factor.

    Several things can make you a bad candidate for Lasik: overly-dry eyes, overly-wide pupils, overly-thin corneas, etc.  You should be aware that there are some cheap Lasik clinics out there that get as many people through as possible for ridiculously low prices.  Beware of clinics that don't reject people that are not good candidates for Laser vision correction!  This is your vision, don't mess with it.

    Surgeon vs. fellow -- save 50% with a fellow-in-training

    I elected to have the operation performed by a surgical fellow under the direct supervision of a surgeon with 20+ years of experience, rather than having an actual surgeon perform the operation.  Receiving surgery from a fellow reduced the cost of the surgery by 50%, from $6000 for both eyes to $3000 for both (and I further talked them down to $2800 for both).  A fellow has completed an MD, a residency, and was on a fellowship, one step away from becoming a full surgeon, and Lasik technology has become almost risk-free in the last three years, so I figured risk was minimal and cost savings were great.

    I received the operation at Tufts New England Eye Center in Boston based on the fact that they provide free or heavily discounted treatment services to the disabled, first responders and the military (none of which I qualify for, but on principle).  They were very professional and knowledgeable, and did a great job.  Highly recommended.


    The First Week of PRK

    Note that as noted elsewhere in these notes, recovery time for standard Lasik mostly takes place within a day or so, whereas PRK recovery takes longer but is better for your eyes long-term so is worth all this discomfort and blurriness.

    Day 0 -- Thurs -- PRK operation
    • Given 15mg of Valium which made me completely uninhibited and incessantly chatty -- I had the whole operating room laughing constantly :-)
    • You sit under the machine and see a ring-shaped light with a dull red glow in the middle, and a spotty interference pattern from another laser below.
    • Received anesthetic drops, then they taped my eyelids open and then put in a metal clamp to hold the lids open.
    • The surgical fellow used a small brush like a dentist's drill to brush away the cellular layer from the front of the cornea.  Painless but a weird sensation.  The lights swirled as the brush moved the eyeball in fast circular motions.
    • The surgical fellow then used a scraper to create a clean edge at the boundary of where the cells had been removed.  The bigshot surgeon checked through the scope a couple of times between scrapings and pointed out areas where the fellow had missed a couple of small spots with groups of cells.
    • They then started the actual laser surgery directly onto the cornea.  There were something like 239 laser pulses that ablated the surface in random order (to avoid overheating).  A camera looked for saccades (fast movements of the eye) with a frequency of something like 1000Hz and used a pre-stored image of the retina to register the ablation plan correctly to the eye (this is the Allegretto Wavefront laser way of doing registration) -- your eyeballs actually rotate a few degrees when you lie down, and it's almost impossible to stare at one spot for a period of time without eye saccades around even without you being aware of it.
    • The laser zapping process all seemed to happen in about 10 seconds.  Each laser pulse made a clicking/sizzling sound.  There was a vacuum tube by my eye but I could still smell burning protein.
    • They washed the surface of the eye with a lot of fluid then put in a contact lens, then repeated the whole process for the other eye.
    • Everything was pretty cloudy when I stood up but my vision seemed somewhat sharp.  The valium had kicked in with being horizontal for over half an hour, so I was pretty dizzy and needed help walking at first.
    • I had to get a ride home, they wouldn't let me even take a cab because of liability.  I went home, figured I could see relatively sharply in spite of the haze, and the valium had mostly worn off, so I stubbornly drove the 5 mins to the pharmacy to pick up my prescriptions for painkillers and eyedrops.  Took painkillers and slept off the rest of the day.
    Day 1 -- Fri -- day after operation
    • I had been given eye guards to tape over my eyes at night.  I woke up without them on -- found one guard strewn across the bed and the other one nowhere in sight, I still haven't found it to this day so I had to get a replacement :]
    • When I woke up my eyes were sore but it wasn't too bad.  Eyesight was hazy all day, had to squint to do much, and had to scale up font size on computer to huge.
    • Had to visit the eye center for the day-after appointment.  Vision tested as 20/15 (better than 20/20) in one eye and about 20/20 in the other -- sharp but very hazy.
    • Ended up taking painkillers and resting, pretty unproductive day.
    Day 2 -- Sat
    • Probably the worst day for pain.  Felt like someone had poked me in the eye for most of the day.  Wanted to avoid painkillers so I just endured it.  Vision still hazy but usable -- took care of some errands, was able to be out and about for most of the day being productive.
    Day 3 -- Sun
    • Probably the worst day for vision.  Also extreme light sensitivity, couldn't get up for 2-3 hrs because I couldn't open my eyes, even with the blinds shut.   Apparently trauma to the cornea makes the iris muscles spasm in reaction to light, "like getting a charlie horse in your eye".
    • Vision very blurry, borderline dangerous to drive.  Avoided people I knew at Church because I knew I wouldn't be able to tell if they were looking at me or not from more than a foot or so away.
    • Looked like I was looking through a steamy window during daylight hours.
    • Took painkillers and slept off most of the afternoon.
    • Distinct-shaped halos around lights at night, the shape of the wavefront of cells regrowing in towards the middle of my cornea.
    Day 4 -- Mon
    • Couldn't open eyes again for about 3 hours after waking up because of light sensitivity.
    • Very little pain left but couldn't be productive in front of a computer screen.
    • Vision good enough to bike in for a checkup in the afternoon, the doctor wanted to make sure I had no infection.
    Day 5 -- Tues -- contacts removed
    • No real problem with light sensitivity this morning, could tell that cells were almost totally regrown because everything was pretty sharp right when I woke up.
    • Got contacts out in the afternoon. Cells had completely regrown over the cornea (with the characteristic ridge or pileup of cells in the middle that was still scattering light) and looked really good according to the surgeon.
    • Vision tested as pretty good, but not quite 20/20 before removing the lens, but was worse afterwards.  The newly exposed cornea surface was not as smooth as the contact lens surface, because there was a (normal) ridge/pileup of cells where the regrowth coming in from both sides joined in the middle.
    • Eyes were pretty uncomfortable after the lens came out, but only for about 10 minutes, as the new cells were exposed directly to the air and eyelids for the first time.
    • Tried getting computer work done after getting lenses out, but needed eyedrops for dryness every 10 minutes or less, eventually ran out of drops and had to go home.  Still had to scale up text font size to huge.  Squinting constantly to see better gave me a pretty bad headache.
    Day 6 -- Wed
    • Eyes were really sensitive to light again after getting lenses out, had to spend whole morning in bed again.  Couldn't do much productive.  Vision was not bad early in the day but got worse.  Had to drive somewhere anyway.
    • Had moments of near-perfect vision after putting in drops -- amazing to see clearly again after everything being blurry for a week.  However things were still a little fuzzy.
    • The extra dryness experienced yesterday was mostly gone, was able to be relatively productive, only needed drops every 30 mins or so.
    Day 7 -- Thurs
    • Woke up with very dry eyes almost stuck to eyelids.  Light sensitivity first thing in the morning only lasted for a few mins once I put drops in.
    • Vision was very clear the instant I put drops in but got fuzzy within a few minutes of each set of drops.
    • I can finally be productive on my computer again at normal font size, albeit with fuziness, but without squinting.
    • I have been informed I'm totally on-track for having perfect vision restored within 2-6 weeks of the operation, so it will in theory only get better from here.  Vision will usually be best in the morning and dry eye problems will be worse when working at a computer (which I do all day).
    • I expect the log will be boring from this point on so this will be the last entry :-)

    Overall experience

    This all sounds bad, but it's worth it if I never have to worry about wearing glasses again!

    Post-op considerations
    • Important: Need to take Vitamin C and wear good sunglasses when outside for at least 6 months after the op to prevent scar tissue forming on the cornea.
    • You're given painkiller drops right after the operation but you're told not to use them after day 2 as they will slow down the healing process.
    • You have to wear eye shields to bed every night for 2 weeks.
    • No shampooing hair for 3 days to prevent infection; wearing swimming goggles for showering for 2 weeks after the operation, or at least have to wash hair/face outside of shower to keep water and soap out of eyes.
    • No eye makeup for 2 weeks for the ladies, no swimming for 3 weeks, only light exercise and no weightlifting for 3 days, sweatband should be worn during exercise for 2 weeks.
    • Antibiotic drops have to be used for a week or more, steroid drops for 3 weeks, preservative-free artificial tears every hour while awake (as much as every 10 minutes as needed).
    • Taking fish oil and flaxseed oil can alleviate dry eye problems.

    Final thoughts


    WORTH THE PAIN/BLURRINESS: Overall I can tell my vision is going to be great and I would recommend PRK due to reduced risk of complications with PRK, and because of the ability to get a touchup operation as needed.

    LASIK HAS COME A LONG WAY: Lasik is far safer today than it was even three years ago, with fewer chances of side-effects.  I wasn't comfortable with getting it until recently.


    THE BEST LUBRICANT EYE DROPS FOR SENSITIVE EYES
    I have tried about eight different types of eye drops, and all of them but one have given me problems with increased redness and other sensitivity issues.  I now use Tears Naturale Free, they give my by far the fewest problems (my eyes react to many other types of lubricant drops).


    CHEAP GLASSES FOR THE LASER-AVERSE: For those that are not ready to take the plunge to get laser vision correction, I recommend Zenni Optical -- prescription glasses for $8 (both lenses and frames)!  It's a company that operates out of Hong Kong but has an office in California, and brings Asian glasses prices to the US market (finally).

    DONATE YOUR OLD GLASSES: One last comment, I learned the best thing to do with used glasses is to donate them, where they can be taken to clinics in the developing world.  Most eye clinics have a donation bin or you can easily find places online.

    I donated four pairs of used glasses, and the realization that there were millions of kids in developing countries who are hindered in their learning and life progress simply because they can't see actually made me feel rather bad about getting PRK.  I had a "wow the gap between rich and poor is huge" moment when I realized these kids can't afford even basic glasses and yet I just had my cornea perfectly reshaped with a laser!!