The sensor and motor pieces have a more polished feel about them. They now use a 6-wire pseudo RJ-45 jack instead of a wired up brick. In the original, the sensor/motor pieces looked like regular Lego pieces with a special features sticking out one end, and now they feel like special feature devices that attach to your Legos. They feel more robust, but only time will tell. There are now 7 control pieces where as the original only had 5.
The new motors can be controlled down to the 1 degree level, but the precision seems to depend on how fast you run the motor. The coolest part is you can now read the motor's rotation, so they can act as rotation sensors too. The new motors contain gears within that gear down the rotational speed. Also they now terminate in a rotating hole, where as the original terminated in a rotating shaft. These two slight changes make a huge difference in the construction complexity near your motors. There are mow three motors, which is also a big improvement, but their dual use means you may not be able to use all 3 as motors.
The ultra sonic sensor is the coolest. It can read distance down to the 1cm level, and seems very accurate. It also looks very cool, like a pair of eyes. The sound sensor registers decibels, maxing out at to 100. I don't think it is an accurate measure of decibels, but relative volume is generally the important thing anyway. The touch sensor is tons better than the original, with a larger touch area that sticks out further. I've not worked with the light sensor as much yet, but it seems to work well, and is comparable to the sensor in the original set.
The new software is a great improvement. With the old package I felt like I had outgrown it in 15 minutes. This new package allows you to build a modular flow program as the old one did, but it also lets you get down to the logic and signal level. Your complex programs feel more like schematics than flow charts. It doesn't play annoying sounds on startup The help system is well done and actually helpful. There is more to do out-of-the-box before resorting to overwriting the firmware with a community-built OS. Also, the software says it requires XP SP 2 or higher, but it seems to run OK on my Windows 2000 box, having crashed once.
The new RCX unit is comparable in size and weight to the original. It feels more like a specialized device with a few connectors for attaching to Legos, where the original felt like a big Lego piece with some buttons. The buttons equate to left, right, select, and escape, and the hierarchical built-in interface is easy to navigate. It has a graphical mono LCD screen. You can download images and WAV files to the unit to use in your programs.
USB is so much easier to set up and use than the infrared COM port. I've not used the BlueTooth interface, but it also appears to be more robust and an easier setup.
The original version relied on the long black beams as the staple construction piece, which included both traditional Lego connectors on the top and bottom, and holes on the side for the MindStorms connectors. The new NXT system has almost entirely done away with the traditional Lego connectors, and now uses rounded end beams with MindStorm-style holes only. This may seem at first like a terrible decision, but it works very well in practice. You can still connect to your basic Lego set when building a base structure, and you can build a more complex structure with the tighter MindStorms pieces. This is a great maturing of the product line.
There are other examples of this maturing - There are few wheels, no more rubber bands, and fewer gears. There are many more connector pieces of various new designs, and new ball joint pieces. Overall there are fewer eye candy pieces, but more useful pieces.
No, the old and new MindStorms RCX and motor/sensor pieces are not compatible, but I don't think I'd want them to be. The new version is so much of a leap forward, the old technology would be holding
The packaging is inferior. I wouldn't mention this for most products, but I store my old MindStorms set in the original box. It had compartments to easily sort the pieces, but the new one is just a flimsy box with two cardbord boxes glued to the bottom to hold the NXT pieces. Expect to find a different solution for storing your NXT pieces. I'm switching to old Tupperware containers, and mixing my sets together.