Archive for the 'Technology' Tag

USB 3.0 works under Linux

I decided that I needed a real backup solution, even though I have a RAID 5 for file storage in my workstation; maintaining a backup of a 2TB array is a pain in the ass if all you have is blank DVDs.

So, I purchased a Vantec NexStar 3 SuperSpeed (NST-380S3) enclosure, a Samsung EcoGreen F3EG 2TB 5400rpm (HD203WI) drive, and a USB 3 PCI-E controller.

It seems the only shipping USB host controllers the moment all use NEC’s USB 3.0 chip, and almost all the PCI-E boards look alike. They all seem to run in the $25-45 range. The great part is Linux supports NEC’s controller as of 2.6.31. The controller worked with no configuration soon as I put the card in.

I chose that specific Samsung drive because it seems to be the only sane 5400 rpm 2TB drive out there. The only other choices were Seagate’s new 5900 rpm drives (which, according to independent reviews on Newegg and enthusiast forums have an unacceptably high failure rate, very unusual of Seagate), and Western Digital’s Caviar Greens (which are 5400 rpm, but suffer from obsessive head parking which is apparently leading to premature drive failure).

Several reviews peg the HD203WI at an average of 90mb/sec writes for sequential writing, or about 2-3x the speed of USB 2.0.

mkfs.ext4 took 7:44 minutes to create the file system (while iotop confirmed it was doing in excess of 100mb/sec writes for much of the process), and hdparm -t /dev/sdx also indicates the drive in this enclosure can push 100mb/sec.

After writing to the drive for an hour straight, the enclosure is warm but not hot, and after removing the drive from the enclosure, the drive itself is warm; this is compared to the Seagate 7200.12s in my RAID 5 array which could burn you at this point.

Many drives fail in enclosures because they overheat; I don’t think this will happen due to Vantec’s thick aluminum design in the NexStar series enclosures, and the fact that the HW203WI has low power usage.

After formatting with ext4, the file system uses 29GB out of 1.82TB total. Its kind of funny when I’ve owned drives smaller than the space consumed by an empty file system.

I’m rather happy with my purchases overall.

Approximate Youtube Bitrates

I’ve been wondering what bitrates Youtube produces on files, but they don’t upfront say.

New videos are encoded in eight formats. However, due to bug in Youtube, some 24 fps videos (such as those from film sources) will have duplicate frames inserted to make them 30 fps, causing a very noticeable jitter approximately twice a second.

Format Video Codec Audio Codec Container
37 H.264 1920×1080 24/30 fps AAC 44.1khz Stereo mp4
22 H.264 1280×720 24/30 fps AAC 44.1khz Stereo mp4
35 H.264 854×480 24/30 fps AAC 44.1khz Stereo flv
34 H.264 640×480 24/30 fps AAC 44.1khz Stereo flv
18 H.264, 480×360 24/30 fps AAC 44.1khz Stereo mp4
5 Sorenson Spark, 320×240 24/30 fps MP3 22khz Stereo flv
17 MPEG-4 ASP, 12 fps, black bordered to fit 176×144 frame AAC 22khz Mono mp4
13 H.263+, 15 fps, stretched to full frame 176×144 ignoring source aspect ratio AMR 8khz Mono 3gp

Note: This does not include WebM videos yet as the support is still experimental, and Youtube is not yet encoding videos in 1080p, only 720p (format 45) and 480p (format 43).

Now lets see how a couple high quality videos fair on Youtube.

Format Resolution Video and audio bitrate in kbit/sec
The Dark Knight Trailer 3 1080p, using the Apple version. 2:30 long. H.264, 6ch 48khz AAC audio, 24 fps. Youtube encoded this as a 30 fps video.
Original 1920×816 10518 260
37 1920×816 3427 108.8
22 1280×544 1998 108.8
35 Missing on Youtube

34 640×272 517 95
18 480×204 500 108.5
5 320×136 257 64
17 176×144 55.3 27
13 176×144 55.6 13
Avatar Trailer 1080p, using the Apple version. 3:29 long. H.264, stereo 44.1khz AAC audio, 24 fps.
Original 1920×800 9726 99
37 1920×800 3502 126
22 1280×534 2003 126
35 854×356 806 103.84
34 640×266 554 103.81
18 480×200 486 103.82
5 400×166 255 59
17 176×144 55 28
13 176×144 54 13
Big Buck Bunny 1080p, using the Blender Foundation‘s original version. 9:57 long. Theora, stereo 48khz Vorbis audio, 24 fps.
Original 1920×1080 11902 175
37 1920×1080 3531 125
22 1280×720 2020 125
35 854×480 990 107.9
34 640×360 494 108.02
18 480×270 435 108.03
5 400×226 250 59
17 176×144 55 30
13 176×144 49 12

With these 3 popular HD videos, its easy to tell what sort of bitrate Youtube tries to hit.

Format Approximate bitrate target (video and audio)
37 3.75mbit/sec
22 2.25mbit/sec
35 1.25mbit/sec
34 768kbit/sec
18 768kbit/sec
5 384kbit/sec
17 100kbit/sec
13 75kbit/sec

Solid state society: The future of common data storage

Fifty-one years ago, IBM did something amazing, something that changed the world and kick-started the computing revolution twenty years before Intel and Apple and Microsoft and everyone else declared they were open for business: IBM invented the hard drive.

A monster of a machine, a behemoth, one ton of spinning metal the size of a fridge held exactly five megabytes via 50 two foot platters and a bunch of controller hardware and buffer memory. This hard drive was the first of it’s kind, and helped spawn an entire industry of data storage; not only was it faster and easier to use and maintain compared to tape media, it was also expensive and only a few companies could afford this.

The technology over the next few years shrank and increased in performance, and stories of “wash machines” dancing across the data center were well known. More and more companies started buying them to replace or supplement their tape drives, and eventually tape died out in the commercial sector.

Eventually, the three or four home computing revolutions come and go, and the two portable device revolutions come and go. Wash machines become small external units, those external drives become internal (5.25″ full height), and then they become smaller (3.5″) and smaller (2.5″) and smaller (1.8″) yet. Megabytes become gigabytes become tens and hundreds of gigabytes and finally, as of a few months ago, terabytes.

All of this technology ultimately works the same way: spinning platters with magnetic heads reading what an IBM engineer once named “magnetic milkshake.” The one single major flaw in this design is that anything that moves will eventually break down. Spinning drives slower won’t decrease the wear and tear, and neither will cooling them; and new bearing designs? They decrease noise and some wear and tear, but do not prevent mechanical failure.

We’ve invented new technologies, such as redundant arrays of inexpensive disks (RAID) to both increase performance and decrease the chances of mechanical failure eating your data. A suitably sized RAID 6 array can have two drive failures before you risk data loss. An array of, say, six to ten drives for such an array is also huge and outside the realm of most people; and I haven’t seen Apple issue iPods with RAID arrays yet.

In addition to all of this, the magnetic heads have to move across the platter to read and write specific areas, which increase the time it takes to read random data (sequentially read data suffers from this less). If mechanical failure was the major issue of this design, seek times is the secondary issue.

In 1984, a Dr. Fujio Masuoka invented flash memory: a non-volatile memory that can be used as data storage in the same way you’d use tape or hard drives, and flash has no moving parts nor does it use large amounts of power like hard drives do because of spinning platters. You see flash everywhere now, in your cell phones, in your digital cameras, in your hand held game systems, and also in your Wiis. We call drives built out of this technology: solid state drives.

Laptops are now the key target: laptops never have enough power, and battery technology is not keeping pace with our advancements with other technology, and until Santa Rosa more than 3 hour battery life under normal conditions on most laptops was impossible… now it’s simply medium difficulty. Flash technology now has gotten very interesting due to the fact everyone from laptop manufacturers to silent computing aficionados to even the enterprise sector wants flash tech to replace their spinning milkshakes.