400GB software RAID-5 in under $350.

1) shopping

Not Maxtor (see diagram 5). 400GB disks are too expensive per byte. So Seagate or Western Digital. USN shop.

2) installation

3) partitioning  (for each ad1, ad2, ad3)

fdisk -BI ad1			ignore "invalid mbr"
disklabel -wB ad1s1 auto
disklabel -e ad1s1		should look like this: 

   8 partitions:
   #        size   offset    fstype   [fsize bsize bps/cpg]
     c: 390721905        0    unused        0     0        # (Cyl.    0 - 387620*)
     h: 390721905        0     vinum                       # (Cyl.    0 - 387620*)

4) Vinum config:

drive a device /dev/ad1s1h
drive b device /dev/ad2s1h
drive c device /dev/ad3s1h
volume raid5
  plex org raid5 512k
    sd length 0 drive a
    sd length 0 drive b
    sd length 0 drive c

vinum create vinum.conf
vinum init raid5.p0		wait for "up", kick it again if it get stuck
echo 'start_vinum="YES"' >>/etc/rc.conf

newfs /dev/vinum/raid5
tunefs -n enable /dev/vinum/raid5
echo '/dev/vinum/raid5 /.1 ufs rw,nodev,nosuid,userquota,groupquota 2 2' >>/etc/fstab
mount -a
quotaon -a
quotacheck /.1

5) copying hundreds of CDs and DVDs - TBD, duh.

6) the funny part - testing.  power off ad3, master ad2 also goes off:

Mar 14 21:44:38 heart /kernel: ad2s1h: hard error reading fsbn 563967633 of 281983785-281983912 (ad2s1 bn 563967633; cn 559491 tn 11 sn 12) trying PIO mode
Mar 14 21:44:38 heart /kernel: ad2: DMA problem fallback to PIO mode
Mar 14 21:44:38 heart /kernel: ad2: read interrupt arrived earlyad2: read error detected (too) late
Mar 14 21:44:38 heart /kernel: vinum: raid5.p0.s1 is crashed by force
Mar 14 21:44:38 heart /kernel: vinum: raid5.p0 is degraded
Mar 14 21:44:48 heart /kernel: ad3: READ command timeout tag=0 serv=0 - resetting
Mar 14 21:44:48 heart /kernel: ata1: resetting devices .. 
Mar 14 21:44:48 heart /kernel: ad2: removed from configuration
Mar 14 21:44:48 heart /kernel: ad3: removed from configuration
Mar 14 21:44:48 heart /kernel: vinum: raid5.p0.s2 is crashed by force
...
Mar 14 21:44:49 heart /kernel: vinum: raid5.p0.s2 is stale by force
Mar 14 21:44:49 heart /kernel: vinum: raid5.p0 is faulty
Mar 14 21:44:49 heart /kernel: vinum: raid5 is down
...
system hangs here (well, duh, it's ata after all or may be it's too hard for vinum), after reboot

# vinum ls raid5.p0.s0
S raid5.p0.s0           State: R 0%     PO:        0  B Size:        186 GB
                        *** Start raid5.p0.s0 with 'start' command ***

# vinum start -w raid5.p0.s0
Can't start raid5.p0.s0: Input/output error (5)			no idea what it means

# vinum ls raid5.p0.s0
S raid5.p0.s0           State: R 0%     PO:        0  B Size:        186 GB
                        *** Revive process for raid5.p0.s0 has died ***

# vinum start -w raid5.p0.s0
(hangs forever here)

(from another shell)
# ps -ax|grep vinum
   15  ??  DLs    0:00.00 vinum: vinum daemon (vinum)
  206  p0  D+     0:00.01 vinum start -w raid5.p0.s0
  207  p0  Z+     0:00.00  (vinum)				f*cking great
  
# vinum ls raid5.p0.s0
S raid5.p0.s0           State: R 0%     PO:        0  B Size:        186 GB
                        *** Revive of raid5.p0.s0 has stalled ***

# vinum setstate up raid5.p0.s0					ok now
# vinum setstate up raid5.p0.s1
# vinum setstate up raid5.p0.s2
# vinum start raid5.p0   		well, more or less, it wasn't exactly like this
# vinum rebuildparity -vV raid5.p0
Parity incorrect at offset 0x2ac000
Rebuilding at 2816 kB (0%)    Parity incorrect at offset 0x2c8000
...
and here it will another few hours to build it up

Preliminary conclusion - should be different channels.

To be continued.