[IceRAM]

Hi everyone,

I'm trying to use globalSAN 3.3.0.43 initiator with an OpenSolaris 2008.11 (upgraded to snv_111a - latest) target over 100MBps Ethernet network, no jumbo frames. The curious thing when using this combination is that the write speed tops 1 MB/sec while the read speed tops 11 MB/sec (maximum for 100 MBps Ethernet). I have tried tunning the target in several ways:
* iscsitadm modify target --maxrecv 2097152 tank/test-sparse (setting maxrecv of 2 MB, on a sparse volume iSCSI shared volume)
* iscsitadm modify admin -f enable (enable fast writes)

Even though I've done these 2 changes, the write speed hasn't gone higher than 1 MB/sec. If I start multiple writes (start copying one file, start copying another file etc.), the write speed increases (it's almost 1 MB/sec per "session"). It's like there's a limitation on the write speed per "session". I do not know exactly where this problem occurs, but it is definitely not a CPU usage (~5%) or a HDD (RAIDZ1) problem. Sustained read/writes over SMB for example top 11 MB/sec without a problem.

Does anyone have any idea why would this be happening?

Many thanks.

[wereHamster]

IceRAM, on May 3 2009, 06:39 PM, said:

Hi everyone,

I'm trying to use globalSAN 3.3.0.43 initiator with an OpenSolaris 2008.11 (upgraded to snv_111a - latest) target over 100MBps Ethernet network, no jumbo frames. The curious thing when using this combination is that the write speed tops 1 MB/sec while the read speed tops 11 MB/sec (maximum for 100 MBps Ethernet). I have tried tunning the target in several ways:
* iscsitadm modify target --maxrecv 2097152 tank/test-sparse (setting maxrecv of 2 MB, on a sparse volume iSCSI shared volume)
* iscsitadm modify admin -f enable (enable fast writes)

Even though I've done these 2 changes, the write speed hasn't gone higher than 1 MB/sec. If I start multiple writes (start copying one file, start copying another file etc.), the write speed increases (it's almost 1 MB/sec per "session"). It's like there's a limitation on the write speed per "session". I do not know exactly where this problem occurs, but it is definitely not a CPU usage (~5%) or a HDD (RAIDZ1) problem. Sustained read/writes over SMB for example top 11 MB/sec without a problem.

Let me guess, the target is a ZFS volume, and the underlying storage is a conventional HDD, right? I've had the exact same problem and I was told it's because of the synchronous nature of iSCSI. If you want higher speeds, you'll have to either buy SSDs and put the ZIL there, or disable the ZIL. See also: http://caurea.org/20...eets-zfs-iscsi/

[piersdd]

I too found performance to be unacceptably slow, even after adding a SSD ZIL! The systems are;
Sun Ultra 40 M2 running opensolaris snv_111b. 12GB RAM
6 x Hitachi Ultrastar SATA drives. (As a ZFS stripe of three mirrored pairs)
1 x SLC SSD (connected to same LSI SAS controller). (32GB SSD ie, above recommended maximum size)

Apple Mac Pro, dual 3.2GHz Xeon. 16GB RAM
8 x Hitachi Ultrastar SATA (As a HFS+ stripe of three mirrored pairs, plus an additional mirrored volume)
1 x SLC SSD

both are connected directly via Myri10GBE, running IPv4

before putting a SSD ZIL in the pool
Mac->Sun via CIFS, I got around 140-180MB/sec
Mac->iSCSI before ZIL, I got around 2MB/sec

with the SSD ZIL
Sun->Sun cat /dev/zero > /data/backup/tmp/junk, I get 198MB/sec
Sun->Sun time dd if=/dev/zero of=/dev/zvol/rdsk/data/backup/rawvol, around 7MB/sec, spiking to 100MB/sec
Sun->Sun time dd if=/dev/zero of=/dev/zvol/dsk/data/backup/rawvol, 160-220MB/sec
Mac->Sun via iSCSI, around 10MB/sec

ZIL disabled
Mac->Sun via iSCSI, around 14MB/sec though very bursty.. nothing for 10-20secs, then 2-3 seconds of 200+MB/s

[piersdd]

Having reviewed the evil tuning guide (http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#How_to_Tune_Cache_Sync_Handling_Per_Storage_Device) I figure perhaps OpenSolaris did not optimise the writes by default to the SSD.
Thus, I shall test, per the guide, and advise. Is anyone aware if SSDs are automatically treated differently?

[russm]

according to a post on the OpenSolaris forums, this may be caused by the nagle algorithm on your clients trying to be efficient but actually killing iSCSI performance... for instructions on disabling this on Leopard, head to (of all places) the World of Warcraft forums...

and seriously, don't disable the ZIL...

[SNSryan]

Hello!

We've just announced a beta version for globalSAN v4.0. This version addresses certain performance issues with 3rd party targets. We'd love to hear if this v4.0 beta helps with performance on this target. Go here for additional details and download link. I look forward to everyones feedback.

-ryan

[amoeba]

Thanks for the software. I am currently testing beta 204 after 197 (OS X 10.6.2 32bit) and basically it works out of the box, but ... speed is incredibly slow, i.e. sluggish.

On the target side I am using opensolaris snv_131 with COMSTAR target architecture residing on a SAS/RAIDZ pool. I also experience extreme bursts followed by long periods of inactivity, typically first maxing out the available connection bandwith (untypically high rates around 100-90MB/s on a single GB connection). What then happens (same in finder):

$ time dd if=/dev/zero of=/Volumes/tm_iscsi/testfile bs=1048576k count=4
4+0 records in
4+0 records out
4294967296 bytes transferred in 379.311404 secs (11323064 bytes/sec)

real	6m19.364s
user	0m0.000s
sys	0m3.225s

Does anyone have a clue what might be the reason for this? Can I supply any other logs which might be helpful?
Thanks!

[Eric Newbauer]

This might be of some help:

https://opensolaris....essageID=388492

For anyone having very low average I/O with globalSAN + OpenSolaris + ZFS: Are you seeing acceptable performance with the Microsoft Initiator?

[dataflash]

Eric Newbauer, on 29 January 2010 - 02:04 AM, said:

This might be of some help:

https://opensolaris....essageID=388492

For anyone having very low average I/O with globalSAN + OpenSolaris + ZFS: Are you seeing acceptable performance with the Microsoft Initiator?

1 MB/s using globalSan 3.3.0.43 on 10.5.8, both on an old PowerBook G4/1500 and on a MacPro 8x2.8/10Gb of ram. The iSCSI target is openSolaris 2009.06 on an identical MacPro.
No good news also on Microsoft side (initiator version 2.08)...4-5 MB/s, nothing more.
Tried disabling tcp delayed ack with no results.

[amoeba]

Eric Newbauer, on 29 January 2010 - 02:04 AM, said:

This might be of some help:
https://opensolaris....essageID=388492
[...]

great, thank you very much for the hint! after doing
[edit: partial success only, this only works on the command line, in the finder the behaviour remains the same ...]

stmfadm list-lu -v
stmfadm modify-lu -p wcd=false <LUN NAME>

the results rose to >60MB/s:

time dd if=/dev/zero of=/Volumes/tm_iscsi/testfile bs=1048576k count=4
4+0 records in
4+0 records out
4294967296 bytes transferred in 65.026626 secs (66049364 bytes/sec)

real	1m5.154s
user	0m0.000s
sys	0m3.657s

p.s. can't try out the ms initiator here, sorry.