@m3x

I can trace the error to an IO completion queue timeout.

Completion queue timeout is triggered by a violation of the maximum timeout (reported by the NVMe drive itself) or by a missing interrupt.

I do not have a sanity check on this timeout value. But maybe I should add it.

Weird because the whole discovery of the NVMe drive works. And the Admin queue is the same construction which is depending on the same timeout value and same interrupt.

(If someone can tell me how MSI or MSI-X works on AmigaOS4 then I can differentiate with interrupt vectors)

I will try my NVMe drive in my SAM440 and see if I can reproduce the issue.

@sailor

There are a lot of PCIe expanders with switch available. Just not cheap.

@Gregor

Interesting. I've read that the SAM460 was capable of booting Linux and Amigaos4 from microSD. But I didn't know that the X5000 was capable of doing to same for at least Linux. Would be nice if it can also boot the Amigaos4 kernel with modules from microSD.

@sailor @gregor

I don't know about the X1000 but the X5000 certainly doesn't support bifurcation. And you can also not freely assign transceiver lanes to PCIe controllers. You can only choose from predefined configurations. But remember that except the x16 slot, all other slots are behind a PCIe switch. So you don't want to change anything for the P5020 anyways. The X5000 PCIe switch might be more flexible. But I have to look into that.

There are dual NVMe cards available with an embedded PCIe switch. These cards come in two flavors as far as I can tell. A cheap 8x card with just a three port x1 switch (why oh why?). But you have to physically modify the X5000 PCIe connector to fit them.
And an expensive x4 card with a three port x4 switch.
The second one is on my radar. But at EUR 129 not very cheap compared to the price of the NVMe drives themselves.

For now my driver supports only 1 unit. This will change soonish after I have a second NVMe SSD in my X5k.

I must say that I am a bit surprised that there is already the need for using more than one NVMe SSD. Maybe next time I should start a survey first.

It is not possible to boot from NVMe at the moment, so the dual NVMe use case for amigaos and Linux is unclear to me.

@sailor
Currently my driver cannot saturate a PCIe2.0 x1 slot anyways so I wouldn't be too concerned about the need for a x4 slot. Unless you have a X1000. Because those lanes run at only PCIe1.0 speed.

@TearsOfMe
Quote:

Found the issue. A small sequence error in my cleanup routine.

@daveyw

Yes, I can imagine.
I don't know the situation on your country but it seems a trend for big tech only.
And 50 in the tech business is still young. Lot's of companies want the experience that you simply cannot learn at school.

@kas1e

Be careful what you wish for. Cloning me 4 times means also 4 times more cynical comments and complaints about missing documentation.

Large speedups are only gained when your daily usage is copying large files.

Speed difference in copying my 800MB sys: contents between ram and SSD is about ~35% faster compared to SATA at the moment.

I haven't done any speed comparisons on how fast games and applications start. So I do not know if there's really much of a difference in that area.

My primary goal was to make sure that we can use NVMe once SATA becomes rare.

Ok, maybe my real primary goal was to make the X5000 PCIe x4 slot useful for anything (Don't use Linux or Morphos so I don't need a second gfx card). Even bought a beautiful new housing to be able to rotate my gfx card out of the box. And this was inspired by your video about your new SAM460.

@Primax

If you want open source drivers then you should check out Linux or maybe AROS. According to user o1i on amiga-news.de, there should be something in development for AROS.

@TearsOfMe

Thanks for the feedback. Will solve that issue in the next release.

@daveyw

And why would you be redundant?

Yes, a newer NGFilesystem than shipped with Final Edition. Rock solid so far.

@virgola

Then make sure that you fork a NVMe capable u-boot

Some quick benchmarks:

p50x0SATA.device: Samsung 840 SSD 120GB

Copy 1GB file from RAM: to NGFS/01: 51.5MB/s (20.4 sec)
Copy 1GB file from NGFS/01 to RAM : 98.9MB/s (10.6 sec)

nvme.device: Samsung 970 EVO SSD 1TB

Copy 1GB file from RAM: to NGFS/01: 129.5MB/s (8.1 sec)
Copy 1GB file from NGFS/01 to RAM : 152.0MB/s (6.9 sec)

Copy 1GB file from RAM: to SFS/02 : 327.7MB/s (3.2 sec)
Copy 1GB file from SFS/02 to RAM : 233.0MB/s (4.5 sec)

Now don't start using SFS/02 just yet because NGFS is a lot faster with small files. And then the speed difference between SATA and NVMe is a lot smaller (~25 - 50%).

Don't bother with diskspeed. This benchmark applications insists that my SATA drive is capable of >400MB/s. So it must be measuring some kind of cache memory.

@sailor

Ow, 1.0 only. Even then, the difference isn't large. When I didn't install my WD Black SN770 properly, it fell back to 1.0 x1. Only peak performance with large files on SFS2 suffered from it.

It should always be possible to use an active PCIe expander to create additional PCIe slots. But I don't k ow the limitations of CFE and u-boot in this respect.

@sailor

I wouldn't be surprise if it is faster. The PCIe slots are directly attached to the processor and we know that memory copies and PCIe performance on gfx cards are faster then the X5000.

@sailor

It's in the upload queue.

First public release of the NVMe driver has been uploaded to os4depot.

From the readme:

What is NVMe:
NVMe is short for Non-Volatile Memory express. It's basically a transfer protocol for attachment of mass storage to a computer.
Unlike SATA, NVMe is specifically designed for solid state flash memory. There is no separate storage controller. Instead the NVMe SSD is directly connected to the PCIe bus through an M.2 slot.
The PCIe interface and NVMe protocol offer bandwidth and "IOs per seconds" which go way beyond anything that SATA can deliver. This high bandwidth is achieved because NVMe drives connect to a M.2 slots with 4 PCIe lanes. At the time of writing, the first PCIe 5.0 drives are emerging. These drives have a maximum interface speed of 16Gbyte/s while the actual peak performance is already approaching 10GByte/s.
Eventually, NVMe SSDs will replace SATA SSDs as the new industry standard. But since modern notebooks and PCs are not shipped with CD/DVD players (modern PC cases don't even have an external 5.25" bay anymore). And NVMe SSDs are rapidly becoming larger and cheaper. This will happen sooner rather then later.

How to attach NVMe to a NG system:
None of the current NG systems offer a M.2 slot directly on the mainboard. But since an NVMe M.2 slot is basically a PCIe x4 slot with a different form-factor, we can use a widely available passive PCIeM.2 adapter card.
These adapter cards are available with a PCIe x1 or PCIe x4 connector. Don't use the adapter cards with multiple M.2 slots because they will not work.

The X5000 has both x1 slots and a x4 slot available but there are some drawbacks:
- The x4 slot is blocked by a double slot GFX card. But with a PCIe extender cable for either the PCIe x4 slot or PCIe graphics cards, this slot becomes accessible.
- All PCIe slots, except the x16 Graphics card slot, are connected behind a PCIe switch. Which means additional latency.
- The PCIe slots are limited to version 2.0. This means an interface speed of "only" 500MB/s (x1) or 2000MB/s (x4). But either way this is still faster then the X5000 SATA2.0 ports. (300MB/s).

Which NVMe SSDs work with this driver:
This driver should work with any NVMe drive. It was developed to be compliant with the latest NVMe specification (2.0c).
A lot optional features are not implemented yet. Like the Host Memory Buffer for cacheless drives. But given the speed of our memory subsystem, I doubt that this will lead to any noticeable performance increase.
For now, the driver has been tested on a X5000 and in the x4 slot with the following drives:
- Samsung 970 EVO 1TB
- WD Black SN770 1TB

-----------------------------------------------------------------------
Installation:

Install the NVME hardware before adding the driver to the kicklayout

Copy "nvme.device" to sys:kickstart/ and add "MODULE Kickstart/nvme.device" to your sys:devs/kicklayout file.

Alternatively you can copy nvme.device" to DEVS: and mount your partitions manually with sys:system/Mounter. (Don't forget to set the tooltypes)

-----------------------------------------------------------------------
FAQ: (Foreseeable asked questions; This is a first release afterall )

Q: Yessss!! I will get >Gigabyte/seconds transfer speeds, right?
A: Erm, no. NVMe is designed for multithreaded and streaming access. Streaming as in: It takes some overhead to setup a transfer but then we get this train moving.
You will most likely use NGFS. This filesystem is single threaded and limits the maximum transfer internally to 32kbyte-128kb. The overhead needed to start those small transfers will kill speed in a single threaded environment.
The Amiga's slow message port system and the fact that it was decided that >AmigaOS3.9 drivers have to do gatekeeping for misaligned transfers is also not helping. (to say the least).
In short: Speed will be higher then a SATA drive with the same filesystem. But no miracles.

Q: So how fast is it?
A: With NGFS, barely faster than a SATA drive. SFS2 uses a maximum transfer size of 16MByte. I was able to measure speeds up to 384 MB/s while transfering a 1000MByte file between ram and a SFS2 partition. This beats SATA speeds but nowhere near its potential.
There is room for improvement but don't expect miracles.

Q: I don't believe you, I will play in media toolbox with blocksize, buffers, Maxtransfer and the Mask. Or?
A: Nope, these settings are completely ignored by PPC filesystems. So don't bother changing them. Apparently they are still there for legacy reasons only. Everything is decided by the filesystems for you.

Q: Wait a minute. Weren't modern SSD like NVMe SSDs supposed to drop 512 byte blocksizes and require 4096 instead?
A: That's an urban myth. NVMe drives require at least 4k host memory pages. The drive itself can support up to 64 different block sizes. The drive will tell which blocksize will give the best speed result. But so far, my Samsung 970 EVO supports 512 bytes only. The more modern WD black SN770 support 512 and 4k as well but is formatted as 512 bytes as factory default. Anyways, my driver will report back the formatted LBA to the filesystem.
In future updates I might provide a tool which allows you to change blocksize and reformat the drive accordingly. (Notice the "might")

Q: Ok, that is all disappointing. But I will be able to boot from this drive, right?
A: Wrong again. Currently, none of the NG systems support booting from NVMe drives. U-boot itself supports NVMe since 2017 with three isolated source files but I was told that it will take a complete rewrite of the X5000 U-boot to add support. In other words. Never.

Q: Then what is the purpose of this driver?
A: Next time, I will ask myself this question before I start the development.

Todo:

A lot. Like:
- A monitoring and control tool
- HMB for cacheless drives
- Support for more than one drive.
- .....

But I am not in a hurry.

@joerg

Let's assume that you're telling the truth that is was the "official" direction to solve alignment issues at the destination (drivers) instead of the source (OS4 applications). Then where can I find this information as mortal being? And what more undocumented skeletons are there waiting for me in the closet? Or are non-os4 development team members not entitled to this information?

And why is there still the option to set those masks in media toolbox of OS4.1FE when it was "officially" decided to ignore them for >os3.9 anyways? Or did the OS4 team envision that the future for OS4 was to use old 68k mass storage drivers?

And why is Sfs2 exposing this issue where ngfs isn't? Or is Sfs2 this fast because it simply passes on the problems where eg. ngfs doesn't?

Anyways, I cannot take responsibility for a device driver when I do not know which design rules apply.

@joerg

Ok, so the rules are: There are no rules in the Amiga jungle.

Mask or other properties are ignored and everybody has to plan for the worst.
I guess this is the result of the demise of commodore and lack of documentation in general. Nobody takes the lead in "how things shall be done" and therefore it's free for all.

Unfortunately this also means that it makes no sense to introduce new technologies to the amiga. Because any speed advantages that this new technology might bring is negated by the necessary gate keeping overhead on top of other speed limiting constructions of AmigaOS4.

If this sounds harsh then please note that it's nothing personal. Just me expressing my disappointement in general for wasting 2.5 months on a new device driver which was supposed to bring new storage technology to Amigaos4. Only to find out that the actual speed will be nowhere near its potential. Just marginally faster than SATA.

@daveyw

CPU docky in ztools as well.

I've encountered an issue when I want to unpack an LHA archive from an SFS02 partition to anywhere.
While doing so I've noticed that SFS2 sometimes request data from the block device (SSD) to a non-32bit aligned memory buffer address. For example a cmd_read with lenght 18532 (=26x 512 bytes blocksize) to memory address 0x60C88FA5. This is a byte aligned address. It is at this point that I get an unpack checksum error. The archive itself is ok. I can copy it from the SFS2 partition to RAM and it unpacks just fine from there.
I have checked Media Toolbox but the Mask is set to 7FFFFFFC for this SFS2 partition. This should be "Any memory - 32bits aligned".

Does SFS2 ignore the Mask setting on purpose? Or is this a bug in SFS2?
Anyways, my block device requires at least 32bit aligned DMA accesses. So is there a way to force SFS2 accesses to be at least 32bit aligned?

I only see this issue with SFS2. NGFS for example seems to align the buffers to the blocksize. Unpacking from a ngfs partition on the same block device works just fine. This seems to rule out LHA/xadmaster itself.

I could work around it to detect the non-aligned transfer and move the read/write contents to a 32bit aligned intermediate buffer. But I like to avoid this because it will result in a performance penalty.

SFS version string: SmartFilesystem 1.293 (19-11-2015) AmigaOS4 PPC A-EON Technology Ltd