x1000 onboard network opensource driver in progress

@kas1e

I tried to download 215 MB file and it got to 57% then the whole machine locked up.

@kas1e

This NIC sounds very unusual. I think you mentioned that there is no publicly available documentation? Is the Linux driver in the main kernel source, and what's it called?

Quote:

Yes, to have compatibility with OS3/AROS/MorphOS. Also, I don't think I was aware of an alternative approach when I first wrote those drivers.

Quote:

AFAIR you're only allowed to call system functions that explicitly say they are interrupt-safe (but I could well be wrong). Is there dynamic information in the config registers that can't be cached during initialisation?

BTW I usually use a local FTP server for network testing. Ping in flood mode can be useful for stress testing too, but unfortunately Roadshow's ping command doesn't allow that.

@TSK
Quote:

Yeah i got strange lockup few times too but weren't sure if it me only or not. I have nothing on serial at this moment :( same for you ? Through need to add i lockup 2 times when even wasn't under heavy downloading and wasn't sure if it about network at all ..

Now tried that : reboot, firstly download from os4depot 186mb file : no lockup. then download 400mb file from same os4depot and then another on 190mb (so in summary i download ~1.1GB : no lockup :( so it's _not_ amount of transferred data per se..

But then os4depot currently gives me 1.1mb/s , maybe it depends on speed of transfer, how fast your downloads were ?


@ncafferkey
Quote:

Yeah its just in linux source (see links in the post #4). Their code is real documentation : for first linux driver written by pa-semi itself, and for second their linux driver works (so no hardware to blame). And i just almost fully mimic what they do, but for os4, with some small differences one of which is they do install 2 interrupts for both RX and TX dma channels, i only install for RX one currently, and for TX one just do "fire-and-forget" with small poll check, but will see if it need it to change at all.

The plan also to first deal with issues, and then to check if it worth to do what JKruger did for a12222/x5000 eth drivers : replace copy functions to inline assembly ones for speed purposes.

ps. is the batter in x1000 are 3v one right ? And i read somewhere that better to go with BR2032 than CR20232 , like BR2032 better for long run

I've also got that system freeze after a few seconds/minutes of use. Nothing on serial.

@Petrol
After seconds ? I.e. right after startup + doing something ? And how your speed are (i mean, can you try do dl something with WGET for example). Maybe speed may have impact on if you on the faster side ..

I've just tried download a 10MB file from speedtest.
The download completed, with a dl average of 1205kb/s.
The system froze just after the completion.
Some times I can do more speed test before the system froze.

Don't now if it is linked, as I don't check on real time, but on serial I've got 4 lines:
[pa6t_eth] ProcessRX: CopyToBuff failed

I'll do some more test later.

@Petrol
Few hours ago i were able to download 1.1gb of data differently from os4depot.net. After few hours (and hard reboot for this time), i tried to dl some minnt iso, and at first i lockup just right when go to the minnt site, then reboot, and second time i were able to just download 1% of iso before lockup. So really random. And i have nothing on serial sadly :( Can you share full serial log after [pa6t_eth] Init: pa6t_eth.device 1.0 (08.03.2026) line, till end (right till the freeze)?

And you test with browser, right ? I think we need to concentrate on wget download firstly, to have full picture. From my side i will setup local server so to see maximum speed of driver, + wget will rule out browser's issues.

@Petrol
Ok wget tests from local network. Tried on some myst.mdf file placed on win11 , which is 1.8GB and download it to the NIL on x1000, i.e.:

wget -O NIL: http://192.168.0.144:100/myst.mdf

Also disconnected internet completely, just local cable to switch and from switch to another computer. Speed is 11.3mb/s stable, sometime drops to 8-9mb/s for a 2-3 second, but rarely, most of time just 11.2/11.3. not sure if it enough for us.. cpu loading in process of download 73% (maybe due to wget itself, dunno at moment).

Now, download 3 times without reboot this file. no lockup. Then tried while download running other apps, no lockup. Damn!

Then i simple running 5 the same instances of the wget (yeah they all reduced ~to 2mb/s) no lockup ! All downloads fine !


So that left me with suggestion that when we run browser on top, it just doing lot more than just wget , such memory/cache/etc/etc/ involved much more and somewhere happens nasty things and lockup.

In other words what we need to understand now : if we run wget and download anything from anywhere (local and/or internet), but not running browser and not surf the net : will we lockup or not ? Then , same with browser on top, And step by step we will find the pattern for sure.

@kas1e

Quote:

Yes, 3V. Personally I find CR2032 is fine, as it lasted several years in my X1000. It didn't give problems until it got extremely low (1V).

@kas1e

Whevener I try to open a website with Odyssey, the entire system freezes a few second after having loaded some bytes :(

And BTW : thanks a lot for trying to get the PA6T Ethernet device working !

@kas1e
Quote:
Accodring to https://en.wikipedia.org/wiki/Button_cell The BR has less self discharge (that probably does not matter as it's powering the nvram so not just self discharge) and has less voltage loss over time so it may avoid problems with lower voltage as it depletes but as CR is more common it's probably easier to get and should last long enough and easy to replace.

@ncafferkey
Quote:

Replaced with new CR20232 (and x1000 was about 20-30 seconds off the battery and didn't loose settings), and sadly to say but "i need to hit 2 times restart button to make NIC works" and "sometime not have even boot log and need to reset" still there.. Anyway new battery is surely not hurt, but that something else. But so far no hurt to working on driver for now.

@K-L
It seems really random.. Sometime i lockup easy, but today i tried so hard many times, and in end i were able still to lockup after opening 5 the same download by wget of 1.8gb file.. Sadly on serial absolutely nothing for me :( I.e. i download 10 or 15GB of data before i were able to lockup. And then, you and Petrol lockup right away, while TSK at least able to download 100mb or so.. Find the difference ..

@All
But at least pattern is "random" and "total lockup" , which looks like writing to some other's (DMA?) memory. Only Petrol have something on serial, then there difference between memory layouts (different size of memory, video memory, etc, on one machine important data got hurt fast, on other not so, and depends on re-boot it changes, so, some memory which not always on same place..) If only have some debug, or, at least some 100% reproducible test case on the same place..

Edit: wrong topic

@kas1e
This looks like problem may not be caused by the driver itself but something else interfering. If it's an interrupt issue try if concurrent interrupts matter so what if you download to disk instead of RAM:, play music, use USB while using network, etc. Is there a way to debug interrupt chain when problem happens? As the card is using memory buffers to communicate it could also be something corrupting that memory which I don't know how to find on an OS without memory protection. Maybe you could try write protecting these with the MMU when the driver should not change them and see if you get a hit. The card should still be able to DMA there though I'm not sure but it may worth a try.

Isn't there some way to investigate lockup from QEmu Monitor?

"info registers" ?

"info pic" ?

To see what it is doing during lockup. If it stays hanging in (endless happening?) interrupt-state (~AOS68k supervisor mode) or something.

Or maybe there's some way from Monitor to manually clear interrupt pending (~AOS68k INTREQ clearing) states. To see if you can break out of the lockup.

If not, I would investigate this from AOS side with something like a high pri watchdog task which every 10 seconds does some debug output to serial. To see if any tasks at all running during lockup.

If not, maybe some INTB_VERTB watchdog interrupt which does same thing.

If tasks do still run during lockup, find out which one of the higher pri tasks is not giving up CPU. By lowering watchdog task pri.

Something like that.

@Georg
Another one for qemu topic probably ?:)

@balaton
I wrote small tool called "rx_stress" all what it do connect to the server and recieve what server send, and as server made small python script which send lots of AAAAAA, in 8 parallel lines, filling whole 100mb limit if possible.

Now, one time i were able to survive whole test (3.7gb of data, ~46000 ring wraps in 300 seconds). And another time. Then, for third time did reboot, tried again, and this time died just on ~120mb. On serial again nothing, machine lockedup fully. Another reboot, another test , but this time with shaderjoy on WB playing some effect, and i can't say if lockup come faster or slower, because, stress test alone sometime pass 4gb, sometime can't even 100mb.

Next, as i was fed up, just tried fast running my rx_stress test and ctrl+c it immediately (almost not data recieved mean). Doing this 5 times -> lockup. Reboot, doing this 15 times -> lockup. So no data transfer at all _in this_ runs, but before was big runs. Then after reboot (2 resets in a row, not power off) then all lockups when i just run and ctrl+c it 15 times.

Again reboot, this time were able to lockup even after 80 times. Then i run ShaderJoy with some effect, and after few ctrl+c lockup. So it was close to find something reproducable if not only one if : reboot, run shaderjoy , ~150 ctrl+c in arow no lockup. Keep running rx_stress for minute or two after no lockup. Again bunch of ctrl+C no lockup.

So.. It start to looks like, like my resets (even if i do 2-3 in a row) do not clear things fully. So, between resets some things in l2,cpu, whatever still keeps. Then when i flood enough in previous session by doing different things, then next one (or at end of first one) i do have lockup. And does not matter if it filled by "download" or by anything else, it just, any action which change the memory can cause it. But sometime is pure insane - i can't lockup for 10 minuts or so. Then bam,bam one after another, like you find the way to reproduce it. That only how i see it, but so far nothing which can expain it. For IRQ shtorm i were sure shaderjoy will hit it, but no.

Continue collecting info for now ..

@kas1e

Nothing was on the serial when locking up. I used pFtp and downloaded a file from Hyperion's FTP. I can't remember the speed.

Tonight I can't use X1000 much at all, it locks up after WB loads every time.

@kas1e

Quote:
Did you measure the voltage of the new battery? I've heard several people say that even if they took a new battery directly from a non-expired pack, it could measure below the full voltage.

Best regards,

Niels

@kas1e
I don't know if cache coherence is an issue on PA6T or not but PPC in general has requirements to access io memory. On QEMU maybe you could just access card registers with volatile pointer but on real machine you likely need sync or eieio or whatever is needed after writing values and can't just read write registers through pointers from C so you'd need to use the appropriate functions for this. (See bboot/include/ppc-mmio.h probably there are similar in AmigaOS for this.) Also when writing the descriptors to memory that the card should read you may need to flush the cache with some cache control assembly ops before starting the transfer otherwise the written data may not be in memory so the card would get wrong data. I don't know about these cache control ops as these are also not a problem with QEMU where the cache is not emulated. These are just guesses though.

If you could reproduce the same problem on QEMU (of course not with the pa6teth driver) then it may be easier to debug there or is that a completely different issue? I've lost track of what you're doing.

@TSK
Quote:
Oh, if your x1000 lockups per see even without driver, then we can't know which lockups come from where :(

@balaton
Quote:
I hope it's completely different issue. For now i just fully concentrate on x1000's driver, so all issues i wrote there is about x1000 driver.

As for those your new ideas, i do have already proper barriers in right places (i hope). I do have write_iob_mmio() (single function to write , but from IRQ context) which doing just this : Calculate the target address, then "stwbrx" (to write byteswapped value) and then "eieio" for barier, so it looks like this now:



I use this one because i can't use WriteConfigLong() in IRQ context where no OS calls allower (at least, that how i got it for now), those WriteConfigLongs() i use just for MAC chip config, DMA channels, etc.

Checked Neil's driver, he also do in his IRQ context "(*((volatile UWORD *)(address)) = value)," , just without
stwbrx + eieio , so maybe just lucky because it's external cards ..

Also i use those when need it:



So should be fine in that regards ..