If you have a handler already for MCE that would probably dump state to serial so maybe just check that in MSR the ME bit is enabled and you might get a dump without having to write your own handler.
Spend 5 hours to learn how install a stub and make it works (that was not easy) , so in end i just run "mce_stub" and it install stub, i then jump on 0x200, and when reboot, on address 0x7f000000 i have all the info i need. I even wrote simple external command which call the 0x200 , when my stub in : yes, stub works, and write data i need. I was so happy that now yes, i will see something which will point on , but then bah, THAT ISN'T MCE !
CPU died, soft-int died, hardware interrupts died, and this is not MCE too. Then what to check next ? I mean, BUS lockup ? Busy-wait ?
Quote:
When you write these descriptors you may need to translate addresses to physical as the card only sees phys addresses.
Not with IOB on x1000, they ECAM, so direct physical writing, no translation/remapping there.
Quote:
If the DMA engine is shared with other usage you might need to go through the dma.resource to make sure not trying to use same channel by different parts. I think in A1222 one channel is reserved for network, maybe it's the same on X1000?
At least in unix sources it shown clear : for network they use "5" (same as me).
Quote:
We could look at it if you published the source. The title says opensource driver and you used Linux code so it should be GPL anyway. Maybe if you show the source people could help better.
As usual need to clean mess, some ai-cruft and then will upload on github in a day or two.
yes, stub works, and write data i need. I was so happy that now yes, i will see something which will point on , but then bah, THAT ISN'T MCE !
Did you check ME bit in MSR so MCE is not disabled? (Without a CPU manual it may be hard to know which bit is that exactly but maybe it's documented somewhere in the ISA version PA6T follows.)
Quote:
Not with IOB on x1000, they ECAM, so direct physical writing, no translation/remapping there.
I don't understand these acronyms. What do you mean?
Quote:
At least in unix sources it shown clear : for network they use "5" (same as me).
Ask dma.resource author if you have contact what channels that can use and what are reserved for network if any. It may not be the same as on Linux and using a channel without reserving it in dma.resource might lead to clashes with other things using DMA. Is it possible to disable DMA usage in AmigaOS so you can be sure only your driver accesses it? Or get a channel from dma.resource and use that.
One more thought: you mentioned the DMA list wrapping around when the driver is stress tested. Is it possible the driver doesn't properly handle the situation where the DMA list wraps around?
Maybe try using a very small number of descriptors (like two) so that this condition is reached easily and you can examine the behaviour.
CPU died, soft-int died, hardware interrupts died, and this is not MCE too. Then what to check next ? I mean, BUS lockup ? Busy-wait ?
Maybe unlikely, but one possibility is some missing protection when a List is modified by more tasks (or task(s) and interrupt). IIRC I did see cases in some other OS where it did cause funny things like a loop in the List (node->ln_Succ == node). If someone then traversed this List while in Disabled() state you would get a lockup like the one you see (no ints happening, etc.)
Or broken protection in the sense of mismatch between lock (forbid/Disable/...) and unlock (permit/enable) calls. Or only lock call, but forgetten unlock call. This once upon a time in another OS caused funny and hard to reproduce disappearing-task problem (disappeared from taskready/taskwait/thistask). Because every so often when the locking nest count - because of overflow/wrappingback - reached the unlocked-value, things were done in a unlocked state, when they shouldn't have been and in the source it did not look like that because the LOCK was there, but missing/mismatched UNLOCK screwed things up.
Did you check ME bit in MSR so MCE is not disabled? (Without a CPU manual it may be hard to know which bit is that exactly but maybe it's documented somewhere in the ISA version PA6T follows.)
You mean when OS4 loaded up ?
On https://en.wikipedia.org/wiki/PWRficient says "Power ISA v.2.04" so same as others, and that mean bit 12 of MSR are ME bit. Then when OS running , i by mfmsr found that MSR = 0xB030. 0xB030 in binary = 1011 0000 0011 0000 - bit 12 is set, mean MCE enabled.
Quote:
I don't understand these acronyms. What do you mean?
Again some of this are new for me too, and i can say some bull, but i understand this that way : there are old (CAM) and new (ECAM) strandards of PCI/PCIe meaning "Configuration Access Mechanism" for CAM and Extendend Configuration Access Mechanism for ECAM.
What it mean is that it's CPU chip dictate in hardware by PCI/PCIe protocol (CAM/ECAM) where placed configuration space registers. Now, when computer (be it x1000, or PC or any) boots, it's already known which address is a start of the PCI devices (from datasheets on CPUs probably, and in our case i take it from linux kernel code port to X1000, as we have no datasheet). So in our case with X1000 its 0xE0000000, so all PCI starts here physically. The way how they placed in are old (CAM) and new (ECAM).
Now, some devices have BAR, some have not (host bridges, pci-to-pcie bridge, integrated devices like our IOC, MAC, DMA, etc, where not need lots of memory like for graphics card,etc). After computer boots, firmware start to check PCI devices, and discover : if there NOBAR devices, simple skip it. This will be accessed directly. if there is BAR devices it then write an address of device to BAR, and then, when OS boots, it then do MMU mapping of this address, while NOBAR ones still untouched as it, just MMU mapped whole region of 0xE0000000 so we can access them from OS, but individual devices which have no BARs are not MMU mapped.
Now, when i say "those are ECAMS", i mean that those addresses of IOB/DMA/MAC do not go the individual MMU-mapping route by OS, and then GetResourceRange() return NULL for them as search for BARs. You can only access them directly by ECAM standard. Which (both and old CAM and ECAM) have a formulas to calculate an address of device. In our case its ECAMs one :
0xE0000000 | (dev<<15) | (fn<<12) | offset
I can't be 100% sure without datasheet on PA6T CPU that it exactly that statdard, but i can with 100% sure say that the caclulation formula of ECAM standard fits very well on PA6T, and indeed found IOB/DMA and MAC where they are on X1000, so, this probably ECAM standard used indeed (and what else it can be: all in all its same PC, just with PPC cpu and CFE).
Quote:
Ask dma.resource author if you have contact what channels that can use and what are reserved for network if any. It may not be the same as on Linux and using a channel without reserving it in dma.resource might lead to clashes with other things using DMA. Is it possible to disable DMA usage in AmigaOS so you can be sure only your driver accesses it? Or get a channel from dma.resource and use that.
X1000 do have pasemi_dma.resource, but i not use it. I do everything from scratch , so to avoid possible issues other ones step in when trying to create onboard driver. Still it's in the system, and probably used by something else, but i can try to simple comment it out to not boot, to see how OS will works without and if it will , and lockup will still be there, it mean this resource not to blame.
But to be honest i not very well understand why pasemi_dma.resource should allocate anything about network, if it's work of network device, which can (or can not) use this pasemi_dma.resource ?
But to be honest i not very well understand why pasemi_dma.resource should allocate anything about network, if it's work of network device, which can (or can not) use this pasemi_dma.resource ?
Aren't this pasemi dma channels ~general purpose things? Meaning there is no specific reserved channels (by cpu/hardware itself) for network or other specific things?
Maybe AOS on X1000 accelerates (big) memcpys with this dma-stuff and uses pasemi_dma.resource to dynamically/on-demand allocate channels (more tasks can be doing this memcpys at the same time), so it's not the resource itself allocating channels but users of the resource request allocations.
@kas1e Make sure you check the right bit for MSR[ME] as PPC docs count bits from the other end than everybody else normally does. You keep talking about writing register addresses in the memory mapped PCIe config space or BARs but what I was talking about is if the card needs transfer descriptor structs in memory like USB controllers or rtl8139 does then you should make sure the addresses you write in those descriptors are correct and that these are written to memory before you tell the card to read them which may need a cache flush or memory mapped non-cachable. But I don't know how this works so not sure this applies. I only know DMA on Sam460EX and A1222 that I had experience emulating but X1000 probably works the same. AmigaOS uses the DMA engine for blitter ops or memory moves and this seems to be coordinated by dma.resource. The Sam460ex is simple as it has only one DMA channel but on A1222 I noticed not all channels are used. Then I was told some channels may be reserved for audio or network that's why not all channels are claimed by dma.resource. If the driver goes through dma.resource to alloc a channel then you could make sure nothing else will use that. Otherwise you may need to use the channel reserved for it but I don't know which is that or is there any. The author of the dma.resource for the X1000 should be able to tell. If you pick a channel that dma.resource can also use then it could be the kernel would also try to use that and break your settings.
[...] 3). Buy JTAG, and check the state when CPU died..
4) Using the unused X1000/X5000 co-processor "Xena" could help. Even if the main CPU is stuck this chip could still do something. I don't know anything at all about it, but I read something on this forum about someone using it for debugging.
5) PerformanceMonitor.resource, if implemented in the X1000 kernel. Not all kernels support it. Very low level and high priority, may still work even if interrupt handlers don't work anymore or are stuck in an endless loop, or if the OS is killed by something like IExec->Disable(); while(1);
I only know DMA on Sam460EX and A1222 that I had experience emulating but X1000 probably works the same. AmigaOS uses the DMA engine for blitter ops or memory moves and this seems to be coordinated by dma.resource. The Sam460ex is simple as it has only one DMA channel but on A1222 I noticed not all channels are used. Then I was told some channels may be reserved for audio or network that's why not all channels are claimed by dma.resource. If the driver goes through dma.resource to alloc a channel then you could make sure nothing else will use that. Otherwise you may need to use the channel reserved for it but I don't know which is that or is there any. The author of the dma.resource for the X1000 should be able to tell. If you pick a channel that dma.resource can also use then it could be the kernel would also try to use that and break your settings.
For sake of tests i simple comment out loading of pasemi_dma.resource , then of course power off / power on with long enough waiting just in case, and lockup still there. So this probably rule out it.
But i think it's time to wait for source to be online, just need a bit more time to clean them up.
@joerg Quote:
Replacing it with a custom one is possible, but can't help.
Already did, but my lockup didn't cause MCE, its something else, like BUS deadlock which i do not know if possible some to catch at the "beginning" of this process so to dump addresses/etc.
Quote:
4) Using the unused X1000/X5000 co-processor "Xena" could help. Even if the main CPU is stuck this chip could still do something. I don't know anything at all about it, but I read something on this forum about someone using it for debugging.
I do not know if i understand it correctly, but is any "lockup" mean you always (does not matter what kind of lockup) can "do something with some hw/sw tools" ? I mean, can't it be such a lockup which then take over everything and everything died just like you hold power off button ? Like those BUS lockups for example i think of ?
Quote:
5) PerformanceMonitor.resource, if implemented in the X1000 kernel.
It implemented for now in all kernels, but i can confirm it works at least on x5k (on beta kernel), and on peg2 (as years before), dunno about x1000, but then, what use it give us there ? I always think that this is something which help collect static data to use with gprof or so after.
Quote:
Not all kernels support it. Very low level and high priority, may still work even if interrupt handlers don't work anymore or are stuck in an endless loop, or if the OS is killed by something like IExec->Disable(); while(1);
For me it didn't looks like anything work: Soft-int with maximum priority died. Interrupts with maximum priority died. MCE never called. It just looks like bus lockup, no ?
One problem with this timer softint loop I forgot is that (unless AOS4 does it differently) the softints caused from a hw interrupt are not executed immediately, but only when the last hw interrupt is done, ie. before returning to usermode.
So even if hw timer interrupt would still be running/runnable, a "stuck" lower hw priority interrupt could still prevent the "loop" (which re-sends the timer request) from working.
Using AOS4 version of maybe undocumented ~PA_CALL/~PA_FASTCALL instead of PA_SOFTINT for the timer replyport would work better. Because the difference to PA_SOFTINT is that when the timer is replied, it does not result in Cause(), but in a direct function call.
Quote:
Interrupts with maximum priority died.
The priority of your interrupt handler is just for multiple handlers among same hw irq. Hw pri is still the same whatever it is. You can set it to max. possible values, but interrupts which have higher hardware priority are not affected.
So even with the test results it's probably still not 100% sure, that it isn't "simply" some (high pri) interrupt handler being stuck in a loop or that's there endless interrupt flood.
The priority of your interrupt handler is just for multiple handlers among same hw irq. Hw pri is still the same whatever it is. You can set it to max. possible values, but interrupts which have higher hardware priority are not affected.
So even with the test results it's probably still not 100% sure, that it isn't "simply" some (high pri) interrupt handler being stuck in a loop or that's there endless interrupt flood.
But i wrote in previous post(s) that i create version also with no interrupts usage at all for sake of tests (at least from driver), and do all by cpu -> same lockup.. So this probably out of question too.. If only you didn't mean not driver's irq flood, but system's one when some bug happens ..
Question is : can we for sure detect what is it : BUS lockup or what ? Only by JTAG ?
Then I fall back to the loop-in-some-list theory. Maybe some stupid little bug that causes some Node (or io request or whatever has a node in it) to be removed while it is not in a list, or added to a new list while it is still part of another list, or added a second time when it already is in the list (io: re-send an io request while it is still pending). Something like that.
@Georg Or i hardware bug :) At least we should remember that others who working before on same kind of driver also meet with some issues like "halt when transfer data", but i do not know what kind of halt it was, and if it all related to what we have there..
@Balaton I go futher : i just wrote small tool which query the DMA table, and: 1). Check how many Interfaces it have and for what devices (numbers are taken from .dts files of x1000 linux port + this small TRM we have for x1000). 2). check how many channels (rx and tx) it have and what ones are active, and what ones in use.
Then result is (without driver running and without pasemi_dma.resource running):
Dunno who own it, but that explain me why with first tests of driver i wernt able to use TX 0, i were sure it's something about video memory, because i had video memory trash when tried to use it, so i skip it in favor of TX channel 1, even have comment about like "video take the channel 0?"
But then, the RX channel i use is 5, and TX channel i use are 1. Now how channels looks like when i run with both pasemi_dma and our driver:
Hi Kas1e, I was looking on your ranger'screenshots and remarked that we've got the same hardware MAC address! Should not it be different? It also differs from the Firmware value of ETH0_HWADDR Env var.
I was looking on your ranger'screenshots and remarked that we've got the same hardware MAC address! Should not it be different? It also differs from the Firmware value of ETH0_HWADDR Env var.
I do not remember, but wasn't there some issue with MAC addresses on x1000 where they all the same? I sure remember something of that sort, just dunno if for x1000.
Are gfx drivers in AOS4 already loaded/running when S:startup-sequence is executed? If not, maybe try to run network stress test there when as few as possible other things (gfx/audio/...) are running in the background.
@Georg I didnt tried Linux myself on x1000 yet, but those who did on hyperion forum keep saying all fine. But maybe i should test it myself anyway.. and probably better go 32bit version of linux not 64bit one (to be close as possible).
As for gfx drivers, etc, you think they even without usage of pasemi_dma.resourse do dma stuff for x1000 internaly ? Then when os loaded without pasemi_dma.resource i dudnt see any channel is used by anyone. I mean, if anyone use channel i should see active one in dma index map, but i dont ..
In end of all i can just try 3-5 tests with random channels like tx12/rx12, tx18/rx23 etc just random and if lets say 3-5 random channels cause same => sure no one mess us with channels, right ?
@kas1e I think it's more likely you screw something up when programming the chip so I'd verify all data passed to the chip in any way (through registers or memory based lists/descriptors whatever it uses) if that's matching what the Linux driver does. Testing that the Linux driver works on same hardware is also a good idea to know it should work in the first place. I think the network interface could lock up the machine if e.g. it gets a wrong DMA address and overwrites something. It could get wrong DMA adress either because of missing virt/phys translation or because of missing cache flush so CPU writes correct address but it does not reach memory yet when the card reads it. Or just some error somewhere so the address or other data is not even correct. If none of the above then maybe the driver could screw something up in AmigaOS as suggested by others. If it's locking up with all interrupts disabled then the only thing that could kick it out from that could be NMI but on PowerPC the only non maskable interrupts seem to be reset and MCE so if you could cause an MCE somehow it should then log state but I don't know how could that be done.
News feed
OS4 Feedback forum
OS4Depot Feedback forum
AmiCygnix forum
ABC shell forum
AmiKit forum
Cinnamon Writer forum
CodeBench forum
Digital Universe forum
Dopus 5 forum
E-UAE forum
Gnash forum
Ibrowse forum
JAmiga forum
Odyssey forum
OWB forum
Qt forum
SmartFileSystem forum
Timberwolf forum
TouchDevice forum
TuneNet forum
Unsatisfactory Software forum
Donate
