GetResource not working locking up system about trampoline

[grundmanns]

With the fix applied my system is running strong for 4 hours now. I will continue testing.

[grundmanns]

4:30h crash again. I have to check.

[jlbirccyn]

Hello Sven

(res)->owner != INVALID_TASK

means the resource is already taken. In this test, the actual process id is used. It should not be confused with what is done in GetTaskIDthat return INVALID_TASK if the running process is not an application task.

TPL_KERN(core_id).s_running->base_priority >  res->ceiling_priority

is true if the running task attempts to get a resource and its base priority (ie the priority given in the OIL file and packed in the OIL compiling process) is greater than the priority of the resource which is computed by doing the max of tasks/ISR2 that use the resource + 1

So a && does not make sense because one OR the other is an error.

In both case, it is a configuration problem.

[grundmanns]

Hello Jean-Luc,

thank you for your quick answer. This is the reason i am consulting you, as I though maybe I understand that stuff not the right way.

I had the case that it was approaching this code and the resource was already taken.
It then returned an error without checking the priority.

1. What could I possibly do wrong, that this happens ?

2. Never approach GetResource if it is taken ?
   This means in principle it should never approach the GetResource code if the resource is already taken because the priority of the task taking the resource is supposed to be high enough to not let that happen, is that correct ?

3. How many tasks can participate in a resource ?
   This means if I have a base priority of 4 bit and a rank of 4 bit, what happens if there are more than 16 task participating in the same resource. Would that be even possible ?
   There would be a ceiling prio that is used for the task that gets the resource but there would not be enough different ranks to give everbody a different prio.

BR
Sven Grundmann

[jlbirccyn]

The dynamic priority is formed by the basic priority (most significant bits) and the inverse order of activation calculated modulo (i.e. the older the job, the greater the order of activation), the rank.

1 and 2. Let's suppose 2 tasks sharing the same resource. t1 with a base priority 1 and t2 with a base priority 2. The priority of the resource is computed when compiling the OIL (or arXML) file and gives 3. When t1 takes the resource its priority increases to 3 and t2 cannot preempt t1. If it appears that t2 preempts t1 and tries to take the resource, this error appears. Does t2 have the resource listed in its OIL description?

3. If the basic priority is 4 bits, then there can be no more than 16 tasks, including idle. The data type for the priority increases in size depending on the number of tasks and the number of possible activations.

2 possibilities:

1. Your configuration is not ok and resource priority is false
2. There is a bug in the templates that leads in some cases in wrong values.

[grundmanns]

Hello Jean-Luc,
I checked tpl_app_config.c and the basic problem i see is this:
I have a resource that is used by most of my task and the ISR.
The ceiling prio of the resource is 7, which is I guess the maximum, as the system somehow assumes that it should not use bit 8 (bit is sometimes does)
The definition in tpl_app_define.h is:
#define PRIORITY_SHIFT 4
#define PRIORITY_MASK 240
#define RANK_MASK 15
This would mean priorities up to 15 were possible and rank 0-15 is possible.

tpl_app_config correctly describes which task and ISR take the resource.
The ISR have the priority of 1 in the oil file.
In the tpl_app_config.c the ISR have a priority of 7.
I guess they must be above the tasks and that is the reason it is calculated like that.
This means the base priority of the ISR and the resource is the same.

Do you see it as a problem, as you described to me the ceiling prio should be max(prio_of_all_task) + 1
which it right now isn't.

BR
Sven Grundmann

PS: To 3. Do you mean there cannot be more that 16 task at all or 16 task with the same prio ?
My total task count is about 50.

[jlbirccyn]

Normally it is not a problem to have the priority of the resource equal to the highest priority of the tasks/ISRs that share it. However, this is not what should be computed as you point out. I will look into what is wrong.

Regarding the PS, I misspoke. It is the number of priority levels, not the number of tasks (unless the tasks have all different priorities).

[jlbirccyn]

I looked at the ISR category 2 declared in the OIL file you sent to me and none of them declares it uses a resource.

[grundmanns]

Hello Jean-Luc, you are right. This was an error i also already discovered and fixed. This is now changed but the problem I describe happened after I fixed it. Should I resend the file ? Do I have to check it for typo? I guess goil would report if I had a typo in the resource declaration. BR Sven Grundmann

…

________________________________ From: Jean-Luc Béchennec ***@***.***> Sent: 31 August 2022 14:37 To: TrampolineRTOS/trampoline ***@***.***> Cc: Sven Grundmann ***@***.***>; Author ***@***.***> Subject: Re: [TrampolineRTOS/trampoline] GetResource not working locking up system (Issue #112) I looked at the ISR category 2 declared in the OIL file you sent to me and none of them declare it uses a resource. — Reply to this email directly, view it on GitHub<https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FTrampolineRTOS%2Ftrampoline%2Fissues%2F112%23issuecomment-1232882327&data=05%7C01%7Csven.grundmann%40renesas.com%7Ce5369689bcbf4e70395208da8b4d966d%7C53d82571da1947e49cb4625a166a4a2a%7C0%7C0%7C637975462596694365%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XOY4D29L16lRjxZ2lOqQ%2BFFSr6Hu11j9ykzwoEoUtX0%3D&reserved=0>, or unsubscribe<https://jpn01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHYRN2KKOZIJL64JXVCUMYDV35GY7ANCNFSM6AAAAAAQASMEZ4&data=05%7C01%7Csven.grundmann%40renesas.com%7Ce5369689bcbf4e70395208da8b4d966d%7C53d82571da1947e49cb4625a166a4a2a%7C0%7C0%7C637975462596694365%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Kzv9JUUMjp2nBmgGaF7kDcacOLAAIBJUJy2bHU%2B9y8M%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.***> Renesas Electronics Europe GmbH, Geschaeftsfuehrer/President: Carsten Jauch, Sitz der Gesellschaft/Registered office: Duesseldorf, Arcadiastrasse 10, 40472 Duesseldorf, Germany, Handelsregister/Commercial Register: Duesseldorf, HRB 3708 USt-IDNr./Tax identification no.: DE 119353406 WEEE-Reg.-Nr./WEEE reg. no.: DE 14978647

[jlbirccyn]

Yes, attach the OIL file below so that I can compile on my side and look at the result.

[grundmanns]

Hello Jean-Luc,
thank you!
project_oil.zip

BR
Sven

[jlbirccyn]

Hello Sven,

Could you include the C sources ?

BTW, shouldn't the high2 task have a priority of 3?

Best regards

[grundmanns]

Hello Jean-Luc,
you are absolutely right about high2 should have a prio of 3. I guess, I made some test because adding task with prio 3 broke the system scheduler as discussed in the other issue. I must have overlooked that one.

Regarding c-Source:
I am not sure if I can easily give all of my project c sources.
I could give you the autogenerated sources in the build directory.
Do you need some specific source file or just all of them ?
BR
Sven Grundmann

[jlbirccyn]

I would need the part where system calls are done

Best regards

[grundmanns]

You mean the GetResource ReleaseResource ?
I can give you that.

[jlbirccyn]

Yes as long as the WaitEvent/ClearEvent that appear in all the tasks since they are all extended.

[grundmanns]

Hello Jean-Luc,
I can check the relevant sources, but I have a series of Debugger images, that show the lockup problem.

This shows the task high6 being executed shortly before acquiring the osal_mutex resource.
Task id is 28 and the priority of the task is 0x3A.
The sync cat 2 isr has a task id of 41 and a prio of 0x72
The resource base prio is 7
Next image is stepping over GetResource:

Task 28 becomes the resource owner and it's dynamic prio is lifted to 0x70
This is a lower prio than isr sync has right now.
3x instructions later (check the trace window on the left) there is an IRQ.
This means it loads the sema variable, that should be protected, compares it and then gets interrupted.
We are now in the cat1 interrupt, right after the irq subarch handler has been executed.

Please check the tpl_kern structure in the watch window.
You see that task 41 (sync cat2 isr) is scheduled for execution and its prio is even lifted from 0x72 to 0x7f
In my understanding this should not happen at all.
The resource ceiling prio should be 8, then the ISR would not be scheduled before high6 is past the ReleaseResource.
I wonder if goil can calculate a base prio of 8, on the other hand side I already saw a prio >=8 for a Cat2 ISR, which was interpreted negative.

Now the scheduler has scheduled the cat2 ISR.
The resource is still owned by high6 with task id 28.
This is shortly before cat2 sync ISR tries to aquire the resource. r0 = 0 means resource 0 is aquired, which is the one that is already taken.

This image is shortly before the error is thrown by GetResource.
It is comparing r2 with r3 and seeing that r2 is a regular task id and not invalid task, which means the resource is taken.
the bne will branch to the error case then.

I compiled it with the oil file is put in here.
The isr high6 are properly connected via oil file to the osal_mutex.

My guess is that the resource calculation is kind of flawed. It does not seem to understand the real prio is the ISR.
The high6 prio taking the resource is already lower than the prio of the sync cat2 ISR. This should not happen.

Can you give me a comment about this ?
Thank you very much for your help. I really appreciate it.

BR
Sven Grundmann

[grundmanns]

I debugged it.
I found that the main_task we had had prio 200 which is kind of annoiing but not the problem.
The problem is in the file:
root.goilTemplate trampoline/goil/templates/root.goilTemplate

This is the root cause:

# increase the RESSCHEDULERPRIORITY by one and have it
# set between the highest priority task and the lowest ISR priority
let OS::RESSCHEDULERPRIORITY := OS::RESSCHEDULERPRIORITY + 1

I guess you want task that use resource to not block the ISR.
After that you shift the ISR prio by one up to have a gap, but not the resources that they use.
And then there is the problem the resource has the same prio as the ISR.
BR
Sven Grundmann

[grundmanns]

I fixed it.
I can give you the patch.

[grundmanns]

0001-Fix-GetResource-not-working-with-cat2-ISR.zip

[jlbirccyn]

Hello Sven.

I see the problem. The resources used by ISR2 don't have the right priority. However, the patch you propose works with your example but may fail for ISR2s that are only used by tasks because their priority will be increased by 1 and will become equal to the priority of the tasks above them and will therefore no longer have priority over a lower priority task that holds the resource. I'll have a look.

Best regards.

[grundmanns]

Hello Jean-Luc,
you are right, there might be this complication. If you have an idea how fix it more universal, this would be nice.
But you increase all CAT2 interrupt. I am wondering how Cat1 have a prio anyway, as they are scheduled by the GIC and not the scheduler.
On the other hand you increase ALL cat2 ISR, which means even if the prio would be among the Cat2 it should work.
The list is a prio sorted list at this stage, which means it should be ok.

What do you think about my debugger image sequence? It shows the problem really good. I was proud, I could isolate that problem pretty well.

BR
Sven Grundmann

[jlbirccyn]

Hi Sven,

Normally, the priorities in GIC should be set so that all ISR2 have a priority ≤ priority of the SVC (they cannot interrupt the kernel). ISR1 may have priority > as the priority of the SVC. Computing actual priorities for the GIC depends on the target platform. So ISR1 and ISR2 live in a different priority world because ISR2 cannot interrupt the kernel since they may trigger the scheduler while ISR1 may.

I have a patch.

root.goilTemplate.patch.txt

I added a resource only shared by task to check the priority is ok for resource that are used by ISR2.

You debugger image sequence is great and very useful 👍. I got the problem immediately.

Best regards

[grundmanns]

Hello Jean-Luc,
thank you very much for your help. I will check your patch on Monday.
BR
Sven

[grundmanns]

Hello Jean-Luc,
just one comment to your comment:

I see the problem. The resources used by ISR2 don't have the right priority. However, the patch you propose works with your example but may fail for ISR2s that are only used by tasks because their priority will be increased by 1 and will become equal to the priority of the tasks above them and will therefore no longer have priority over a lower priority task that holds the resource.

If there is a resource that is only used by ISR2, then it's prio must be MAX(ISR2_1, ISR2_2) + 1 .
This means in the list it is coming later than the ISR2, as the list is priority sorted.
This means in the list first all tasks are there and then als ISR2 are there.
All ISR2 are lifted by one by your code.
This means lifting the resource with a prio > than the smallest ISR2 should allways work, right ?
Did I understand something wrong ?
This is why I started to increase the resource prio only if the resource prio is greater than any ISR2 prio.

BR
Sven Grundmann

[jlbirccyn]

You are right ! I overlooked the fact that since the list was sorted from low to high priorities, resources only assigned to tasks did not have their priority increased by 1 since they are checked before ISR2s and thus before isr_start becomes equal to 1.

By the way you can get a documentation on our template language here: https://github.com/TrampolineRTOS/GTL/blob/master/documentation/GTL.pdf

Best regards

[grundmanns]

Hello Jean-Luc,
at least my patch was good! I am happy.
Thank you for the Link about GTL. You are right, I was struggeling a bit with the GTL stuff, but a real programmer only needs a working printf and then can debug / fix everything. ;P
Once I found display / println, I was good.
Have a nice weekend.
BR
Sven Grundmann

[grundmanns]

Hello Jean-Luc,
for me the topic is closed as the error was identified and fixed by my patch and I guess yours.
BR
Sven

[jlbirccyn]

Hi Sven,

Could you check it is ok with my patch ?

Best regards

--
Jean-Luc Béchennec