StatusCode 429

[NickZeeuw]

Hi all,

I am experiencing errors in starting the SP from the pipelines. We did not have this before and in production we luckily don't have the issue, but in dev we do. It seems that we run into some Rate Limiting in the API requests.

This does not give me a good feeling about scaling up the amount of entities to ingest. The logging can't become the bottleneck of the process. Anyone else experiencing this?

Kind regards,
Nick

[edkreuk]

@NickZeeuw , it is hard to say where it coming from. Are you using one database config for Dev and Prod, because prod is still working What is the size of the capacity on Dev? How many cores are you using for the Fabric Database 4 or 32.

[NickZeeuw]

Hi Erwin, thanks for the quick reply. I actually found the issue what caused this. There was a change in dev where we changed all the lkp queries in the pipelines by adding a schedule in the filter. In one of the command pipelines the group by was missing. So het did not start 1 copy from pipeline for 14 entities. But 14 times the copy from pipeline to copy 14 tables. So 14 x 14.

We are using 2 databases, so 1 for dev and 1 for prod, both 32 cores.

Really happy with this framework!

[edkreuk]

Great to hear