You generally want to use a trusted protection module (TPM) chip like what is on most current computers and Pixel phones. The thing to understand about the TPM chips is that they have a set of unique internal keys that cannot be accessed at all. These keys are used to hash against and create other keys. The inaccessibility of this unique keyset is the critical factor. If you store keys in any regular memory, you are taking a chance.

Maybe check out Joe Grand's YT stuff. He has posted about hacking legit keys to recover large crypto amounts. Joe is behind the JTAGulator, if you have ever seen that one, and was a famous child hacker going by "Kingpin."

I recall reading somewhere about a software implementation of TPM for secure boot, but I didn't look into it very deeply and do not recall where I read about it. Probably on Gentoo, Arch, or maybe in the book Beyond Bios ^(terrible)^

Andrew Huang used to have stuff up on YT that would be relevant to real security of such a device, but you usually need to know where he wrote articles to find links because most of his stuff isn't publicly listed on YT. He has also removed a good bit over the years when certain exploits are unfixable like accessing the 8051 microcontroller built into most SD cards and running transparently. Andrew is the author of Hacking the Xbox which involved basically a man in the middle attack on a high speed PCIE (IIRC) connection.

It would be a ton of work to try to reverse engineer what you have created and implemented in such a device. Unless you're storing millions, it is probably not something anyone is going to mess with.