STM32 gotchas
264. exitSecureArea() in SMM in STM32H735 overwrites GPIOA_MODER and GPIOB_MODER, possibly leading e.g. to OCTOSPI failure

This article has been contributed by Armin Brauns.

The STM32H735 (and many others, but didn't test those) contains the "Root secure services (RSS)" as part of its Secure Memory Management (SMM) module. It's a bit of ROM code in System Memory that allows entering/exiting Secure mode. Specifically, exitSecureArea() is called at the end of a custom bootloader implementation to switch to Normal mode and jump to the application image.

exitSecureArea() takes a parameter that is either RSS_ENABLE_JTAG_AT_EXIT or RSS_KEEP_JTAG_DISABLED_AT_EXIT. Depending on this choice, the ROM code configures the JTAG pins either in Alternate Function mode, or in Analog Input (reset) mode.

However, it does so not by reading the current MODER register, modifying the value, and writing it back, but simply by writing hard-coded values to the MODER registers, overwriting any previous pin configuration in those banks. This annotated assembly dump (from a System Memory image where RSS->getVersion() returns 3.1.0) shows the culprits:

disable_jtag_pins:
0x1ff088f8:     4808            ldr     r0, [pc, #32]   ; 0x1ff0891c: RCC_AHB4ENR
0x1ff088fa:     6801            ldr     r1, [r0, #0]
0x1ff088fc:     f041 0101       orr.w   r1, r1, #1      ; GPIOAEN
0x1ff08900:     6001            str     r1, [r0, #0]
0x1ff08902:     6801            ldr     r1, [r0, #0]
0x1ff08904:     f041 0102       orr.w   r1, r1, #2      ; GPIOBEN
0x1ff08908:     6001            str     r1, [r0, #0]
0x1ff0890a:     4905            ldr     r1, [pc, #20]   ; 0x1ff08920: GPIOA_MODER
0x1ff0890c:     f06f 5080       mvn.w   r0, #268435456  ; 0xefffffff: PA14(JTCK)=AF, others=reset
0x1ff08910:     6008            str     r0, [r1, #0]
0x1ff08912:     f46f 7040       mvn.w   r0, #768        ; 0xfffffcff: PB4(NJTRST)=input, others=reset
0x1ff08916:     4903            ldr     r1, [pc, #12]   ; 0x1ff08924: GPIOB_MODER
0x1ff08918:     6008            str     r0, [r1, #0]
0x1ff0891a:     4770            bx      lr
0x1ff0891c:     .data 0x580244e0 ; RCC_AHB4ENR
0x1ff08920:     .data 0x58020000 ; GPIOA_MODER
0x1ff08924:     .data 0x58020400 ; GPIOB_MODER

enable_jtag_pins:
0x1ff08928:     490a            ldr     r1, [pc, #40]   ; 0x1ff08954: RSS_ENABLE_JTAG_AT_EXIT
0x1ff0892a:     4288            cmp     r0, r1
0x1ff0892c:     d110            bne.n   0x1ff08950
0x1ff0892e:     480a            ldr     r0, [pc, #40]   ; 0x1ff08958: RCC_AHB4ENR
0x1ff08930:     6801            ldr     r1, [r0, #0]
0x1ff08932:     f041 0101       orr.w   r1, r1, #1      ; GPIOAEN
0x1ff08936:     6001            str     r1, [r0, #0]
0x1ff08938:     6801            ldr     r1, [r0, #0]
0x1ff0893a:     f041 0102       orr.w   r1, r1, #2      ; GPIOBEN
0x1ff0893e:     6001            str     r1, [r0, #0]
0x1ff08940:     4906            ldr     r1, [pc, #24]   ; 0x1ff0895c: GPIOA_MODER
0x1ff08942:     f06f 40a8       mvn.w   r0, #1409286144 ; 0xabffffff: PA15(JTDI)=AF, PA14(JTCK)=AF, PA13(JTMS)=AF, others=reset
0x1ff08946:     6008            str     r0, [r1, #0]
0x1ff08948:     f46f 70a0       mvn.w   r0, #320        ; 0xfffffebf: PB4(NJTRST)=AF, PB3(JTDO)=AF, others=reset
0x1ff0894c:     4904            ldr     r1, [pc, #16]   ; 0x1ff08960: GPIOB_MODER
0x1ff0894e:     6008            str     r0, [r1, #0]
0x1ff08950:     4770            bx      lr
0x1ff08952:     .data 0x0000
0x1ff08954:     .data 0x97a0fa17 ; RSS_ENABLE_JTAG_AT_EXIT
0x1ff08958:     .data 0x580244e0 ; RCC_AHB4ENR
0x1ff0895c:     .data 0x58020000 ; GPIOA_MODER
0x1ff08960:     .data 0x58020400 ; GPIOB_MODER

Usually, this isn't much of an issue because the application firmware will (re-)initialize all required pins anyway. There is at least one glaring exception though: If the application firmware is executed from memory-mapped OCTOSPI flash, and some of the OCTOSPI pins happen to be in GPIO banks that also contain JTAG pins, those pins will have their Alternate Function mode reset by the ROM code and jumping to the application will immediately run into an invalid instruction (because the affected OCTOSPI bits read as 0).

A potential workaround for this would be to load a small piece of code into RAM and exit into that, which can then fix the OCTOSPI pin configuration and jump to the actual application image on the external flash.

Another option might be to reimplement the exitSecureArea() functionality in your own code, it seems to come down to clearing bit 0x2 in the undocumented peripheral register at 0x4fff1000 before enabling JTAG pins and jumping to the application.