This article has been contributed by Armin Brauns.
The STM32H735 (and many others, but didn't test those) contains the "Root secure services (RSS)" as part of its Secure Memory Management (SMM) module. It's a bit of ROM code in System Memory that allows entering/exiting Secure mode. Specifically, exitSecureArea() is called at the end of a custom bootloader implementation to switch to Normal mode and jump to the application image.
exitSecureArea() takes a parameter that is either RSS_ENABLE_JTAG_AT_EXIT or RSS_KEEP_JTAG_DISABLED_AT_EXIT. Depending on this choice, the ROM code configures the JTAG pins either in Alternate Function mode, or in Analog Input (reset) mode.
However, it does so not by reading the current MODER register, modifying the value, and writing it back, but simply by writing hard-coded values to the MODER registers, overwriting any previous pin configuration in those banks. This annotated assembly dump (from a System Memory image where RSS->getVersion() returns 3.1.0) shows the culprits:
disable_jtag_pins: 0x1ff088f8: 4808 ldr r0, [pc, #32] ; 0x1ff0891c: RCC_AHB4ENR 0x1ff088fa: 6801 ldr r1, [r0, #0] 0x1ff088fc: f041 0101 orr.w r1, r1, #1 ; GPIOAEN 0x1ff08900: 6001 str r1, [r0, #0] 0x1ff08902: 6801 ldr r1, [r0, #0] 0x1ff08904: f041 0102 orr.w r1, r1, #2 ; GPIOBEN 0x1ff08908: 6001 str r1, [r0, #0] 0x1ff0890a: 4905 ldr r1, [pc, #20] ; 0x1ff08920: GPIOA_MODER 0x1ff0890c: f06f 5080 mvn.w r0, #268435456 ; 0xefffffff: PA14(JTCK)=AF, others=reset 0x1ff08910: 6008 str r0, [r1, #0] 0x1ff08912: f46f 7040 mvn.w r0, #768 ; 0xfffffcff: PB4(NJTRST)=input, others=reset 0x1ff08916: 4903 ldr r1, [pc, #12] ; 0x1ff08924: GPIOB_MODER 0x1ff08918: 6008 str r0, [r1, #0] 0x1ff0891a: 4770 bx lr 0x1ff0891c: .data 0x580244e0 ; RCC_AHB4ENR 0x1ff08920: .data 0x58020000 ; GPIOA_MODER 0x1ff08924: .data 0x58020400 ; GPIOB_MODER enable_jtag_pins: 0x1ff08928: 490a ldr r1, [pc, #40] ; 0x1ff08954: RSS_ENABLE_JTAG_AT_EXIT 0x1ff0892a: 4288 cmp r0, r1 0x1ff0892c: d110 bne.n 0x1ff08950 0x1ff0892e: 480a ldr r0, [pc, #40] ; 0x1ff08958: RCC_AHB4ENR 0x1ff08930: 6801 ldr r1, [r0, #0] 0x1ff08932: f041 0101 orr.w r1, r1, #1 ; GPIOAEN 0x1ff08936: 6001 str r1, [r0, #0] 0x1ff08938: 6801 ldr r1, [r0, #0] 0x1ff0893a: f041 0102 orr.w r1, r1, #2 ; GPIOBEN 0x1ff0893e: 6001 str r1, [r0, #0] 0x1ff08940: 4906 ldr r1, [pc, #24] ; 0x1ff0895c: GPIOA_MODER 0x1ff08942: f06f 40a8 mvn.w r0, #1409286144 ; 0xabffffff: PA15(JTDI)=AF, PA14(JTCK)=AF, PA13(JTMS)=AF, others=reset 0x1ff08946: 6008 str r0, [r1, #0] 0x1ff08948: f46f 70a0 mvn.w r0, #320 ; 0xfffffebf: PB4(NJTRST)=AF, PB3(JTDO)=AF, others=reset 0x1ff0894c: 4904 ldr r1, [pc, #16] ; 0x1ff08960: GPIOB_MODER 0x1ff0894e: 6008 str r0, [r1, #0] 0x1ff08950: 4770 bx lr 0x1ff08952: .data 0x0000 0x1ff08954: .data 0x97a0fa17 ; RSS_ENABLE_JTAG_AT_EXIT 0x1ff08958: .data 0x580244e0 ; RCC_AHB4ENR 0x1ff0895c: .data 0x58020000 ; GPIOA_MODER 0x1ff08960: .data 0x58020400 ; GPIOB_MODER
Usually, this isn't much of an issue because the application firmware will (re-)initialize all required pins anyway. There is at least one glaring exception though: If the application firmware is executed from memory-mapped OCTOSPI flash, and some of the OCTOSPI pins happen to be in GPIO banks that also contain JTAG pins, those pins will have their Alternate Function mode reset by the ROM code and jumping to the application will immediately run into an invalid instruction (because the affected OCTOSPI bits read as 0).
A potential workaround for this would be to load a small piece of code into RAM and exit into that, which can then fix the OCTOSPI pin configuration and jump to the actual application image on the external flash.
Another option might be to reimplement the exitSecureArea() functionality in your own code, it seems to come down to clearing bit 0x2 in the undocumented peripheral register at 0x4fff1000 before enabling JTAG pins and jumping to the application.