Migrating from Optiboot to Urboot about mightycore

[stefanrueger]

I recommend the typically 384-byte urboot vector bootloaders with EEPROM r/w capabilities and reset vector protection for use with avrdude -c urclock. Only if the board does not have an accessible SPI header and if reflashing is only possible using bootloader and if reflashing is expected during deployment and it is in an expensive product would I recommend to use a hardware-protected bootloader that is compatible with the old STK500v1 protocol (and avrdude -c arduino).

A couple more comments:

automatic baud rate detection

Urboot provides autobaud detection for classic parts with USARTs. This costs 16 bytes and is normally possible within a 384-byte urboot vector bootloader.

Flashing LED

Unlike Optiboot that flashes for 300 ms or so after reset, urboot switches the LED on at the start of a bootloader byte read function getch() and off again at the end of getch(); this saves code space and instills a visual sense of upload and download.

The Optiboot flasher library needs to be replaced with an Urboot-compatible one

Here some library code for this: urboot.zip It will need to be slighty ported (it's not written for the Arduino IDE). Using avrdude -c urclock for uploading a sketch creates a metadata interface just below the bootloader. That interface allows the unused flash (between sketch and metadata) to be used like EEPROM.

vector bootloaders

They are cool. Here the urboot recommendation with more background/tips how to select bootloader options

support the older stk500v1 protocol as well, for backward compatibility?

Only useful if avrdude -c urclock is assumed to be unavailable. Backward compatibility here means that the urboot bootloader can work with avrdude -c arduino of old AVRDUDE versions as well as avrdude -c urclock of AVRDUDE v7.1. This backward compatibility costs a lot of bytes in the bootloader.

EEPROM support is nice, but Arduino IDE doesn't utilize this feature

EEPROM support is very nice. Even if the Arduino IDE does not utilise this, any application might utilise it

• Get parameters for the sketch from EEPROM. Users can inspect and set these parameters with AVRDUDE in terminal mode interactively, eg, through

  $ avrdude -qq -t -c urclock
  avrdude> read eeprom 0 4
  0000  ff ff ff ff                                       |....            |
  avrdude> write eeprom 0 0xC0CAC01A
  avrdude> quit
  

• A sketch can write output into EEPROM; this can then be read by, eg, avrdude -U eeprom:r:-:h and further processed externally without the need to interface I/O in the sketch.

[mcuee]

@stefanrueger

Maybe you can give some guideances here. Thanks.

[MCUdude]

The thing is that Optiboot works really, really well, but I'd like to give Urboot a try because it provides more features and a smaller footprint than Optiboot.

Unlike Optiboot that flashes for 300 ms or so after reset, urboot switches the LED on at the start of a bootloader byte read function getch() and off again at the end of getch(); this saves code space and instills a visual sense of upload and download.

Usually, there are LEDs on the RX/TX lines on either the target board or the USB to serial adapter. I'm willing to sacrifice some flash in order to get the same behavior as Optiboot.

This is why a flashing sequence is useful:

• You can quickly check if the board has a bootloader installed or not by pressing the reset button and check if the LED toggles (twice for Optiboot)
• You can look at the LED toggle speed and somewhat figure or if the bootloader that's present is compiled for a different F_CPU. LED is toggling too slow? The chip is probably running at its intenal 8 or 1 MHz oscillator but it running a bootloader compiled for a 16 MHz clock
• You can clearly see that the auto reset circuity is doing its job. This is neat when I'm testing the the boards I sell on Tindie

Here some library code for this: urboot.zip It will need to be slighty ported (it's not written for the Arduino IDE). Using avrdude -c urclock for uploading a sketch creates a metadata interface just below the bootloader. That interface allows the unused flash (between sketch and metadata) to be used like EEPROM.

Isn't this what the flash looks like when Urboot is used?

[ Start of flash ]
[ Vectors ]
[ User program ]
[ Unused flash space ]
[ Bootloader ]
[ Metadata ]
[ End of flash ]

With the current Optiboot flash library the user defines a progmem array which ends up as a section with an start address that lines up with the flash pages. It's very simple and understandable for the average user (I hope). Will the "Urboot-way" of doing this be more or the same as Optiboot does it? A library that interacts with some built-in functionality of the bootloader?

MightyCore/avr/libraries/Optiboot_flasher/src/optiboot.h

Lines 43 to 63 in ce441c4

	// 'typedef' (in following line) and 'const' (few lines below) are a way to define external function at some arbitrary address
	typedef void (*do_spm_t)(uint16_t address, uint8_t command, uint16_t data);
	/*
	* Devices with more than 64KB of flash:
	* - have larger bootloader area (1KB) (they are BIGBOOT targets)
	* - have RAMPZ register :-)
	* - need larger variable to hold address (pgmspace.h uses uint32_t)
	*/
	#ifdef RAMPZ
	typedef uint32_t optiboot_addr_t;
	#else
	typedef uint16_t optiboot_addr_t;
	#endif
	#if FLASHEND > 65534
	const do_spm_t do_spm = (do_spm_t)((FLASHEND-1023+2)>>1);
	#else
	const do_spm_t do_spm = (do_spm_t)((FLASHEND-511+2)>>1);
	#endif

[stefanrueger]

[blinkenlights]

The urboot LED actually has similar utility. It will tell you that the chip has been reset: the bootloader LED comes on after reset in the getc() of the bootloader. This is different from any LEDs on the RX/TX because the urboot LED tells you that the bootloader sees the correct host TX line toggle. It does not use up 300 ms at every reset. I prefer simple bootloader queries such as which bootloader capabilities avrdude -c urclock -xshowall not be delayed. The urboot LED shows bootloader activity during its lifetime (eg, when being kept alive in terminal mode). It only costs 6 bytes flash in contrast to optiboot's typically 26 bytes.

Urboot has a separate option -DDEBUG_FREQ=<n> (in Hz) that swings a square wave of 5 periods on pin -DFREQ_PIN=AtmelP<pin> (can be the LED pin) to enable exact F_CPU measurement with a scope (eg, in case of an imprecise resonator). I do not recommend that as default option as it is pretty wasteful in terms of flash.

Unused flash

I don't think a sketch on a chip with optiboot can easily figure out how large the actual optiboot bootloader is. This is one of the deficiencies of optiboot, which has failed to ask the question How would a sketch know where the bootloader starts? Your example, @MCUdude, out of necessity uses a guess(!) that can easily be wrong as you can compile different sizes of optiboot for any chip. Urboot's metadata interface gets avrdude -c urclock to store the start and length of unused flash in the metadata section, ie, there is no ambiguity. The functions urstoreRead(uint8_t *sram, uintpgm_t where, size_t n) and urstoreWrite(uint8_t *sram, uintpgm_t where, size_t n) have a byte-wise granularity, and every byte of unused flash can be used. where counts from 0 to the number of unused flash bytes - 1. BTW, unused flash, which I call Store, sits between the application and the metadata:

[ Vectors ]
[ Application ]
[ Store (unused flash) ]
[ Metadata ]
[ Bootloader ]

The urboot bootloader itself has a small table at FLASHEND that contains, amongst other things the size of the bootloader, ie, an application sketch can figure out where the metadata sit that in turn define the Store section with unused flash.

There is also a function void urbootPageWrite(void *sram, progmem_t pgm) that writes a memory page from SRAM into flash. Look at urboot.zip and try that out... you may need to do a little bt of porting but if you are lucky the mentioned functions work out of the box.

Urboot does not export a do_spm function (though it could at the expense of extending the top FLASH bootloader table by two bytes); instead it exports at FLASHEND+1-4 a complete function writepage() that is interfaced by urbootPageWrite(). The trick in urboot is to put that function at the known end of flash rather than the (generally) unknown begin of bootloader where optiboot puts the do_spm interface.

[MCUdude]

I'm about to push a test branch where I've replaced Optiboot with Urboot in the boards.txt file.

However, I thought I'd look at the metadata that was stored along with the program, but to my surprise, no metadata gets stored. Is this a bug or am I just doing things wrong here? The ATmega1284P is currently running the urboot_atmega1284p_autobaud_ee_lednop_fr_ce_ur_vbl.hex bootloader

$ ./avrdude -curclock -p atmega1284p -Uflash:w:blink_1284p.hex:i

avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e9705 (probably m1284p)
avrdude: Note: flash memory has been specified, an erase cycle will be performed.
         To disable this feature, specify the -D option.
avrdude: erasing chip
avrdude: reading input file blink_1284p.hex for flash
         with 1160 bytes in 1 section within [0, 0x487]
         using 5 pages and 120 pad bytes
avrdude: preparing flash input for device bootloader
avrdude: writing 1160 bytes flash ...

Writing | ################################################## | 100% 0.18 s 

avrdude: 1160 bytes of flash written
avrdude: verifying flash memory against blink_1284p.hex

Reading | ################################################## | 100% 0.12 s 

avrdude: 1160 bytes of flash verified

avrdude done.  Thank you.

$ ./avrdude -curclock -p atmega1284p -xshowall

avrdude: AVR device initialized and ready to accept instructions
ffffffffffff 0000-00-00 00.00  application 0 store 0 meta 0 boot 512 u7.7 weu-jPrac vector 27 (SPM_READY) ATmega1284P

$ ./avrdude -curclock -p atmega1284p -xshowfilename

avrdude: AVR device initialized and ready to accept instructions

$

[MCUdude]

However, metadata gets stored when I'm using a hardware bootloader (both with auto-baud support).

$ ./avrdude -curclock -p atmega1284p -xshowall

avrdude: AVR device initialized and ready to accept instructions
ffffffffffff 0000-00-00 00.00  application 0 store 0 meta 0 boot 1024 u7.7 weu-hprac vector 0 (RESET) ATmega1284P

$ ./avrdude -curclock -p atmega1284p -Uflash:w:blink_1284p.hex:i

avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e9705 (probably m1284p)
avrdude: Note: flash memory has been specified, an erase cycle will be performed.
         To disable this feature, specify the -D option.
avrdude: erasing chip
avrdude: reading input file blink_1284p.hex for flash
         with 1160 bytes in 1 section within [0, 0x487]
         using 5 pages and 120 pad bytes
avrdude: preparing flash input for device bootloader
avrdude: writing 1160 bytes flash ...

Writing | ################################################## | 100% 0.18 s 

avrdude: 1160 bytes of flash written
avrdude: verifying flash memory against blink_1284p.hex

Reading | ################################################## | 100% 0.12 s 

avrdude: 1160 bytes of flash verified

avrdude done.  Thank you.

$ ./avrdude -curclock -p atmega1284p -xshowall

avrdude: AVR device initialized and ready to accept instructions
ffffffffffff 2022-03-28 11.16 blink_1284p.hex 1160 store 128857 meta 31 boot 1024 u7.7 weu-hprac vector 0 (RESET) ATmega1284P

$ ./avrdude -cusbasp -patmega324pb -Uhfuse:r:-:h

avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e9517 (probably m324pb)
avrdude: reading hfuse memory ...
avrdude: writing output file <stdout>
0xd7

avrdude done.  Thank you.

[MCUdude]

It turned out to be the lock bits (I think?). It was set to 0xCF by Arduino IDE, but it's only able to write metadata when the lock fuse is set to 0xFF.

EDIT:
BTW, the test Urboot branch is here: https://github.com/MCUdude/MightyCore/tree/urboot

[stefanrueger]

about to push a test branch where I've replaced Optiboot with Urboot

Very cool!

It turned out to be the lock bits

Yes, one needs to switch off hardware protection of the boot section in order to use the free space from smaller bootloaders. It's documented in the Usage section of urboot, but forgetting that has happened to myself and @mcuee as well. It is probably a good idea to add metadata verification for avrdude -c urclock. Maybe there is even a way to give a neat error message that points to the lock bits as a possible source of metadata verification failure...

I am at the cusp of publishing pre-compiled urboot bootloaders for all possible UARTs on the chip. Seeing that you have your own boards, @MCUdude, you could add them in the script that generated pre-compiled urboot bootloaders: https://github.com/stefanrueger/urboot/blob/e25f00c8785201cb03c4e1bd36c97ecb4ce12654/src/mkurboots#L144-L187

BTW, I'll change the naming scheme to always include the rx/tx line in the file name as well as which uart/swio is used or whether an ALT UART pins were assigned. So that a user can tell from the file name where to hook up comms.

Names will look like below. Note this part is one of the very few classic parts that actually has an alt UART pin assignment.

Size	Features	Hex file
254	w-u-jpra-	urboot_attiny441_autobaud_uart0_alt1_rxb2_txa7_ur_vbl.hex
254	w-u-jpra-	urboot_attiny441_autobaud_uart0_rxa2_txa1_lednop_ur_vbl.hex
254	w-u-jpra-	urboot_attiny441_autobaud_uart1_rxa4_txa5_lednop_ur_vbl.hex
296	w-u-jPra-	urboot_attiny441_autobaud_uart0_alt1_rxb2_txa7_lednop_fr_ur_vbl.hex
308	w-u-jprac	urboot_attiny441_autobaud_uart0_alt1_rxb2_txa7_lednop_fr_ce_ur_vbl.hex
316	w-u-jPrac	urboot_attiny441_autobaud_uart0_rxa2_txa1_lednop_fr_ce_ur_vbl.hex
316	w-u-jPrac	urboot_attiny441_autobaud_uart1_rxa4_txa5_lednop_fr_ce_ur_vbl.hex
320	weu-jpra-	urboot_attiny441_autobaud_uart0_alt1_rxb2_txa7_ee_ur_vbl.hex
320	weu-jpra-	urboot_attiny441_autobaud_uart0_rxa2_txa1_ee_lednop_ur_vbl.hex
320	weu-jpra-	urboot_attiny441_autobaud_uart1_rxa4_txa5_ee_lednop_ur_vbl.hex
378	weu-jPrac	urboot_attiny441_autobaud_uart0_rxa2_txa1_ee_lednop_fr_ce_ur_vbl.hex
378	weu-jPrac	urboot_attiny441_autobaud_uart1_rxa4_txa5_ee_lednop_fr_ce_ur_vbl.hex
384	weu-jPrac	urboot_attiny441_autobaud_uart0_alt1_rxb2_txa7_ee_lednop_fr_ce_ur_vbl.hex

[MCUdude]

Seeing that you have your own boards, @MCUdude, you could add them in the script that generated pre-compiled urboot bootloaders: https://github.com/stefanrueger/urboot/blob/e25f00c8785201cb03c4e1bd36c97ecb4ce12654/src/mkurboots#L144-L187

But wouldn't this generate a new "boards" folder where the bootloaders I've added get stored? I'd prefer if the bootloaders with various LED options were stored in their respective /mcu/[mcu type] folders instead. There's no need to create folders in the "boards" folder called "MightyCore", "MiniCore", "MegaCore" and "MajorCore" as there's nothing special about these bootloaders apart from their LED pin.

As mentioned in the first post, these are the LED configurations my cores use. Could it be an idea to add these as pre-compiled bootloaders for your urboot.hex repo? Optiboot alternatives with this exact LED configuration have existed for years, so my guess is that bootloaders for these parts with this particular LED pin would be the most popular ones anyways.

MightyCore targets

| MightyCore  | LED option 1 | LED option 2 |
|-------------|--------------|--------------|
| ATmega1284P | PB0          | PB7          |
| ATmega1284  | PB0          | PB7          |
| ATmega644P  | PB0          | PB7          |
| ATmega644A  | PB0          | PB7          |
| ATmega324PB | PB0          | PB7          |
| ATmega324PA | PB0          | PB7          |
| ATmega324P  | PB0          | PB7          |
| ATmega324A  | PB0          | PB7          |
| ATmega164P  | PB0          | PB7          |
| ATmega164A  | PB0          | PB7          |
| ATmega32    | PB0          | PB7          |
| ATmega16    | PB0          | PB7          |
| ATmega8535  | PB0          | PB7          |

MiniCore targets

| MiniCore    | LED option 1 |
|-------------|--------------|
| ATmega328PB | PB5          |
| ATmega328P  | PB5          |
| ATmega328   | PB5          |
| ATmega168PB | PB5          |
| ATmega168P  | PB5          |
| ATmega168   | PB5          |
| ATmega88PB  | PB5          |
| ATmega88P   | PB5          |
| ATmega88    | PB5          |
| ATmega48PB  | PB5          |
| ATmega48P   | PB5          |
| ATmega48    | PB5          |
| ATmega8     | PB5          |

MegaCore targets

| MegaCore    | LED option 1 |
|-------------|--------------|
| ATmega6490P | PB7          |
| ATmega6490  | PB7          |
| ATmega6450P | PB7          |
| ATmega6450  | PB7          |
| ATmega3290P | PB7          |
| ATmega3290  | PB7          |
| ATmega3250P | PB7          |
| ATmega3250  | PB7          |
| ATmega2560  | PB7          |
| ATmega1280  | PB7          |
| ATmega2561  | PB5          |
| ATmega1281  | PB5          |
| ATmega640   | PB7          |
|             |              |
| Atmega649P  | PB5          |
| ATmega649   | PB5          |
| ATmega645P  | PB5          |
| ATmega645   | PB5          |
| ATmega329P  | PB5          |
| ATmega329   | PB5          |
| ATmega325P  | PB5          |
| ATmega325   | PB5          |
| ATmega169P  | PB5          |
| ATmega169A  | PB5          |
| ATmega165P  | PB5          |
| ATmega165A  | PB5          |
| ATmega128   | PB5          |
| ATmega64    | PB5          |
| AT90CAN128  | PB5          |
| AT90CAN64   | PB5          |
| AT90CAN32   | PB5          |

MajorCore targets

| MajorCore   | LED option 1 |
|-------------|--------------|
| ATmega162   | PB0          |
| ATmega8515  | PB0          |

[stefanrueger]

mkurboots treats a boards as a soldered, ie, fixed, combination of (MCU, LED, LEDPOLARITY, SFMCS), where the latter is the chip select line of an optional SPI flash memory for dual boot. The corresponding bootloaders are actually generated under mcus/<mcu>/... but then copied to boards/<board>/... with a slightly changed bootloader name (eg, _atmega328p_ replaced with _uno_).

OK, I see that for the cores both the MCU and the LED position is variable. I still suggest to treat them initially as boards and then copy the generated bootloaders to cores/<core>/<mcu>/...

Having copies of the relevant bootloaders under boards/ and cores/, respectively, makes it easier for a user to find/select the right one. A part with many boards/cores (think ATmega328P) quickly generates a shedload of bootloaders.

How about sth like the following?

# Treat MightyCore, MiniCore, MegaCore and MajorCore initially as boards
if [[ /atmega1284p/atmega1284/atmega644p/atmega644a/atmega324pb/atmega324pa/atmega324p/atmega324a/atmega164p/atmega164a/atmega32/atmega16/atmega8535/ =~ /$mcu/ ]]; then
  board+=/mightycore:LED=AtmelPB0
  board+=/mightycore:LED=AtmelPB7
elif [[ /atmega328pb/atmega328p/atmega328/atmega168pb/atmega168p/atmega168/atmega88pb/atmega88p/atmega88/atmega48pb/atmega48p/atmega48/atmega8/ =~ /$mcu/ ]]; then
  board+=/minicore:LED=AtmelPB5
elif [[ /atmega6490p/atmega6490/atmega6450p/atmega6450/atmega3290p/atmega3290/atmega3250p/atmega3250/atmega2560/atmega1280/atmega640/ =~ /$mcu/ ]]; then
  board+=/megacore:LED=AtmelPB7
elif [[ /atmega2561/atmega1281/atmega649p/atmega649/atmega645p/atmega645/atmega329p/atmega329/atmega325p/atmega325/atmega169p/atmega169a/atmega165p/atmega165a/atmega128/atmega64/at90can128/at90can64/at90can32/ =~ /$mcu/ ]]; then
  board+=/megacore:LED=AtmelPB5
elif [[ /atmega162/atmega8515/ =~ /$mcu/ ]]; then
  board+=/majorcore:LED=AtmelPB0
fi

@MCUdude Please check whether I translated your tables correctly.

[MCUdude]

Thanks, @stefanrueger! The table looks correct to me.

I'm looking forward to give it a try!

[stefanrueger]

The table looks correct to me.

Good, I only asked b/c in your MegaCore table, ATmega2561 and ATmega1281 are listed with PB5 amongst PB7 LEDs.

I'm looking forward to give it a try!

OK, I'll add this and some code to create and copy the correct core/<core>/<mcu>/... structure.

[stefanrueger]

@MCUdude: Finished; have a look at the cores directory of the urboot.hex repository.

[DRSDavidSoft]

I'm trying to make the Urboot bootloader work with ATmega32 using MightyCore:

stefanrueger/urboot#25