Preface
I am a big fan of the Fallout TV series, specifically the sense of a capitalistic impending apocalypse etc. There is nothing I like more than making a good scenario in my head (outside of the entire world being annihilated and of course me dying – I have main character syndrome).
This was where this project was born. I was taking interest in free and local information sources and one of the ones that I think is very information rich (and potentially likely to survive some form of apocalypse) is terrestrial radio e.g. AM/FM. From radio, you can get news updates, weather updates, general entertainment and much more. I say this, I’m more of a spotify user than a radio listener but the principle is there at least…
Back to the project: I originally wanted to make a system that can produce summarised news excerpts on a display, entirely offline and via some form of speech to text (ASR) pipeline. In classic fashion, I had to make this project way harder for myself that it needed to be. My self imposed pre-requisites are:
- Use UK FM radio sources
- FM could be a module but must not have an external antenna
- Be completely offline – no online ASR inference (I’m looking at you Whisper)
- Be solar powered – This is a little more of a stretch goal. As I delve deeper into how these types of systems work, it looks more and more unlikely that I can fit the project into a small solar budget for a handheld device without needing to resort to a humungous solar panel (relative to the device size)
- Given the low power requirement, use some low powered embedded platform.
- I am better with STM32 than Linux however ASR outside of maybe single keyword recognition requires drastically larger quantities of RAM than an STM32 can offer. There is more on this later.
- Be able to summarise a full news program into snippets but have some form of LCD to let the user navigate transcripts etc.
Along with this, there are of course some stretch goals:
- Include a speaker to play back recorded radio excerpts
- Have a PCB FM antenna
- Include some method of user navigation
- Record radio transcripts for future playback.
Next revision
After doing a bit of preliminary research into this, I ended up constraining the domain further to just weather. Initially I wanted full news coverage but after determining what would be required for the ASR pipeline, it seemed very RAM consuming. I guess now is a good time to add that I had personally locked myself into the Rockchip RV1103G1 – a chip used on the Luckfox Pico, a device I have been using for my development. The chip is also available on JLCPCB so that is a massive win too.
I want to add here that I am going to be writing a few of these posts after the fact of doing the work so the prose and style may be a little confusing (sorry).
Rockchip RV1103
The Rockchip RV1103 is a great little chip, it includes an ARM Cortex A7 at 1.2GHz and embeds 64MB of DDR2 RAM, all in a nice and easy low pin QFN style package. It does also include an NPU which I could look to using at the future but for now, I’m only going to be looking at CPU inference.
Some of the great features of the RV1103 that will be used for this project are:
- Native SPI and I2C support via Linux
- Very easy to configure via Buildroot in generating the OS
- Integrated audio CODEC interface
- Low power consumption
Architecture
Hardware
From my self imposed architecture, I could start to put together a bit of an architecture. This architecture is very likely to change over the iterations but I will add updates as I progress through the development.
Some factors of this architecture are determined by what is available on JLCPCB – just to remove the amount of stuff I would need to solder by hand.
Software
I am by trade an electronics engineer. I have been known to do software when required but I would in no way call myself a software developer. It may be frowned upon but I have been using ChatGPT for my software development and can safely say how impressed with it I am. I have got working software, along with demonstrable results so even if it isn’t the most efficient code, it has worked and has given me outcomes that I am happy with!
I have taken a reasonable amount of time to look into this section of the project. A stock version of buildroot reserves half of the RAM for the NPU leaving only 32MB left over to run Buildroot and any user software, as confirmed by meminfo.
[root@luckfox pico pocketsphinx]# cat /proc/meminfo | head MemTotal: 33216 kB MemFree: 8320 kB MemAvailable: 13192 kB Buffers: 300 kB Cached: 3876 kB SwapCached: 44 kB Active: 1384 kB Inactive: 3724 kB Active(anon): 176 kB Inactive(anon): 772 kB
The first step was to reclaim this code from the NPU. This meant a small nugget of modification to the files that influence the Buildroot build. There is a portion of the RAM in the device tree reserved for the NPU which is allocated from the CMA area. By removing that allocation and not using the NPU, that RAM can be reclaimed by Linux for the entire system to use. I may do a full post on this…
Continuing with the architecture:
The audio output from the Si4702 is just normal audio which can feed directly into the RV1103G1 audio CODEC. From this, normal Linux tools can be used to play and record (namely normal ALSA tools – arecord and aplay). arecord can then be piped into opusenc to save the radio recordings in a low bitrate format. I am currently looking at around 8-12kbps for the audio. I did try lower bitrates but the WER increased pretty significantly. Even at 12kbps, a full day of radio recordings would just be 129.6MB of space.
Once saved, the audio will be reloaded chunk by chunk into the ASR pipeline as the ASR model is the largest consumer of RAM. The text outputs are then saved to a tiny ramdrive to be handled by the next stage. The intermediate ramdrive stage is to release RAM from the ASR pipeline.
Once summarised, the summaries will be listed on the LCD with up and down buttons capable of selecting each summary for an extended read, or potentially scope for audio playback.
Initial results
I started on the most unknown to me parts, the ASR section. Knowing I only had a 64MB total RAM limitation (moreso, only 50MB after Buildroot). I originally thought I’d be able to get a nice quantized version of Whisper Tiny to run on here being a mere 39M parameters though this is already 156MB of fp32 space anyway. Even with an int8 quantization, its 39MB without any intermediate or other buffers – sadly infeasible for my platform.
I then looked to the current smallest possible ASR I could get on the platform – Pocketsphinx. Pocketsphinx has been shown to be capable of running on constrained platforms previously, especially with low RAM requirements. After wrestling with the cross compiler, I was able to get Pocketsphinx working on the Luckfox Pico, with a reduced size but custom LM. I did however need to install and optimize zram for the platform. One of the advantages I have for this project is the lack of real time transcription requirements. This means I can essentially trade compute time for RAM, which in this case means I can use the most efficient compression algorithm for zram. I don’t know the performance of algorithms but ChatGPT did inform me taht this was zstd. After another Buildroot recompile test, I was able to get zram working with the zstd algorithm.
Using a zram size of 80M allowed me to run Pocketsphinx with my custom LM and successfully produce text transcript outputs with no OOMs or SD card swaps, wahoo!
Further research
After successfully running Pocketsphinx, I was happy that there was at least some form of output I was capable of getting however the performance of PS was pretty poor, even with a custom LM. On a piece of BBC Radio 4 audio, I was getting around a WER of 0.665. With a custom dictionary, I was hitting A WER of 0.463, a Rouge L1 of 0.665 and a Bleu of 0.31.
Whilst a WER of 0.463 might seem quite high, I wanted some better metrics for evaluation. In the case of what I’m looking for, I want better weighting on semantic accuracy vs absolute word accuracy. This is why I started to include Rouge and Bleu. Somewhat confusingly, a WER of 0 is best but a Rouge/Bleu of 1 is best.
There were quite a lot of substitution errors that Pocketsphinx was causing so I wanted to see if there were ways that I could correct for these. My first method was a rule based approach i.e. Any time Pocketsphinx produces “the good weather is”, replace with “the weather is good”. These substitutions were analysed from the original transcripts and a corpus of replacements was generated. I then tried to train a Seq2Seq model for this but the results weren’t great. I know this would be a feasible thing to do but at least the method I was looking at wasn’t good enough. This could however be something I’d return to in the future if required.
Next steps
Now the bare minimum of a Pocketsphinx model is working, I wanted to look and see if I could improve on the WER. I tried searching to see if there were any intermediate models between pocketsphinx and Whisper Tiny in size and came across Vosk but even Vosk small has a size of 40MB. I just couldn’t seem to find anything sub 30M parameters (at least from my initial searches). This lead me to the next step… Training my own ASR model. I can safely say I have trained a 12M ASR model based on a conformer architecture that has a quantized size (using ONNX) of 12.7MB (INT8) with negligible WER increase due to quantization (0.087 vs 0.091 WER for FP32 vs INT8).
Actual WER for my conformer model is 0.414 vs Pocketsphinx 0.463.
I do have more work to do on the system architecture from an electronics perspective but the majority of my time recently has been spent on the ASR section. One thing to add however is that this model was trained using scraped BBC weather and met office data from Youtube. This is of course dubiously licensed at best and without wanting to add to poorly licensed and copyrighted AI models, I will not be using this model for the final product – instead looking for fully open source or friendly licensed data sets which can be used for my training.
At this point I realised the only remaining option was to build my own speech recogniser from scratch. I expected this to be the hardest part of the project. Surprisingly, it ended up becoming my favourite
Harris' Electronics 