ODROID-N2 doesn't boot from Kingston eMMC modules #3250
Comments
|
Just jumping in to say I have a very similar issue with my new n2+ as well... |
|
I have been struggling with this again today, seems to not be based on the version. neither 12.0 or 12.1 work for me. However, if i flash ubuntu the device boots fine. So i dont think it can be a device or emmc issue. No idea what to do any more but it seems like I cant use HAOS with my odroid n2+ |
|
Just FYI - I had to buy a USB to serial converter to get any indication of what's going on. I've been informed they changed the bootloader between 11.4 and 11.5. 11.4 boots fine but 11.5 gives me a message over serial saying that it's unable to find the eMMC card (see my thread above for specifics). |
Yeah, im following that now too, unfortunately 11.4 doesnt work for me either. No idea what to do any more, on yours i noticed a successful boot seems to use mmc 1 but unsucessful is trying mmc 2. No idea why that would be but im at a loss on what else to do any more. I basically just cant use HAOS at the moment, even though i had it running for 3 days on 12.0 before it died... i have no idea how to even begin trying to solve this |
|
I tried 12.2.rc1 today, the issue still persists |
|
Let's track it in #3227 as this seems to be the issue with most information at this point. |
|
Reopening this after discoveries from #3227 (comment). This issue is clearly a problem with the Kingston eMMC modules which never worked with the upstream U-Boot and the 24 MHz we carry to mitigate some other eMMC-related issues. Since we pulled back the 12.3 release for ODROID N2 which was the only one that fixed booting with the Kingston eMMC modules, it was actually never resolved. |
sairon
changed the title
|
Copying my commment from #3227 here:
|
|
Regarding version, I haven't changed my HAOS versions since installation. So I would assume its still on 12.3. I have been updating HA Core but I imagine that doesn't affect this. |
|
So I didn't upgrade, in fact deliberately skipped the upgrade. And on a reboot it now wont boot again. Literally so sick of this, first trip into home assistant, ordroid and MMCs and I end with a PoS that barely works. sorry for the rant but its so frustrating how anyone is supposed to know that their MMC is going to be an issue. |
|
Just a further update. I have decided to no longer use HA. This was my first time using HA. This experience has been utterly awful and left an incredibly sour taste in my mouth. I am the unlucky person to buy a "supported" - or at least it wasn't unsupported - EMMC, and then have the developers unsupport it on me. I don't believe I have received much help regarding this issue, or getting it to work. The small amount of help I did receive only ended up in the EMMC being unsupported. Which is basically telling me to get stuffed and deal with it. I realise this is open source, and there's no reason for anyone to provide support, but there is also an entire company behind it but yet here I am struggling. There only seems to be one company available for me to buy an EMMC from, and they cannot guarantee to provide me with a non-Kingston emmc. So i am basically screwed until someone with knowledge decides to pick this up and fix it. I have given up on HAOS, and as a first experience into it, its been utterly terrible. I thank anyone who has tried, and sorry this may come across as ungrateful but please understand i've spent 100s of pounds on supported hardware only to be told, yeah your not supported any more - Good luck on your own. |
|
Link to another two people with the issue, as reported on issue #3227 Also, seems to be multiple people here with the same/similar issue: |
|
Kingston 32GB device does not work on multiple N2+ systems. I have tried every single HAOS version from 11.3 up to the latest 13.1. None worked. Device tries to boot from network with "serverIP not set" console message. Just hangs. Other OS (Ubuntu) flashed on this 32GB eMMC does work without problems. Pics attached of working and non-working eMMC device. |
|
Same here. Screenshot from non working v10.5: |
|
In a strange, turn of events, I am currently running 12.4 on my 32gb kingston emmc. I am not restarting it, for fear that it wont turn back on.. however, it does seem very strange that I am now working fine on 12.4. Posting this mainly to maybe help the developers here identify what may be going on. It doesn't seem as simple as it doesn't work with kingston emmc's, but something a little more intricate as it works randomly |
|
Sorry for leaving you hanging here for such a long time, and sorry @piercy for the bad experience you had with Home Assistant. Unfortunately, this is something that struck us unprepared and it took a while to discover what was the cause of the sudden issues on N2. It also wasn't straightforward to obtain these eMMCs, as the distributors haven't had them or refused to check what model they have in stock, which prolonged the time before we could move forward. The problem with N2 is that the eMMC driver in U-Boot is known to have issues for quite a long time. There is a hacky patch for quite a while in HAOS that helped to fix the intermittent issues on the previous hardware. It's not only problem of HAOS but of other distributions using mainstream U-Boot on meson-g12b (Amlogic S922X) in general - only just a few months ago Armbian included it too. The downside of the patch is that it breaks communication with new Kingstom eMMC modules, which fail (intermittently, as you noticed) to operate at the lower frequency (although they should work fine in this setting too). On the other hand, when we adjusted the patch to fix Kingston issues, it triggered regression on hardware that worked well up until that point, which is more serious than something not working "out of the box". The nature of this hardware bug is most probably in the clock timing, and if you don't know what you're doing, you can only shoot in the dark and try fiddling with numbers, hoping it will change. To reliably analyze the issue, you need specialized hardware that can help you see what's going on on the eMMC lines and why it sometimes fails. Even though Nabu Casa is behind the Home Assitant, it doesn't mean we have ways to solve every issue that is out there, especially if it requires specialized knowledge and other resources. At this point, only telling you that there is a bug that we can't resolve and the hardware is unsupported (yet) is the only "support" we can provide you. There are still workarounds though - you don't need to boot from an eMMC - you can use an SD card for the boot. You still might be able to use the Kingston eMMC as the data disk (which is where the most data is written), since the Linux eMMC driver doesn't seem to have the same issues as the U-Boot's does. You can also stay at HAOS 12.3, or use the U-Boot binary from this version to run any other HAOS version (I'll be glad to provide instructions how to do so) - you just need to make sure you don't upgrade the OS before the U-Boot bug is resolved - if you do, you'll have to replace the U-Boot binary again. Also, it looks like that the Kingston eMMCs only come in 32 GB size variants, that can serve as a buying guide to avoid these issues. For some good news, I tried to find someone who could help me identify why the Kingston eMMC behaves differently, and potentially what's the difference in driving the eMMC from U-Boot and from the kernel. There is something in the works but it would not be sooner than by the end of the September though, so patience is the key now. |
|
@sairon thnx for the update! |
|
I've struggled with this issue on 32 and 64 GB Kingston eMMC. I am now running HA but I am afraid to reboot or upgrade the system. I bought myself a back power supply so my n2+ will stay alive when a power interrupt should occur. |
|
Fingers crossed that someone can find a solution for this, either here or upstream somewhere. I was going to order one of the new Odroid M2s until I noticed that the soldered eMMC it comes with appears to have the word "Kingston" on it. I'm going to hold off for now if booting off a microSD is the only easy option for it. |
|
I got a new eMMC module (64GB) to replace 32GB Kingston and, unfortunately, I got Kingston again. I have working system with non-Kingston eMMC that works in both N2+ systems. Comparing debug messages between the two (failing and passing boot with two different eMMC), it show that failing boot is trying to boot from MMC device at slot 2, continues to try to boot from network and fails. Both cards did not respond to "voltage select" but that did not stop non-Kingston eMMC from booting successfully. Failing boot shows an old U-Boot version because I was trying to flash an old version of HAOS (I believe it was 12.3 or older) while good boot is using the latest 13.1 version of HAOS. When HAOS is flashed onto SD card (any version), system boots without problems. |
|
No help for debugging. But just the information that it seems that Hardkernel emmc modules with Kingston get more common. Bought some in Germany from different suppliers to set up a new Home Assistant on ODROID-N2+. All with Kingston. And all show the same boot up error as described above (trying to boot from slot 2, not able to set voltage). |
|
Hi @sairon, is there any news about a fix for emmcs with Kingston chip? I bought the ODROID-N2+ because it is still recommended hardware on the Home Assistant website. And on the hassio setup page there's still no information that users with new hardware are currently unable to set up a new system as described on the website. So I guess more and more users will run into this problem. Is there a timeline for when we can expect a fix? I know its not easy to fix, but any information or guess is better than nothing. Thanks for the update! |
This is a good point, HA need to stop recommending this if they aren't going to support it. The EMMC chips are getting more common. More and more people are going to get duped. There's no mention of this incompatibility / lack of hardware support. |
|
Also could we please remove the duplicate flag from this thread, this issue isn't a duplicate. |
Testing shows that the 24MHz cap is problematic for Kingston. Bumping this cap to 40 MHz in #3319 fixed Kingston but reintroduced issues with some other eMMCs that the frequency cap was supposed to fix. This commit adds another patch that selects a different clock for eMMC. While it's still unclear why Kingston fails with the frequencies of slightly below 24 MHz (see patch message why it's below), using 24 MHz from xtal seems to be fine and hopefully shouldn't introduce regressions for old eMMCs. Alternative approach would be using the same clock source but setting the frequency cap to 25 MHz. This results in exact 25 MHz from PLL which is even the defined frequency for some eMMC modes, Kingston eMMC works fine with it and the difference from the previously used value wouldn't be too high, but with 24 MHz we're even closer. refs #3250, refs #3227
|
I've made some progress and things are looking good on my side, I will appreciate wider testing with the images available from this build (should finish in couple of hours top). Eventually, the patch could be included in 14.0 and its pre-releases in the coming weeks. I'm hesitant to include it in the coming 13.2 as there's still a slight chance it could introduce regression on other eMMC modules. |
|
Edit: neither 13.2, 12.4, nor the 12.3 dev build you provided a while back is running for me now. I don't know if its just luck. Going to keep trying with it see if I can work anything out. I was 100% running 12.4 since august..so seems to be temperamental Edit 2: I tried again via micro-USB (rather than my MMC adapter), then flashing with balenaEtcher and no luck. So i've tried the 13.2 dev build, the 12.4 build that I have been using since august, and a 12.3 dev build you tried for us before. None of them work via micro USB or flashing with my mmc adapter. It's really odd that I managed to get 12.4 working since august, but now I cant get anything to work. I havent restarted since august, but i was definitely running 12.4. Edit 3: I accidentally left it plugged in over night. It still didn't boot. However, I power cycled it and it started booting. It was on the 13.2 dev build. However, I didn't reflash it, just power cycled and it worked. Very odd. @sairon Unfortunately that version doesn't boot for me. The N2+ shows a red light for pwr but the blue sys light that usually flashes, isn't flashing (which correct me if I am wrong, usually means it has an issue and isn't trying any more). I also tried to communicate with it via the HA web interface but it wasn't available (which I expected but thought I would try). There is no output on the HDMI, so I am unable to get any additional information from it. Is there another way for me to get more information out of it? Steps for what I did:
|
|
Patch that @sairon put in devel branch does not work for me. Same message during boot Card did not respond to voltage select! : -110 Trying to source non-Kingston eMMC module. |
|
Patch tested from the devel branch @sairon posted 3 days ago. Also tested 12.3 (didn't test this one before, but several other releases I did test as well, all with same result). It all ends in *** ERROR: `serverip' not set This is on a 64GB Kingston eMMC, see photo. Ubuntu boots from this eMMC though.
|
|
Thank you all for testing and reporting back, I was off by the end of the week and couldn't reply sooner. While the fix worked in isolated U-Boot tests and quick HAOS boot test, for some reason it still fails with higher rate in the normal build/initialization procedure.
It's not that odd, given that if the boot fails, it doesn't reboot and try again, but tries to boot from the ethernet and then presents a U-Boot shell prompt (it's the default behavior of upstream U-Boot config and it's not documented for HAOS, so I'd like to change that as well). Anyway, even though the bad news is that the fix doesn't work, there's still good news that I'm able to reproduce it and I still have couple of ideas what to try, just this time I'll do some more extensive/overnight testing to verify the issue is really gone. Thanks for the patience. |
|
Thanks @sairon , if it helps I am piercy, on the HA discord. If you need more immediate feedback feel free to message me. I am happy to try random devbuilds and all sorts of stuff if it helps in the debug process. I realize it can be painful doing a build then waiting a couple of days for people to see if it works. I do struggle to get debug information off of the n2+, i dont know if theres some kind of debug connector I can get to get more information from it? Either way, if it helps I am happy to spend some time just trying stuff until we find something that works |
|
Thanks @piercy for the offer, you've been very helpful with the testing here as well, next time I'll just need to do test everything more thoroughly myself first. But surely I can ping you the next time I have something to test too. About the debug information, there's a dedicated UART connector next to the microSD card - there's either a Hardkernel's proprietary USB-UART module, or you can use a generic 3.3V USB-TTL converter an connect it according to the pinout documented on the ODROID wiki. Disregard the instructions for UART_EE_A/UART_EE_B connection, U-Boot only uses the one on the CON5. |
|
Quick note for everyone - I received an alert notifying me of an update to the supervisor to 2024.10.2; I've been skipping Core and Supervisor updates until the issue with the eMMC is resolved but this time I was presented with a message telling me auto updates are enabled and the version can't be skipped. After some digging around, I found the cli instructions to disable auto-updates: Maybe this is unnecessary but I'd rather not lose my working system to an unavoidable auto update. |
…ROID-N2 Replace the original patch that limits the frequency to 24 MHz with a more sophisticated version that first tries to initialize the eMMC at this limited frequency, and if it fails, then tries disabling the limit to use one defined in the upstream driver. While still hacky, this seems to finally fix the Kingston eMMC issues, and with the current implementation, the initialization should appear the same as on older HAOS versions, as 24 MHz is used first. Better solution that addresses the root cause on some eMMCs (128 GB Microns?) would be still nice though... refs #3250, refs #3227
Replace the original patch that limits the frequency to 24 MHz with a more sophisticated version that first tries to initialize the eMMC at this limited frequency, and if it fails, then tries disabling the limit to use one defined in the upstream driver. While still hacky, this seems to finally fix the Kingston eMMC issues, and with the current implementation, the initialization should appear the same as on older HAOS versions, as 24 MHz is used first. Better solution that addresses the root cause on some eMMCs (128 GB Microns?) would be still nice though... refs #3250, refs #3227
Replace the original patch that limits the frequency to 24 MHz with a more sophisticated version that first tries to initialize the eMMC at this limited frequency, and if it fails, then tries disabling the limit to use one defined in the upstream driver. While still hacky, this seems to finally fix the Kingston eMMC issues, and with the current implementation, the initialization should appear the same as on older HAOS versions, as 24 MHz is used first. Better solution that addresses the root cause on some eMMCs (128 GB Microns?) would be still nice though... refs #3250, refs #3227
|
There's another build which should fix the Kingston issues, this time I hope for good, as it also passed a bit more extensive testing this time. Again, feedback from testing on other systems is welcome. |
|
@sairon looks good, mine booted immediately. I am currently waiting for it initialize, but usually I wouldn't even get this far. Thank you! I really appreciate the effort.
|
|
How do you install this build? I am waiting to update the HA OS because of this issue so if it would be solved then that would be great. |
Hey @TibsD , you can find the dev release mentioned above here: https://os-artifacts.home-assistant.io/14.0.dev20241107/haos_odroid-n2-14.0.dev20241107.img.xz Note, I believe its only for the Odroid N2 (and N2+). For install, i used balenaEtcher following this: https://www.home-assistant.io/installation/odroid/ It's a developer build so take it with a little pinch of salt, but i've been running it since last week and had no issues. The device has been rebooted a couple times and it all works for me. Which prior to this build I couldn't us HAOS at all. |
|
@piercy will this then let me update the home assistant OS on a regular basis? Now I am holding back on the update HAOS 13.2 and HA Core 2024.11.2 because of the boot issue. |
|
@TibsD It should be safe now to upgrade to 14.0.rc1 (and future OS updates) with Kingston eMMC. |
Describe the issue you are experiencing
I recently purchased an odroid n2+ and 32gb emmc module. After flashing
haos_odroid-n2-12.1.img.xz, first with a usb emmc device and then trying to do it via USB connection with the odroid n2+ device. The flash was successful in both cases, but once the device was powered up I couldnt access HAOS via IP or homeassistant.local. I knew the IP was correct as I could compare the mac address.There was no blue light on the board, just a red one. After some trial and error I went back to HAOS 12.0 and it flashed and worked first time. I had a blue light and I could immediately access it through a browser.
What operating system image do you use?
odroid-n2 (Hardkernel ODROID-N2/N2+)
What version of Home Assistant Operating System is installed?
12.1
Did you upgrade the Operating System.
No
Steps to reproduce the issue
haos_odroid-n2-12.1.img.xzto an odroid device...
Anything in the Supervisor logs that might be useful for us?
Anything in the Host logs that might be useful for us?
System information
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: