Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation fault when flashing bitstream into sipeed FPGA #345

Open
wonderfullook opened this issue May 25, 2023 · 15 comments
Open

Segmentation fault when flashing bitstream into sipeed FPGA #345

wonderfullook opened this issue May 25, 2023 · 15 comments

Comments

@wonderfullook
Copy link
Contributor

This is the log when I use the latest openFPGAloader, hope this can be fixed soon.

lee@ubuntu:/mnt/hgfs/share$ sudo openFPGALoader -b tangnano9k nanows2812.fs 
Jtag frequency : requested 6.00MHz   -> real 6.00MHz  
Parse file Parse nanows2812.fs: 
Done
DONE
Jtag frequency : requested 2.50MHz   -> real 2.00MHz  
erase SRAM Done
Flash SRAM: [==================================================] 100.00%
Done
lee@ubuntu:/mnt/hgfs/share$ sudo openFPGALoader -b tangnano9k -f nanows2812.fs 
write to flash
Jtag frequency : requested 6.00MHz   -> real 6.00MHz  
Parse file Parse nanows2812.fs: 
Done
DONE
Jtag frequency : requested 2.50MHz   -> real 2.00MHz  
Jtag frequency : requested 10.00MHz  -> real 6.00MHz  
erase SRAM Done
Segmentation fault
@trabucayre
Copy link
Owner

could you add --verbose ?
I assume it's a tangNano20k instead of 9k?

@wonderfullook
Copy link
Contributor Author

Hi, see this log, I change to use tang primer 20k.

lee@ubuntu:~/Desktop/openFPGALoader$ sudo openFPGALoader -b tangprimer20k -f WS2812.fs --verbose
write to flash
Jtag frequency : requested 6.00MHz   -> real 6.00MHz  
found 1 devices
index 0:
        idcode 0x81b
        manufacturer Gowin
        family GW2A
        model  GW2A(R)-18(C)
        irlength 8
File type : fs
Parse file Parse WS2812.fs: 
checksum 0x941a
Done
DONE
bitstream header infos
CRCCheck: ON
Compress: OFF
ConfDataLength: 1342
ProgramDoneBypass: OFF
SPIAddr: 00000000
SecurityBit: ON
idcode: 0000081b
loading_rate: 212
Jtag frequency : requested 2.50MHz   -> real 2.00MHz  
Jtag frequency : requested 10.00MHz  -> real 6.00MHz  
pollFlag: a0
erase SRAM pollFlag: 80
pollFlag: 80
pollFlag: 80
pollFlag: 80
pollFlag: a0
Done
pollFlag: 20
b 40 16 b read b40160b
Segmentation fault

And this log is flashing SRAM.

lee@ubuntu:~/Desktop/openFPGALoader$ sudo openFPGALoader -b tangprimer20k WS2812.fs --verbose
Jtag frequency : requested 6.00MHz   -> real 6.00MHz  
found 1 devices
index 0:
        idcode 0x81b
        manufacturer Gowin
        family GW2A
        model  GW2A(R)-18(C)
        irlength 8
File type : fs
Parse file Parse WS2812.fs: 
checksum 0x941a
Done
DONE
bitstream header infos
CRCCheck: ON
Compress: OFF
ConfDataLength: 1342
ProgramDoneBypass: OFF
SPIAddr: 00000000
SecurityBit: ON
idcode: 0000081b
loading_rate: 0
Jtag frequency : requested 2.50MHz   -> real 2.00MHz  
displayReadReg 00000020
        Memory Erase
pollFlag: a0
erase SRAM pollFlag: 80
pollFlag: 80
pollFlag: 80
pollFlag: 80
pollFlag: a0
Done
pollFlag: 20
pollFlag: a0
Flash SRAM: [=========================================         ] 80.24%pollFlag: 60e0
Flash SRAM: [==================================================] 100.00%
Done
pollFlag: 6020
displayReadReg 00006020
        Memory Erase
        Done Final
        Security Final

@trabucayre
Copy link
Owner

Ok: seems to have a problem somewhere between gowin class and spiFlash.
Could you share your gateware? At least .fs file but if it's possible the full project?

@wonderfullook
Copy link
Contributor Author

Ok: seems to have a problem somewhere between gowin class and spiFlash. Could you share your gateware? At least .fs file but if it's possible the full project?

This https://github.com/sipeed/TangPrimer-20K-example/blob/main/WS2812/WS2812.fs , and projuct is also here: https://github.com/sipeed/TangPrimer-20K-example/tree/main/WS2812

@trabucayre
Copy link
Owner

I have tried this project with my tang primer 20k. I'm unable to reproduce your segfault. It's not the first time an issue appear with this board and I'm unable to reproduce it (I dislike this situation because I can't fix it).
Maybe it's because I have an early version or because I have a dock light.

could you try with:

ulimit -c unlimited
openFPGALoader xxxx
gdb openFPGALoader core

and with btdisplaying back track?

@alanbarker
Copy link

alanbarker commented Jun 24, 2023 •
edited
Loading

@trabucayre I also have the same problem on the Tang Nano 20K (Ubuntu 20.04 LTS running under a VMWare Workstation VM).

Loading to SRAM works just fine. Selecting Flash however, causes a Segmentation Fault as per the OP.
Here is the output you requested:

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/openFPGALoader...
(No debugging symbols found in /usr/local/bin/openFPGALoader)
[New LWP 4589]
[New LWP 4590]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `openFPGALoader -b tangnano20k -d /dev/ttyUSB1 -f --verbose ws2812.fs.

Program terminated with signal SIGSEGV, Segmentation fault.
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x000055f268a202e4 in Gowin::spi_put(unsigned char*, unsigned char*, unsigned int) ()
[Current thread is 1 (Thread 0x7f7671e9c740 (LWP 4589))]
(gdb) bt

#0  0x000055f268a202e4 in Gowin::spi_put(unsigned char*, unsigned char*, unsigned int) ()
#1  0xa1d005a1d005a1d0 in ?? ()
#2  0x000000000000009f in ?? ()
#3  0x0000000000000000 in ?? ()
(gdb)

EDIT: recompiled with Debug symbols and dumped again (might help a little more?):

Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/openFPGALoader...
[New LWP 6023]
[New LWP 6024]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `openFPGALoader -b tangnano20k -d /dev/ttyUSB1 -f --verbose ws2812.fs'.

Program terminated with signal SIGSEGV, Segmentation fault.
#0  SPIFlash::read_id (this=0xff007ffd993044d0)
--Type <RET> for more, q to quit, c to continue without paging--
    at /home/alan/Apps/openFPGALoader/src/spiFlash.cpp:552
552				if (_verbose > 0) {
[Current thread is 1 (Thread 0x7f563c137740 (LWP 6023))]
(gdb) bt

#0  SPIFlash::read_id (this=0xff007ffd993044d0)
    at /home/alan/Apps/openFPGALoader/src/spiFlash.cpp:552
#1  0x000055d0791395ef in SPIFlash::SPIFlash (this=0x7ffd993044d0, spi=<optimized out>, 
    unprotect=<optimized out>, verbose=<optimized out>)
    at /home/alan/Apps/openFPGALoader/src/spiFlash.cpp:89
#2  0x000055d07917e3e2 in Gowin::program (this=0x55d079d78e60, offset=0, 
    unprotect_flash=<optimized out>) at /home/alan/Apps/openFPGALoader/src/gowin.cpp:443
#3  0x000055d0790b5f3f in main (argc=<optimized out>, argv=<optimized out>)
    at /home/alan/Apps/openFPGALoader/src/main.cpp:586
(gdb)

Second run:


#0  0x00005631473ae432 in Gowin::spi_put (this=<optimized out>, tx=<optimized out>, 
--Type <RET> for more, q to quit, c to continue without paging--
    rx=<optimized out>, len=17) at /home/alan/Apps/openFPGALoader/src/gowin.cpp:759
759					rx[i] = FsParser::reverseByte(jrx[i]>>1) |
[Current thread is 1 (Thread 0x7f3102260740 (LWP 6130))]
(gdb) bt
#0  0x00005631473ae432 in Gowin::spi_put (this=<optimized out>, tx=<optimized out>, 
    rx=<optimized out>, len=17) at /home/alan/Apps/openFPGALoader/src/gowin.cpp:759
#1  0x00005631473ad8bd in non-virtual thunk to Gowin::spi_put(unsigned char, unsigned char*, unsigned char*, unsigned int) () at /usr/include/c++/9/ext/new_allocator.h:89
#2  0x000056314736b2b4 in SPIFlash::read_id (this=0x7ffda3b06070)
    at /home/alan/Apps/openFPGALoader/src/spiFlash.cpp:539
#3  0x000056314736b5ef in SPIFlash::SPIFlash (this=0x7ffda3b06070, spi=<optimized out>, 
    unprotect=<optimized out>, verbose=<optimized out>)
    at /home/alan/Apps/openFPGALoader/src/spiFlash.cpp:89
#4  0x00005631473b03e2 in Gowin::program (this=0x563148e7d520, offset=0, 
    unprotect_flash=<optimized out>) at /home/alan/Apps/openFPGALoader/src/gowin.cpp:443
#5  0x00005631472e7f3f in main (argc=<optimized out>, argv=<optimized out>)
    at /home/alan/Apps/openFPGALoader/src/main.cpp:586

Finally, might be relevant, might not, but I had to run openFPGALoader as root (Sudo) since no amount of faffing around on my Ubuntu would let openFPGALoader open the /dev/ttyUSB without it. Not dailup group, not tty group, nothing. Again, Flashing SRAM works just fine using sudo.

@alanbarker
Copy link

alanbarker commented Jun 24, 2023 •
edited
Loading

Hmm, some extra context that might help narrow down the issue:

If I upload to SRAM first (successfully), then immediately change the destination to the Flash, openFPGALoader gives me a Stack Smashing error* instead of a SegFault (dump below). Further attempts to upload to Flash revert to the original SegFault error as previous comment.


#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
--Type <RET> for more, q to quit, c to continue without paging--
[Current thread is 1 (Thread 0x7f49c02dd740 (LWP 6098))]
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f49c0304859 in __GI_abort () at abort.c:79
#2  0x00007f49c036f26e in __libc_message (action=action@entry=do_abort, 
    fmt=fmt@entry=0x7f49c049908f "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x00007f49c0411aba in __GI___fortify_fail (
    msg=msg@entry=0x7f49c0499077 "stack smashing detected") at fortify_fail.c:26
#4  0x00007f49c0411a86 in __stack_chk_fail () at stack_chk_fail.c:24
#5  0x000055f67d42f940 in non-virtual thunk to Gowin::spi_put(unsigned char, unsigned char*, unsigned char*, unsigned int) () at /usr/include/c++/9/ext/new_allocator.h:89
#6  0x000055f67d3ed2b4 in SPIFlash::read_id (this=0x7fff4388d9f0)
    at /home/alan/Apps/openFPGALoader/src/spiFlash.cpp:539
#7  0x000055f67d3ed5ef in SPIFlash::SPIFlash (this=0x7fff4388d9f0, spi=<optimized out>, 
    unprotect=<optimized out>, verbose=<optimized out>)
    at /home/alan/Apps/openFPGALoader/src/spiFlash.cpp:89
#8  0x000055f67d4323e2 in Gowin::program (this=0x55f67ddfb520, offset=0, 
    unprotect_flash=<optimized out>) at /home/alan/Apps/openFPGALoader/src/gowin.cpp:443
#9  0x000055f67d369f3f in main (argc=<optimized out>, argv=<optimized out>)
    at /home/alan/Apps/openFPGALoader/src/main.cpp:586

Seems like it is the same stack trace from call number 5 down but interesting that this is different after a successful upload to SRAM.

@trabucayre
Copy link
Owner

Is it possible to test without vmware ? Sometime a VM may (or not) introduces issues with USB?

@alanbarker
Copy link

alanbarker commented Jun 24, 2023 •
edited
Loading

your suspicions seem to be correct; it works when tested on my bare-metal Linux box.

Whilst I have successfully loaded to the Flash without issue now, I can't say for sure that this is the problem since I didn't have a Ubuntu 20.04 LTS bare-metal box lying around. Instead I had to use a Fedora FC36 Workstation I had on an old dev machine. I did however, recompile from source (same source as the Ubuntu) rather than use the loader that is in the Fedora repos so that at least levels some of the test parameters.

It could still be an Ubuntu 20.04 problem and the VM being anywhere from irrelevant or partially involved.
I aim to eliminate the VM by doing the same install from a Fedora VM and seeing if I can isolate this problem to a distribution.

UPDATE ----

Installed a fresh Fedora Workstation 38 in a VMWare Workstation and built from source. Loader successfully flashes the SRAM and the Flash chip.

Looking like this is a stock Ubuntu 20.04 LTS issue....

@trabucayre what distro/platform are you running/developing on?

@alanbarker
Copy link

tldr; I've solved my problem (and maybe 'the' problem): make sure the build instructions are followed to-the-letter.

I think this is the UDEV rules. Remember my needing to use sudo to get the app to run because of a failure to open the ttyUSBx device? I upgraded my Ubuntu to 22.04 but that exhibited the same problem (segmentation fault). However, issue #245 caught my eye since this is the error I used to get if I ran the app without sudo.

Following that issue's comments, I noticed my udev rules folder didn't have the 99-openfpgaloader.rules file (as per the build instructions). Not sure why this was the case as i'm sure i followed, step by step! However,

Following the udev section from the Ubuntu build instructions:

sudo cp 99-openfpgaloader.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger # force udev to take new rule
sudo usermod -a $USER -G plugdev # add user to plugdev gro

and now running the system without sudo, resulted in a working app that flashes both to SRAM and Flash.

Seems like the issue was related to udev, device permissions and the subsequent use of sudo when running openFPGALoader. Usually always a sign of an issue if userland stuff needs to be run with sudo....

@wonderfullook would you be interested in verifying any of my findings above and perhaps we can close this off for Gwenhael?

@trabucayre
Copy link
Owner

It's really interesting and may make sense to update FAQ by adding a section.
I don't know why this fails using sudo...
I have to try by myself on all distro I have to check this behavior.

Thanks for this information!

@jorgenkraghjakobsen
Copy link

jorgenkraghjakobsen commented Jul 22, 2023 •
edited
Loading

Hi All
Tangnano20k
Bare metal on Ubuntu 22.04 - openFPGALoader compile from source - I can program device, but flash only works 1 out of 3 times.

jakobsen@t490s:~/work/fpga/gowin/project_20k/fpga_project/impl/pnr$ openFPGALoader -v -b tangnano20k  fpga_project.fs -f
write to flash
Jtag frequency : requested 6.00MHz   -> real 6.00MHz  
found 1 devices
index 0:
	idcode 0x81b
	manufacturer Gowin
	family GW2A
	model  GW2A(R)-18(C)
	irlength 8
File type : fs
Parse file Parse fpga_project.fs: 
checksum 0xb9f4
Done
DONE
bitstream header infos
CRCCheck: ON
Compress: OFF
ConfDataLength: 1342
ProgramDoneBypass: OFF
SPIAddr: 00000000
SecurityBit: ON
idcode: 0000081b
loading_rate: 0
Jtag frequency : requested 2.50MHz   -> real 2.00MHz  
Jtag frequency : requested 10.00MHz  -> real 6.00MHz  
pollFlag: 60a0
erase SRAM pollFlag: 4080
pollFlag: 4080
pollFlag: 4080
pollFlag: 4080
pollFlag: 4080
pollFlag: a0
Done
pollFlag: 20
b 40 17 b read b40170b
Segmentation fault (core dumped)

and all good :

akobsen@t490s:~/work/fpga/gowin/project_20k/fpga_project/impl/pnr$ openFPGALoader -v -b tangnano20k  fpga_project.fs -f
write to flash
Jtag frequency : requested 6.00MHz   -> real 6.00MHz  
found 1 devices
index 0:
	idcode 0x81b
	manufacturer Gowin
	family GW2A
	model  GW2A(R)-18(C)
	irlength 8
File type : fs
Parse file Parse fpga_project.fs: 
checksum 0xb9f4
Done
DONE
bitstream header infos
CRCCheck: ON
Compress: OFF
ConfDataLength: 1342
ProgramDoneBypass: OFF
SPIAddr: 00000000
SecurityBit: ON
idcode: 0000081b
loading_rate: 0
Jtag frequency : requested 2.50MHz   -> real 2.00MHz  
Jtag frequency : requested 10.00MHz  -> real 6.00MHz  
pollFlag: a0
erase SRAM pollFlag: 80
pollFlag: 80
pollFlag: 80
pollFlag: 80
pollFlag: a0
Done
pollFlag: 20
b 40 17 b read b40170b
Segmentation fault (core dumped)
jakobsen@t490s:~/work/fpga/gowin/project_20k/fpga_project/impl/pnr$ openFPGALoader -v -b tangnano20k  fpga_project.fs -f
write to flash
Jtag frequency : requested 6.00MHz   -> real 6.00MHz  
found 1 devices
index 0:
	idcode 0x81b
	manufacturer Gowin
	family GW2A
	model  GW2A(R)-18(C)
	irlength 8
File type : fs
Parse file Parse fpga_project.fs: 
checksum 0xb9f4
Done
DONE
bitstream header infos
CRCCheck: ON
Compress: OFF
ConfDataLength: 1342
ProgramDoneBypass: OFF
SPIAddr: 00000000
SecurityBit: ON
idcode: 0000081b
loading_rate: 0
Jtag frequency : requested 2.50MHz   -> real 2.00MHz  
Jtag frequency : requested 10.00MHz  -> real 6.00MHz  
pollFlag: a0
erase SRAM pollFlag: 80
pollFlag: 80
pollFlag: 80
pollFlag: 80
pollFlag: a0
Done
pollFlag: 20
b 40 17 b read b40170b
Detail: 
Jedec ID          : 0b
memory type       : 40
memory capacity   : 17
EDID + CFD length : 0b
EDID              : 1740
CFD               : 0b 40 17 0b 40 17 0b 40 17 
b 40 17 b read b40170b
Detail: 
Jedec ID          : 0b
memory type       : 40
memory capacity   : 17
EDID + CFD length : 0b
EDID              : 1740
CFD               : 0b 40 17 0b 40 17 0b 40 17 
RDSR : 00
WIP  : 0
WEL  : 0
BP   : 0
TB   : 0
SRWD : 0
RDSR : 00
WIP  : 0
WEL  : 0
BP   : 0
TB   : 0
SRWD : 0
flash chip unknown: use basic protection detection
Erasing: [==================================================] 100.00%
Done
Writing: [==================================================] 100.00%
Done

@trabucayre
Copy link
Owner

The segmentation fault is really weird again.
I see a core is dumped: could you provides backtrace using gdb ?

gdb openFPGALoader core

followed by bt.

@jorgenkraghjakobsen
Copy link

jorgenkraghjakobsen commented Jul 24, 2023 •
edited
Loading

When i comment out _spi->spi_put(0x9F, NULL, rx, len); in spiFlash.cpp around line 550 I do not get seg_faults any more
The info read by the spi_put call is not used it looks like.
/Jørgne

printf("Read Exteted --------------------------------------\n");
		/* read extended */
		if ((_jedec_id & 0xff) != 0) {
			has_edid = true;
			len += (_jedec_id & 0x0ff);
			printf("will call spi_put len = %d\n",len);
			int spi_ok; 
			//spi_ok =_spi->spi_put(0x9F, NULL, rx, len);
			printf("spi_put return %d\n",spi_ok);
		}

trabucayre added a commit that referenced this issue Jul 24, 2023
@trabucayre
spiFlash: temporary disable read extended (#345)
8ffa024
@trabucayre
Copy link
Owner

Thanks for your feadback: I have disabled this code section.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@alanbarker @trabucayre @jorgenkraghjakobsen @wonderfullook