Repository navigation
BEE3 Top Design
This page describes the bee3_top design which defines all the interfaces required to implement networks on the Microsoft BEE3 board, such as clock management, chip-to-chip communications and debugging via chipscope.
The BEE3 board contains 4 Virtex-5 FPGAs which are connected in a ring via 72-wire parallel interfaces as shown in the diagram below. (I believe that there are also diagonal connections between the FPGAs but this has not been tested yet). Also available, but not currently in the example code, are CX4 board-to-board connections and Boris' serial links to PC. For full details of the BEE3 board, see the links here
The example design in the NetEmulation/BEE3_top folder performs the following functions:
- Generates clocks required for the chip-to-chip and general logic from the 100MHz crystal oscillator on the BEE3 PCB. The current clock frequency is 150 MHz but this can be changed by modifying the clock_manager.sv module;
- Implements a simple 64-bit counter;
- Sets up the chip-to-chip links with each of its neighbouring FPGAs;
- Forwards the counter to each of its neighbouring FPGAs over the chip-to-chip links;
- Displays the values of the local and neighbouring counters in chipscope along with status signals. A 4 chip network design can be built on the BEE3 by replacing the counter with the network functions from NetEmulation.
As we want to primarily use SystemVerilog and ISE does not support it, Mentor Graphics Precision Synthesis (PS) is used for initial synthesis and ISE is used for map, place and route and bit file generation. Although this is a bit cumbersome, Precision Synthesis should give better results than the Xilinx XST synthesiser, i.e. higher clock frequencies. The process described below uses bot guis - probably a more efficient process can be found by using script files or integrating ISE into PS or the other way around. Anyway, this works for now!
(1) In a terminal, navigate to the BEE3_top folder and open PS:
cd "path to project folder"
mentorps &
Create a new project (in the project tab on the left). In the Design tab, click on 'Setup Design'. If using the BEE3 board, select Technhology = VIRTEX-5, Device = 5VLX110TFF1136 and Speed Grade = 3.
Click on 'Add input files' and select bee3_top.sv, counter.sv, ring32b_bidir.v and bee3_top.sdc. Repeat and add the three .edif files in the 'ipcore_dir' folder - these are the precompiled chipscope modules.
(2) Click 'Synthesize' and wait for it to complete. There are a few warnings which need investigation but they do not seem to affect the final result.
(3) In the terminal window again, open Xilinx ISE:
ise &
and open the ise bee3_top project. The ucf file containing the pin assignments is already in the project.
(4) Select Project/Add Source menu options. Enter the bee3_top_impl_1 folder. Select bee3_top.edf and bee3_top.ucf. Click open and ok. The edf file should now be the top module (with the triangular hierarchy symbol) with the ucf files below it.
(5) In the processes box, double click on 'Generate Programming File' and wait for completion.
(6) For the next steps, you need to be connect to 'trigger' which is connected to the BEE3 board via USB cable. Either work in the CONNET lab or connect via ssh:
ssh -X trigger
Open impact. The full path is:
/apps/Xilinx/14.6/ISE_DS/ISE/bin/lin64/impact &
Double click on 'Boundary Scan'. Click on the 'initialize chain' icon (four green squares connected together). Answer yes to assign configuration files. We do not use the first chip in the chain, so click 'Bypass'. The other four chips are the FPGAs - apply the same bee3_top.bit file to all of them (answer no when asked about SPI devices). Click 'Ok' on the summary window.
(7) Right click on each of the FPGAs in turn and select 'Program' and 'Ok'.
(8) Open chipscope. The full path is:
/apps/Xilinx/14.6/ISE_DS/ISE/bin/lin64/analyzer &
Select File/Open project menu item and open the bee3_top.cpj project file. Click on the 'initialize chain' icon (this time, black squares connected). Chipscope should detect the 5 chips again. Three windows should be open for the first FPGA - a trigger setup, waveform and VIO console. From the VIO, you can control the resets for the chip2chip and counter. Click on the waveform window and click on the 'T!' icon in the top bar. Use the + icon to zoom in until you can read the counter values. The RingStatus signals should both be 6 which means that the delay calibration for the chip2chip interconnect has completed and normal operation has started. The PLL_Locked signal should be high. The counter_local signal shows the value of the counter on the first FPGA. The counter_up and counter_down signals show the values of the counters on the neighbouring FPGAs communicated over the chip2chip interfaces. ou can also look at the chipscope units on other FPGAs, for example, in the project box, double click on the DEV2/UNIT0/waveform item. However, setting up the busses is rather tedious - if you do this make sure that you save the project so you only have to do it once!
Firstly, you need to decide which network functions will be implemented on each FPGA. Unless it is a symmetrical topology like a mesh/torus, different designs will be required for each FPGA, e.g. in the case of the optical network, one FPGA may represent the central switch and arbiter, the others network nodes. In parallel, you need to decide how you will use the chip2chip interface, e.g. what communication is required between the FPGAs and how you will share the connection between all modules which need to use it. The current chip2chip links can communicate 64-bits (32 wires double data rate (DDR)) in each direction.
Deciide if you need Xilinx IP modules in your design, e.g. for memories. If so, do the following: (1) Open the bee3_top projects in both ISE and PS as before.
(2) In ISE, select the Project/Add Files menu option. Select IP and choose a name for the module. Click 'Next' and follow the process to set up the options for the module.
(3) After generation of the IP, it will be part of the ISE project. But PS needs an EDIF file for the module. You can convert using the ngc2edif application. In a terminal, navigate to the ipcore_dir folder and type:
/apps/Xilinx/14.6/ISE_DS/ISE/bin/lin64/ngc2edif -bd asis -w <IP_name>o.ngc <IP_name>.edif
The ipcore_dir should now contain an EDIF file for the IP.
(4) In PS, add the EDIF file to the project
(1) In PS, open the bee3_top.sv file. Delete the counter and add your network modules and signals. Connect the relevant signals to the din/dout ports of the chip2chip modules.
(2) Modify the clock frequencies in clock_manager.sv if necessary. It is convenient to keep the network clock (clk) at the same frequency as the chip2chip interface, but I have not tested how fast the chip2chip will run.
(3) Modify the chipscope connections so that you can observe network signals in chipscope.
(4) Run the synthesis in PS and map/place and route in ISE as before. Check the place and route report in the design summary. You should see the messages 'All constaints were met' and 'All signals were completely routed' at the end if the process has run sucessfully.
(5) Generate bit files, program the FPGAs and open chipscope as before. You will need to setup the signals and busses in the chipscope project which is quite tedious.
Good luck!
Philip Watts, 6th December 2013.