#LyX 1.4.3 created this file. For more info see http://www.lyx.org/ \lyxformat 245 \begin_document \begin_header \textclass article \language english \inputencoding auto \fontscheme default \graphics default \paperfontsize default \spacing single \papersize default \use_geometry true \use_amsmath 1 \cite_engine basic \use_bibtopic false \paperorientation portrait \rightmargin 3.5cm \secnumdepth -1 \tocdepth 3 \paragraph_separation indent \defskip medskip \quotes_language english \papercolumns 1 \papersides 1 \paperpagestyle default \tracking_changes false \output_changes false \end_header \begin_body \begin_layout Title Generic Operator Discovery \end_layout \begin_layout Date \begin_inset Formula $\today$ \end_inset \end_layout \begin_layout Author Laura Harris and Bryan Newbold for 6.945 \newline {lch,bnewbold}@mit.edu \end_layout \begin_layout Abstract We present a hardware design to convert captured images to an audio stream. There are a wealth of real time \emph on software \emph default implementations of the Fast Fourier Transform (FFT), but we use a Field Programable Gate Array \end_layout \begin_layout Standard \begin_inset LatexCommand \tableofcontents{} \end_inset \end_layout \begin_layout Section Overview \end_layout \begin_layout Standard The Real-Time Audio Composition system allows for the conversion of visually encoded information into audio. A typical visual encoding program, such as Baudline, takes audio input and produces a scrolling visualization of the fast-Fourier transform (FFT). Our system could take one of these visualizations and produce ch serve as the visual spectrograph to interpreted and played back through headphones or speakers. A PS/2 mouse allows for control of the GUI system. Using the mouse, the user can interact with the buttons, the image itself, and a number of other features. An image of the GUI is shown in Figure \begin_inset LatexCommand \ref{fig:gui_photo} \end_inset . \begin_inset Float figure wide false sideways false status collapsed \begin_layout Standard \begin_inset Graphics filename gui_photo.jpg display none width 5in keepAspectRatio \end_inset \end_layout \begin_layout Caption A photograph of the GUI implemented in the Real Time Audio Composition system. Each area of the screen will be explained in detail. \emph on \newline (Source: Team Member) \emph default \begin_inset LatexCommand \label{fig:gui_photo} \end_inset \end_layout \end_inset \end_layout \begin_layout Standard The GUI allows for a number of functions. An edit function, regrettably never fully implemented due to time constraints, would allow for the direct editing of captured images by manually selecting an eight bit binary lved with the time domain audio data to produce the audio signal. The AC97 audio chip in the FPGA runs at a clock speed of 48kHz so the audio samples must also be up-sampled to produce the final signal. \end_layout \begin_layout Standard t vector with at least 1024 bins is required. Aliever, a full wrapper module was necessary to load samples in and out of the module, enable and disable processing, ensure sample index synchronizati on, etc. The ifft_wrapper module ensures that only the 12-bit real components are written and that both the 8-bit phase and 12-bit complex components are tied to 0 at input and ignored at output. \end_layout \begin_layout Standard \align center \begin_inset Float table placement h wide false sideways false status collapsed \begin_layout Standard \align center \begin_inset Tabular \begin_inset Text \begin_layout Standard \family roman \series bold \shape up \size normal \emph off \bar no \noun off \color none Setting \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series bold \shape up \size normal \emph off \bar no \noun off \color none Value \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none Generator Version \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none Fast Fourier Transform 3.2 \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none Input Length \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none 2048 samples \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none Processing Mode \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none Pipelined \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none Sample bitwidth \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none 12-bit signed \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none Output ordering \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none Natural \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none Clock enable pin \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none Enabled \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none Processing Stages \end_layout \end_inset \begin_inset Text \begin_layout Standard \family roman \series medium \shape up \size normal \emph off \bar no \noun off \color none 3 using BRAM \end_layout \end_inset \end_inset \end_layout \begin_layout Caption IFFT Auto-generation Settings \begin_inset LatexCommand \label{tab:ifft_settings} \end_inset \end_layout \end_inset \end_layout \begin_layout Standard onversion is implemented by using each time-domain sample twice, then low pass filtering the output to remove the high frequency artifacts introduced. This low pass filter also conveniently filters out any noise or aliasing occurring in the frequencies above the 720 or so lowest frequencies actually specified by image data. Note that the up-sampling process also effectively halves the frequency corresponding to each bin in our frequency-domain spectral vectors, giving better frequency resolution in the lower, more human discernible regime. \end_layout \begin_layout Standard The low pass filter is implemented almost exactly like in Lab 3, using 31 tap filters generated in matlab using the command: \end_layout \begin_layout Verse \family typewriter round(fir1(30,.5)*(2^13)) \end_layout \begin_layout Standard Each tap is 14-bits signed, thus the scaling by \begin_inset Formula $2^{13}$ \end_inset . The parameter \begin_inset Formula $W_{n}=0.5$ \end_inset specifies a cutoff frequency \end_layout \begin_layout Standard entire packet length is filled (128 values in our case), the IFFT is computed. Natural order refers to the ordering of bits on the output. Reverse ordering of bits is the natural mode of the IFFT, and since timing is not a huge concern at this stage, producing natural order bits is worth the time. The different stages of the IFFT input and output are controlled by an FSM as shown in Figure \end_layout \begin_layout Standard Unless just reset, at which the FSM is immediately clock enabled and put in the setup state so the time domain taps can be initialized, the FSM naturally idles in the wait state. With the edge detection of a button press release within the bode plot, the FSM enters the setup state since it can be assumed that a value within the bode plot has changed. As soon as the ready for data flag is set high, the FSM enters the write state. The values from the bode array are fed into the IFFT, lagging behind three cycles as designated by the datasheet. After all data has been entered, the FSM enters the read state. The resulting tap indices and values are completely synchronous at this point and can be fed directly into the tap manager which will hold the resultant values. \end_layout \begin_layout Standard Memory Management \end_layout \begin_layout Address \emph on Author: Dimitri Turbiner \end_layout \begin_layout Standard The memory control module uses four interface modules to communicate with the Camera, GUI, and IFFT subsystems and two interface modules to control the physical SRAM chips on the LabKit. The camera_to_zbt interface module, keeps track of the sync signals coming from the Camera subsystem in order to compute the memory destination address. Incomming pixels are buffered and packed together into groups of four. When such a group of four is ready, the memory controller is signalled by camera_we. The input signals are all synchronized to the clock in the Camera subsystem -- 27mHz, while the output signals have to be synchronized at the memory clock -- 65mHz. Thus, the camera_to_zbt module has to perform clock resynchronization between the two subsystems it connects. The zbt_to_display, zbt_to_ifft, edit_to_zbt, modules all compute the memory destination address based on the requested pixel horizontal and vertical index (hcount and vcount), and when signalled by the memory controller that data is valid, pack or unpack four 8bit pixels to/from the 36 bit data bus. \end_layout \begin_layout Standard The memory controller controls the two zbt ram access modules depending on which memory IO operations have been requested by the subsystems. The controller supports latching the various subsystem's data and address busses in order to guarantee minimum hold delays and to resolve simultaneous memory access requests. The latches control logic is built around a memory access priority hierarchy: \end_layout \begin_layout Quotation \family typewriter assign main_addr = camera_capture_request ? camera_addr : \end_layout \begin_layout Quotation \family typewriter \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ (display_read_request ? display_addr : \end_layout \begin_layout Quotation \family typewriter \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ (ifft_read_request ? ifft_addr : \end_layout \begin_layout Quotation \family typewriter \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ \InsetSpace ~ (edit_write_request ? edit_addr :0))); \end_layout \begin_layout Standard problem was also noticed after this implementation. When the pixel itself was updated by multiplying the stored bode value by the incoming pixel data, a great deal of scattering across the boundaries of the image was noticed. By pipelining the data within the clocked region, this problem was removed and smooth fading occurred. (Note: the unimplemented edit mode code uses only slightly different math than the bode plot draw function, and should theoretically work, but was never tested). \end_layout \begin_layout Subsection Audio Pipeline \end_layout \begin_layout Standard A number of external tools were crucial for the development and debugging of the audio processing pipeline. An entire separate labkit module with a parameterized dummy sample generator replacing the camera and memory management and no GUI output. This allowed for debugging with consistent, simple, noise free spectral input. This labkit module was also wired to make full use of the hex display (to show bode tap index and both horizontal and vertical memory indexes) and the logic analyzer connections (all 4 16-bit connectors were used!). For instance, Figure shows the IFFT module output of a simple sine wave; the clock-like ticks of channels above and below the sine waveform show the request and ready signals of the various other stages of the audio pipeline. Note that this section of the spectral waveform shows the all zero padding from the windowing module; the single non-zero sample_value is off screen to the left, and the dummy ram module request/ready pins are not toggled. \end_layout \begin_layout Standard A few other buttons and switches came in useful for debugging: Switch #7 disables the regular audio output \end_layout \begin_layout Standard Conclusion \end_layout \begin_layout Address \emph on Author: Tyler Hutchison \end_layout \begin_layout Standard The implementation of a real time audio composition system has been described. Using the GUI interface, coupled with complex memory interactions, and robust audio output, one can use visual information to encode \end_layout \begin_layout Section \start_of_appendix Appendix: Code Listing \begin_inset LatexCommand \label{sub:code} \end_inset \end_layout \begin_layout Subsection ghelper.scm \end_layout \begin_layout LyX-Code //////////////////////////////////////////////////////////////////////////////// / \end_layout \begin_layout LyX-Code // \end_layout \begin_layout LyX-Code // 6.111 FPGA Labkit -- Template Toplevel Module \end_layout \begin_layout LyX-Code // \end_layout \begin_layout LyX-Code // Author: Nathan Ickes \end_layout \begin_layout LyX-Code // \end_layout \begin_layout LyX-Code // Edited: Nov 4, 2008, FTW \end_layout \begin_layout LyX-Code \end_layout \begin_layout LyX-Code \end_layout \begin_layout Subsection discovery.scm \end_layout \begin_layout LyX-Code //Dima Turbiner \end_layout \begin_layout LyX-Code module zbt_to_ifft(input clk, reset, input [9:0]hcount, input [8:0]vcount, \end_layout \begin_layout LyX-Code input [35:0]data_main, input [35:0]data_overlay, \end_layout \begin_layout LyX-Code output reg [7:0]main_pixel, output reg [7:0]overlay_pixel, output [18:0]addr); \end_layout \begin_layout LyX-Code //**** CHECK THIS ADDRESSING \end_layout \begin_layout LyX-Code assign addr = {2'b0, vcount, hcount[9:2]}; \end_layout \begin_layout LyX-Code \end_layout \begin_layout LyX-Code wire [1:0] hc4 = hcount[1:0]; \end_layout \begin_layout LyX-Code \end_layout \end_body \end_document