Quick Start
Contents
The TE-cooled single OTA CCD microcam cameras we have are extremely valuable for device testing, board testing, and standalone software experiments in our lab. On occasion, we have also sent these to other institutions to demonstrate the STARGRASP controller system. The following instructions would be what we send along with the camera. Even without a microcam in hand, this section gives the reader an idea of what is involved in plugging in a STARGRASP controller and getting it to take a picture.
Unpack.
- Metal frame holding:
- Microcam with shutter and Nikon lens
- Controller chassis with STARGRASP cards
- Compact Excelsys power supply
- Ethernet Switch with fiber ports
- Cheapo notebook computer
- Some cables and fibers.
Cable up and power on.
- Connect power to the "Dell 2724" Ethernet switch first (it can take a minute or two to initialize.) (If you are using a media convert instead, connect and power that up.)
- Connect a twisted pair Ethernet cable between the switch and the notebook's RJ45 jack. (Note: do not try to connect to your own Ethernet infrastructure, or try to enable the wireless on the Notebook just yet. For this demo, everything will function like an island.)
- Connect the notebook AC power adaptor and boot the notebook into "Linux" (the default.)
- Remove dust covers and plug in the orange fiber between the STARGRASP and the fiber module in the switch. Both ends should click into place, with minimal force required. (Take care not to chip the exposed ends of the fibers as they are vulnerable when they're out. A spare fiber patch cord is included.)
- Do not connect AC power to the camera just yet. We'll want to bring up the necessary software that receives log messages first, so it can tell you if a fan is blocked or something else is going wrong.
Build the source code.
- If you browse around for files on the notebook, you'll notice that there are 2 source directories in /gpc/
- /gpc/src/
- /gpc/src-ppc405/
We leave it as an (easy) exercise for you to build the code, including the embedded code that runs on the controller. Here's how. Don't worry if you screw anything up. We have a backup. This is your playground, to learn about the system. Open a Terminal. "root" is the only user on the notebook, so just use that, and run:
notebook# cd /gpc/src notebook# configure --prefix=/gpc notebook# make world
Now there are a set of tools in /gpc/bin/* which we'll use in the next section. If you change any of the software, re-running "make world" (or just "make install" from the subdirectory where you changed stuff) will quickly find and rebuild the changed parts. If you don't have an "SREC" file with your microcam, then you get to build the embedded software too:
notebook# cd /gpc/src-ppc405/u-boot notebook# make
Now there is a small file /gpc/src-ppc405/u-boot/u-boot.srec freshly built by the PowerPC cross-compiler which you will load onto the controller in just a minute.
Bootstrap -- DHCP
You're about to turn on microcam. When it comes up, there is a limited version of the u-boot.srec already built in to the hardware. We have chosen to have this initial "stage 1" version only handle the digital functions of the board, including the network. Then, over that network connection, you will load a more capable "stage 2" version which you just compiled in the previous step. (In the future, we may place a stage 2 directly on the controller, or allow the controller to pull one via TFTP rather than this push model, but this works well for engineering.)
The STARGRASP controller behaves like any typical Ethernet appliance. It will bring up its Ethernet link and send a standard broadcast discovery for DHCP. (In the future, we could make it possible to program a static IP into the device, but DHCP is easier to manage when you have as many boards as we do.) Your Ethernet MAC address will be of the form:
00:50:C2:64:90:xx
We have included a DHCP server on the notebook which will answer, for the purposes of this demo. (The server is not "authoritative", and will only answer to the STARGRASP's MAC address, so it would not interfere with any existing DHCP infrastructure at your site.) The IP addresses should end up as follows:
192.168.20.1 compaq3 # (an alias address, running on eth0:0) 192.168.20.2 stargrasp90xx ucam
The DHCP server starts automatically when you boot the notebook (try "ps -ef | egrep dhcpd" to see it running.) It will give the STARGRASP two important IP addresses: (1) its own, 192.168.20.2, and (2) where it should send logging messages, using the standard "next-server" feature of DHCP. In this case (2) is the notebook, 192.168.20.1. STARGRASP controllers will attempt to connect to this IP on port 55500 + their "chassis ID." Hardware sets the chassis ID, and for microcams we use "15". So if DHCP works as it is set up in this demo, the STARGRASP is immediately going to try to log messages to port 55515 on the notebook.
Logging -- grasp_receiver
We have started DHCP for you, but to let you see how all the STARGRASP-specific stuff works, we leave it to you to start the log receiving service on port 55515 before powering up. Open a Terminal on the notebook and run one of the programs you compiled earlier:
notebook# /gpc/bin/grasp_receiver 55515
Leave that window up, and make it at least 132 columns wide. (Normally, we simply pipe the output of grasp_receiver into another program which colorizes, line wraps, and does other fancy things but here you get to see the raw operations to learn about the system.)
If you want, open another Terminal with "tail -f /var/log/messages" where you'll see the DHCP activity logged.
Now plug in microcam! In a few seconds, you should see this in the grasp_receiver window:
stargrasp90xx: https://svn/gpc/repos/sw/trunk/detector/ppc405/u-boot rev 2073 on stargrasp90xx: https://svn.ifa.hawaii.edu/gpc/repos/hw/hdl/stargrasp_v2/trunk/fpgapcb/ rev 1479 stargrasp90xx: status:stargrasp90xx: Network logging established. stargrasp90xx: Instruction Cache is ON stargrasp90xx: Data Cache is ON (DCCR=f0000000 DCWR=00000000) status:stargrasp90xx: DAQ3U self-test... stargrasp90xx: DRAM 256 MB........OK (10666Mbps/10666Mbps read/write) stargrasp90xx: SRAM 0 KB........NONE stargrasp90xx: BRAM 80 KB........OK (358Mbps/403Mbps read/write) stargrasp90xx: DS-OCM 16 KB........OK (1658Mbps/1658Mbps read/write) stargrasp90xx: LM86 Temperatures...OK (48C CPU, 34C Board) stargrasp90xx: Network IP Addr.....OK (192.168.20.2 from DHCP) status:stargrasp90xx: TCP listening ports 23(ucash) and 915(ucash) info4:stargrasp90xx: x.2 up 17s|cpu 48C|board 34C|dev0 0C|dev1 0C|malloc 4k|stack 1k|free 261755k|
If you intercept these messages, you're supposed to translate these "status:" "logonly:" "warning:" "error:", and "progress:" type messages into something meaningful for your own specific choice of logging system. It is not necessary to do so to detect if a command passed or failed. The messages are just extra info for the human operator (or pathologist.)
Booting Embedded "stage 2" -- grasp_boot
Now use another tool in the /gpc/bin/ toolkit that was just compiled, to load the "stage 2" embedded code cross-compiled in /gpc/src-ppc405/. Open a Terminal on the notebook:
notebook# /gpc/bin/grasp_boot 192.168.20.2 /gpc/src-ppc405/u-boot/u-boot.srec
The selftest feedback the grasp_receiver window will look similar, but new entries for the analog parts of the board will show up:
stargrasp90xx: DAQ3U DAC...........OK stargrasp90xx: DAQ3U CLK...........OK stargrasp90xx: DAQ3U ADC...........OK
After the selftest, the STARGRASP will continue to report its stage2 uptime. In the case of OTAs, the temperature and package ID(s) will be reported in the "info" line as well. You may get some warnings about DAC calibration, and inability to read temperatures from "dev1". (When running OTAs, each STARGRASP board can run 2 of them with 8 channels each, and we call them "dev 0" and "dev 1". In your demo setup, "dev 1" is not connected to anything, as shipped.)
Demo Logging System
Click on the grasp_receiver Terminal and press Control-C. We're going to restart it in a more manageable way:
notebook# /gpc/bin/director -i5 -p /gpc/bin/grasp_receiver 55515
After a few seconds, STARGRASP will notice that the connection is broken and reconnect to the new instance of grasp_receiver. All of the "logonly:" prefixed messages are now hidden, but you can reveal them by typing "verbose on" in that Terminal window. This command will be accepted even when no prompt is visible in the Director window.
Applying Power to the Detector -- grasp_pon_ota
It's time to use another utility in the /gpc/bin/ toolkit. This one re-applies voltages and clocking patterns from a previously taken OTA FITS image. Similar tools should be written for more generic CCD types, but this one enforces the voltage limits specified in the "OTA Bible" (though, there is an override.) Any FITS file saved by the standard STARGRASP toolkit will contain full header info for reproducing the voltages and timing of the clocking patterns used to get that image. We have provided a default image, taken in our lab, that you can use to power on the detector. Before doing this, take a look at the image itself first with a FITS viewer, such as "ds9", just to see what we're talking about:
notebook# ds9 -mosaicimage /gpc/microcam.fits
Other than the image itself, you can also look at the header with ds9. The interesting stuff is all in the Primary Header Unit:
CLV_PPG4= 'ecbb:cbb2:bb2e:65d8:5d97:38ba:6622:3154' / Coded parallel timings CLV_PG3 = '4c13:c130:1304:4094:0942:0421:d993:0124' / Coded serial timings CLV_PG4 = '0c26:c060:0903:3134:09c1:001f:3732:08a2' / Coded SW/RST/VCLAMP/SAMP CLV_ADC = '1500 ' / Coded ADC parameters CLV_TRIG= 0 / Cross-trigger phase delay (x 10nsec) CLV_PRES= 0 / Serial prescan, (skipped in addition to CNPIX1) CLV_PIPE= 1 / Clocking pattern pipeline (in addition to ADC) CLV_PXTP= 0 / OT Pixel Type 1 or 2 (or 0 for normal CCD) ADC_CONF= '0c8:0c8:0c8:0c8:0c8:0c8:0c8:0c8' / ADC configuration ADC_MUX = '0c0:0c0:0c0:0c0:0c0:0c0:0c0:0c0' / ADC MUX configuration ADC_PGAR= '000:000:000:000:000:000:000:000' / Red ADC pga (adcgain 1.0) ADC_PGAG= '000:000:000:000:000:000:000:000' / Grn ADC pga (adcgain val) ADC_PGAB= '000:000:000:000:000:000:000:000' / Blu ADC pga (adcgain val) ADC_OFSR= '000:000:000:000:000:000:000:000' / Red ADC offset ADC_OFSG= '000:000:000:000:000:000:000:000' / Grn ADC offset ADC_OFSB= '000:000:000:000:000:000:000:000' / Blu ADC offset DAC_INIT= 1600 / DAC internal vref offset voltage (millivolts) DAC_SW_L= -3000 / Summing Well LO (millivolts) DAC_SW_H= 6000 / Summing Well HI (millivolts) DAC_S1_L= -3000 / Serial 1 LO (millivolts) DAC_S1_H= 6000 / Serial 1 HI (millivolts) . . .
Feel free to experiment on your own device, but please do not change the voltages on our microcam (even though, the grasp_pon_ota C program ought to prevent you from doing anything bad to it.) WARNING: Some of the voltages get scaled from the raw millivolt values shown in the header. Refer to the preamp datasheet, and use the following "grasp_pon_ota -s" tool to check what the final voltages applied to the detector will be! This program knows about the correct scaling factors.
The keywords are intended to be looked at by tools. We'll show the one that edits clocking patterns (the CLV_ stuff) later. You can view the voltages with the tool, using the "-s" (summary) option, or with no -s it will print a more verbose listing of the voltages:
notebook# /gpc/bin/grasp_pon_ota /gpc/microcam.fits Parallel P1 Low,High : 0.5V, 10.0V Parallel P2 Low,High : 0.5V, 10.0V Parallel P3 Low,High : 0.5V, 10.0V Parallel P4 Low,High : 0.5V, 10.0V Parallel Standby Low,High : 0.5V, 10.0V Serial S1 Low,High : -3.0V, 6.0V Serial S2 Low,High : -3.0V, 6.0V Serial S3 Low,High : -7.0V, 6.0V Summing Well SW Low,High : -3.0V, 6.0V Reset Gate RG Low,High : -2.0V, 10.0V Logic LREF, VSS : -1.0V, -0.5V Logic VDDL, VDDH : 6.0V, 16.0V Substrate SUB : -10.0V Scupper SCP : 20.0V Source SO : 4.1V Output Gate OG1 : 0.0V Drains DR,RD : 19.5V, 12.4V warning: CCID58-1-26a1 P2_L = 0.5V is 1.0V below optimal range.
A "warning:" will not stop you from applying the voltages, but an "error:" will. Limits are defined in the program, and based on our OTA documentation.
Apply the voltages!
notebook# /gpc/bin/grasp_pon_ota -a -c 192.168.20.2 /gpc/microcam.fits
Take an exposure -- grasp_cmd
Microcam's shutter can be controlled by STARGRASP. Here's how to clean, expose, and readout:
Do this first since it might be quite flooded when you first power on:
notebook# /gpc/bin/grasp_cmd 192.168.20.2 clean 20
Then a normal command sequence:
notebook# /gpc/bin/grasp_cmd 192.168.20.2 etype o notebook# /gpc/bin/grasp_cmd 192.168.20.2 etime 200 notebook# /gpc/bin/grasp_cmd 192.168.20.2 clean notebook# /gpc/bin/grasp_cmd 192.168.20.2 expose notebook# /gpc/bin/grasp_cmd 192.168.20.2 readout dev=0 bpp=16
Now the full image is in controller memory. There are a couple of ways to look at it. The following is just nice for a demo...
Web Browser Quicklook
Open firefox Web browser on the notebook and tell it to connect to the controller: http://192.168.20.2/
In the case of microcam, you can change settings, exposure type, time, and take new images. Clicking on a cell zooms in. Images load slowly, partly because the notebook only has 100 Mbit Ethernet, but also because the TCP stack we are currently using in the embedded system is not very fancy.
Saving a FITS -- grasp_save_ota
We use our own UDP protocol to bulk-download the image. Yet another tool from /gpc/bin/ implements that. (Once you do save the FITS image, it is freed from controller memory and no longer available in the quicklook browser view.)
Before saving the file, send a command which, with the 100 MBit Ethernet notebook, will help performance of grasp_save_ota and the video example below:
notebook# /gpc/bin/grasp_cmd 192.168.20.2 udp freq=8000
The flow/congestion control in our UDP protocol is primitive, and it is generally best to set the maximum frequency at which it will send data packets explicitly like this. In the case where you have a server that is much more powerful than the controller, and can keep up with Gigabit traffic, throttling the UDP traffic is not necessary. Now issue a command to transfer the file to the notebook:
notebook# /gpc/bin/grasp_save_ota -d 0 192.168.20.2 /tmp/dev0.fits
Note that we have included OTA-specific versions of the grasp_ toolkit since you are interested in OTA support when using microcam. A generic version grasp_save C program also exists, but it does not know about cell mosaic geometry, so you definitely want to use grasp_save_ota. (You might need to specify additional options to get it to work with an STA OTA device since there will be no package ID for the tool to read. Try something like "-n ccid58a" if yours are similar to CCID58s.)
Video Mode -- grasp_vidgrab
This is another OTA-specific tool, though it should work for shutterless video on any CCD. It will just be more efficient on OTAs because of the design (small cells) and because it uses OT-shifts to get the region of interest to the output amplifier faster (if you set the OTA "pixel type" to I or II.)
Take 50 frames of video from cell xy22:
notebook# /gpc/bin/grasp_vidgrab 192.168.20.2 50 xy64 > /tmp/video.fits
This file can be viewed and played back with ds9:
notebook# ds9 /tmp/video.fits
No "-mosaicimage" option this time -- this is a basic FITS file, not MEF. It is 3d though, and ds9 will provide a slider and the ability to play back the frames at various rates.
Normally we would want to do video with a faster timing pattern than the 2500nsec/pixel one we gave you in /gpc/microcam.fits. Try the test again with a 1200nsec/pixel rate, as was used in the file /gpc/video.fits:
notebook# /gpc/bin/grasp_pon_ota -a -c 192.168.20.2 /gpc/video.fits notebook# /gpc/bin/grasp_vidgrab 192.168.20.2 100 xy64 > /tmp/fast_video.fits
Note: The Video FITS files current don't have some geometry keywords that grasp_pon_ota expects. Ignore the error about being unable to set DETSIZE.
Again, you can view this file with ds9 after recording it.
We have more tools ready for video mode, including network tools that can broadcast the video, take centroids, remove the vertical streaks that occur from the lack of any shutter, and a live display tool that can correct for the asymmetric binning, among other things. The particular region (1/4 of a cell) and the binning mode are just the example mode built in to grasp_vidgrab for now. The controller supports many other modes and possibilities.
Timing Patterns -- cestlavie
If you're wondering what the clocking looks like, you should be able to feed any FITS file to a graphical clocking editor we created called "cestlavie". Some older versions didn't read FITS headers, but you can always cut-and-paste 4 keywords onto its command line too:
CLV_PPG4= 'ecbb:cbb2:bb2e:65d8:5d97:38ba:6622:3154' / Coded parallel timings CLV_PG3 = '4c13:c130:1304:4094:0942:0421:d993:0124' / Coded serial timings CLV_PG4 = '0c26:c060:0903:3134:09c1:001f:3732:08a2' / Coded SW/RST/VCLAMP/SAMP CLV_ADC = '1500 ' / Coded ADC parameters
gets translated to invoking cestlavie with:
notebook# /gpc/bin/cestlavie ppg4=ecbb:cbb2:bb2e:65d8:5d97:38ba:6622:3154 pg3=4c13:c130:1304:4094:0942:0421:d993:0124 pg4=0c26:c060:0903:3134:09c1:001f:3732:08a2 adc=1500
or, for newer versions:
notebook# /gpc/bin/cestlavie /gpc/microcam.fits
Try it now to bring up the timing pattern. (Cut and paste from the header of /gpc/microcam.fits if you want.) If you change something and want to apply it to the controller, simply type the controller's name or IP address in at the bottom and click Send.
Performance
Some aspects of performance will be severely limited because we're using a cheapo notebook. A server costing under 3K$ can easily handle a full STARGRASP chassis worth of OTAs (8 OTAs) with >2TB disk included in the price.
A recent STARGRASP controller upgrade option allows the possibility for 512MB of on-board ECC DDR (compared the 256MB your demo has.) With 256MB, there's already plenty of room for dual-OTA operation though.
Another performance trick is the ability to run "grasp_save" any time during the readout. We use this regularly now with GPC1 and read 8 OTAs per chassis into a single server within a hundred or two milliseconds of the end of a 7 second readout. Noise performance was completely unhurt by doing this. We can actually read over 6 full 1400 Megapixel bias frames per minute to a set of only 8 1U servers indefinitely, or until the disk fills up. And if not also busy doing a readout, each controller boardset is capable of outputting 37 Mpixels per second, or 150 per chassis (2400 Mbits/sec.) Detector performance and controller performance are continually improving as we develop more advanced code.