HOMEPAGE
Contact Search Disclaimer Intranet German
Max Planck Society Homepage
MPS Eingangsseite  

About the Institute

News and
Announcements

Research Areas
Fields of Activity
Staff

Institute Projects
Research Teams

Publications

Research School IMPRS

Services

RAPID Caveats
/ en / projekte / cluster / rapid /

RAPID Caveats Reports

Updated: April 17, 2007

PDF Tables:
> DPU Resets and Patch Corruptions about resets and corruptions
> IIMS HV settings about HV
> Patch Dates including changes to IES LUTs, see also > RAL page
> Times of histogram modes (when IES being tested, no other data available)
> Times of NM3 mode (RAPID in burst mode format during nominal mode)

Contents:

  1. > DPU Reprogramming
  2. > RAPID DPU Resets
  3. > Sun Synchronization
  4. > Bug in new procedure
  5. > Deterioration of IIMS head
  6. > Uploading new patches and IES LUTs
  7. > Patch corruption on turn off
  8. > Central IIMS head improved
  9. > Central IIMS head bad again
  10. > Loss of head 3 on SC4
  11. > Table of HV settings
  12. > Revised time stamps for HSPCT, MDATA
  13. > Revised time stamps for Electrons
  14. > Which spins go into I3DD?
  15. > Some problems April to Sept, 2002
  16. > Autoswitching and the Dead Time Flag
  17. > Missing IES data on SC4, Feb 2004
  18. > Ions only in head 1 on SC4, Jun 2003 to May 2006
  19. > Loss of Ions on SC1, March 2007

DPU Reprogramming (Feb 1, 2001)

As a result of the commissioning analysis, we have discovered a number of bugs in the DPU programming.
  • The ISPCT data which are accumulated and read out every 4 spins is in fact only reset every 8 spins. This means we get identical values on the second set of 4 spins. Effectively we are measuring only every 8 spins.
  • The Direct Events have been programmed without any priority for ion species.
  • The on-board magnetometer data (for generating the EPAD and IPAD pitch angle distributions) are being incorrectly processed. The DPU assumes the vectors are despun when in fact they are always referenced to fixed axes in the spacecraft.
  • The IES look-up tables that determine the electron energy thresholds need to be reset in light of experience from the commissioning phase.
All of these bugs have been repaired by patches uploaded in April and May. See >report below.
 

RAPID DPU Resets (Feb 26, 2001)

The RAPID units are subject to spontaneous resetting of the DPU, probably caused by a Single Event Upset triggered by cosmic rays. It was always expected that such things would happen but the frequency now is disturbing.

To date it has occurred many times since Dec 14, 2000, and has occurred on all 4 spacecraft. See >accompanying table for details.

When RAPID is restarted automatically by the reset, there are two things wrong with the configuration:

  1. The patches are not activated.

    2. Since there no patches at the moment, this is not a big problem.
  2. The IES pedestals are default (wrong) values.

    3. The correct values will have been loaded from the memory but until the next IES look-up table is selected, the values are not in effect. Such a command will be given automatically at the beginning of the next observation period.
  3. The IIMS high voltage power supply is turned off.

    4. This is a serious problem. The standard commanding sequences, planned weeks in advance, usually assume that the power supply is always turned on. Thus we have loss of TOF information until the supply is manully turned on. This means, no ion species analysis.
For the time being, both ESOC and MPAe are monitoring the HK data daily to check for new resets. When they occur, ESOC manually issues the commands to activate patches and turn on power supply.

We are changing the startup procedures as registered at JSOC so that at the start of each observation period the patches are activated and the power supply turned on, whether it is already on or not.
 

Update from Apr 20, 2001

The new procedure went into effect on April 12, but contains a serious bug. See new >caveat below.

Another DPU reset occurred on SC3 on April 13.

Sun synchronization (Feb 26, 2001)

There are two commands to determine the location of sector 0 relative to the sun reference pulse. One of these sets the sector, the other the position within the sector. According to the values that we use, the sun should be located about 1/3 the way into sector 13 (counting 0-15).

We have been puzzled as to why the sunlight that we see seems to be coming predominantly in sector 12, with some fewer counts in sector 13. We have now discovered another bug in the DPU:

When the DPU is turned on and the stored configuration (including sun synchronization) is loaded into memory, the value for the position within the sector is loaded into the software but not into the hardware. This means, the HK data return the expected value, but in fact, it is not being used at all. Only by reissuing the command to set this offset do we get the right value.
On 2001 Feb 21, the sun sector offset was manually set, with the result that the sunlight did indeed move from sector 12 to 13. This command will in future always be issued when RAPID is turned on.
 

Bug in new procedure (April 20, 2001)

The new procedure MP35 that is to reconfigure RAPID at every turn-on in case a DPU reset occurred since the last time, (turn on HV supply, turn up the voltages, set the sun synchronisation) has a bug in it from the ESOC side, starting 2001 April 12. The two commands to set the sun synchronisation are reversed (on SC 2,3,4 only) but the arguments are not. Thus the HK parameters indicate the sun sync is completely wrong. One would expect the sun to be in the middle of sector 2 from these values. In fact, it seems as though the sun is still in sector 13. What is likely to have happened is that the invalid value of the argument to the command was ignored, but it went into the HK parameters anyway. Strange, but similar things have already happened with these parameters. The bug in the procedure has been corrected by ESOC; the sun sync was set to correct value on April 21.

Deterioration of IIMS head? (Apr 20, 2001)

See >separate report.

See >update below.

Uploading the new patches and IES look-up tables (May 18, 2001)

S/C Time of repatching
1 2001 May 18 09:07
2 2001 May 18 12:40
3 2001 Apr 26 15:07
4 2001 May 18 07:21

At the same time, the IIMS live times were changed from 65ms per sector per head, to 60ms, to allow more deadtime for the extra DPU processing. In parallel mode, the live time is thus 180ms per sector.

The considerable delay occurred because additional DPU problems arose when the patch was uploaded, and this involved in more reprogramming. Then spacecraft problems caused loss of all data during the first half of May 2001.

Patch corruptions on turn off (June 1, 2001)

A side effect of the new patches is that they often get corrupted when the RAPID unit is turned off. An >explanation of the effect has been presented by Christian Dierker of IDA.

The consequences are that

  • Whenever RAPID is turned off (for manoeuvers or eclipses) there is a finite chance that the patches are corrupted and at the next turn-on, they are deactivated.
  • Any attempt to activate them leads to a DPU reset, switching off the HV relay.
  • The IES look-up tables are also gone, reverting to the default values, which are totally useless. The pedestal ends up in the middle of the spectrum.
There is no way to avoid these corruptions; a reloading of the patches is necessary each time.

See >Table for a list of these events.

"Corruption" Update, Sep 1, 2003

The patch corruptions are avoided by not turning RAPID fully off during manoeuvres. The high voltages are turned down, only the IES is still operating in the cold standby mode.
RAPID is now only turned off during long eclipses, when the entire SC is powered down.

Central IIMS head improved with higher voltages (Aug 18, 2001)

There were several attempts to increase the voltages on the MCPs in order to improve the gain of the central IIMS head; however, these were done in real time for only a hour or two, often without any particle fluxes to check the results.

Experimental increases of the voltages were then done for entire orbits, as follows:
 

Spacecraft Steps HV increased Date Result Plot (PDF)
4 +1 2001 Jul 16 10:05 Promising > Click
4 +2 2001 Jul 17 16:35 Encouraging > Click
4 +2 2001 Jul 18 to 18:45 Encouraging > Click
1 +1 2001 Aug 4 10:55 Promising > Click
2 +1 2001 Aug 4 10:55 Disappointing > Click
3 +1 2001 Aug 4 11:35 Promising > Click
4 +2 2001 Aug 4 10:55 Encouraging > Click
1 +2 2001 Aug 5 12:15 Encouraging > Click
2 +2 2001 Aug 5 12:15 Disappointing > Click
3 +2 2001 Aug 5 12:15 Encouraging > Click
4 +2 2001 Aug 5 12:15 Encouraging > Click
2 +3 2001 Aug 9-10 Disappointing > Click
2 +4 2001 Aug 15-16 Disappointing > Click

The voltages on SC 1,3,4 were raised permanently by 2 steps as of: 2001 August 18

SC 2 has been left at its previous voltages steps at this stage.

Return of the donut (Dec 10, 2001)

The central heads did not stay improved very long. By Sept 13, 2001, it had gone again on SC1, and the others soon followed.

Donut update (March 29, 2004)

The voltage increase of Aug 24, 2003 did bring back some few counts in the central head on SC1.

Here are 9-month overview plots of the 3 heads on 4 spacecraft:
> Jan 1, 2001 to Sep 30, 2001
> Oct 1, 2001 to Jun 30, 2002
> Jul 1, 2002 to Mar 28, 2003
> Apr 1, 2003 to Dec 31, 2003
> Jan 1, 2004 to Sep 30, 2004
> Oct 1, 2004 to Jun 30, 2005
> Jul 1, 2005 to Mar 31, 2006

Loss of head 3 on SC4 (March 29, 2004)

On June 21, 2003, IIMS head 3 appears to have died on SC4. (See >figure.) This is of a different nature to the donut, which appears to be a problem with the STOP signals. This new problem is more serious, with a loss of directional signals too. And it also appears in head 2! That is, this is a loss in both heads 2 and 3.

A proper explanation has yet to be found.

Table of IIMS HV Settings (Updated Nov 15, 2004)

Listed here are the values of the voltage levels for the Start and Stop MCPs and when they were permanently changed on each SC.
(The experimental changes are listed in the table above.)
Date 1 2 3 4
Beginning 4/5 7/7 4/4 5/5
2001 Aug 18 6/7   6/6 7/7
2001 Oct 16   8/8    
2001 Nov 01 7/8   7/7 8/8
2001 Dec 02   9/9    
2002 Apr 08     8/8  
2003 Aug 24 8/9 10/10 9/9 9/9
2004 Nov 15 9/10 11/11 10/10 10/10

A >full table of all voltage changes is also available.

Time stamping of HSPCT, MDATA, etc (Mar 29, 2004)

Examining the DPU code for RAPID reveals that several non-subcommutated data classes, such as HSPCT, MDATA, are buffered for one spin before being output. (A comparison between MDATA and high resolution magnetic field data had already revealed this discrepancy.) The CSDS and SCI files produced with SW version 5.3 or earlier therefore have time stamps that are 4 s too late. With version 5.4 (from March 19, 2004) this error has been corrected.

The CSDS (prime parameters) data have been reprocessed to correct this.

Time stamping of Electron Data (Dec 22, 2005)

Through various comparisons with other instruments and with spacecraft "noise", it has been determined that the electrons are also wrongly time-stamped, by one spin too late. The CSDS data and SCI files produced with SW version 6.0 and later have this error corrected.

The CSDS (prime parameters) data are correct as of October 1, 2005; the earlier data will have the electron time stamps wrong by one spin (too late) until a reprocessing is carried out to correct this.

Which Spins go into I3DD? (Update Mar 29, 2004)

The I3DD data accummulate over 8 out of 32 spins, in nominal mode. But which 8 spins?
I have updated my >report on this question, as well as the diagram below. (Original >report.)

Some problems from April to Sept, 2002 (Oct 18, 2002)

  1. On SC2, FGM was unable to deliver on-board magnetic data from April 5 to May 24. Thus EPAD and IPAD data cannot be referenced to magnetic data, but rather to the spin axis.
  2. On SC1, the inter-experiment link (IEL) went down on RAPID only, on July 25, at 07:12. Thus no FGM data is available on board, thus EPAD, IPAD data cannot be carried out relative to magnetic field. The link was re-established on Aug 22.
  3. On SC4, the DPU started to do funny things beginning July 30, at 07:40, during real-time patching. As a result, some commands were refused. For example, the DPU continued to format for BM even though the telemetry was in NM. The problem was relieved on August 8, when we rebooted the DPU.

    The data during this interval are just too peculiar. There are repeated EDB numbers, funny flag settings and who knows what else. Processing is difficult. We recommend that these data not be used at all. Version 5.1 of the processing SW blanks this time out entirely, so as to avoid wrong interpretations.

  4. On SC1, Sept. 21, from 01:17 to 04:26, the spacecraft had a problem with the sun sensor, and no sun pulses were delivered to the instruments. RAPID then uses an artificial internal clock to generate EDBs, but it is impossible get the correct time stamps for them. Also, the sectorization bears no relationship to the sun. Thus the RAPID data on SC1 have been suppressed (by the analysis SW) for this time interval.

IES Autoswitching and the Dead Time Flag (Oct 18, 2002)

It has become apparent that the IES autoswitching feature is causing the DPU dead time flag to be triggered frequently. The cause and consequences are given the >accompanying report.

Missing IES data on SC4 (Mar 30, 2004)

On January 18, 2004, the IES on SC4 was inadvertently left in Autoswitching OFF mode after a set of histogram tests. This arose from a timing problem between the command for histogram test and the restoration of autoswitching. As a result, IES on SC4 was permanently in 50us integration time, something that can lead to pile-up effects by higher count rates.

However, on Feb 8, 2004, there was a watchdog reset on SC4 which caused the unit to switch to integration time 2us, still with autoswitching OFF. This means that the count rates are far too low, and the pedestals are too strange to be handled by subsequent software. On Feb 20, following a restart after an eclipse, the autoswitching was restored to ON. The CSDS data for this interval has been suppressed. The SCI data are also removed automatically since the pedestals are funny, except for high count rates around perigee.

Ions only in head 1 on SC4, June 2003 to May 2006

On June 21, 2003, after a manoeuvre switch-off, time-of-flight signals vanished completely on SC4, even in head 2, which was not functioning fully anyway. As a result, there are ion data only in head 1 as of this date.

On May 9, 2006, after a repatching following patch corruption, the time-of-flight signals in heads 2 and 3 were restord. (Head 2 still had the "donut" problem.) After three years of an apparent hardware failure, we have a "miraculous" recovery.

Loss of Ions on SC1 (April 17, 2007)

On March 16, 2007, the MCPs on SC1 started behaving strangely, with loss of time-of-flight signals. The voltages have been lowered several steps to preserve what is left.

See fuller report for details.
top  Top P. W. Daly, 17-04-2007 drucken   Print−friendly Page
© 2006, Max Planck Institute for Solar System Research, Lindau Disclaimer