Dear all,

I updated the TAS to include the upcoming user runs up to and including the winter shutdown. As you can see, there will only be access to the XTDs in cases of urgency. In these cases, please contact Martin Dommach, Harald Sinn or Thomas Tschentscher.

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules&listViewLinkBack=true

Cheers, Gerd

General

• Recomissioning of the FEL has been bumpy and up to this point only very weak lasing could be achieved. 
• So far SPB had only little beam and FXE had no beam at all. Both instruments state that the commissioning time for the up coming user run is insufficient

Instruments

• FXE:
  • First SW and DAQ stress test on Friday went well.
  • Some problems appeared with the slow data that should be fixed by now
  • K. Wrona states that these problem originated from missing permissions due to change of userid
• SPB 
  • Entire instruments needed re-alignment with beam since the CSS was moved during the shutdown
  • Very low intensity and not very stable beams were available on the Weekend. The beam could not be properly focussed
  • The bender for M2 cannot reach the flat condition. Is this the temperature problem again?
  • One pump was overheating
  • AGIPD tests with new bunch patterns started this morning
  • Significantly more time needed before user operation
  • There were some problems with the interlock system that trigger beam aborts

Tunnel

• Commissioning program was very limited in the last few days
• KMONO got some first interesting results
  • Rings and Kicks could be seen
  • Some wrong undulator settings were identified

Detectors

• Firmware update was done to record all 30 bunches. However this still has problems
• Need beam for calibrations
• The planned update by ITDM of the DAQ was not yet done

Advanced Electronics

• The mentioned interlock issues in SPB require an update to loop 7
• In loop 4 need to reset the "all valves open" signal by hand if it trips
• The SPB Master loop needs updating both for MPS and timing

Software

• Tests of the scantool took place with hardware. These tests continue today
• A new data model is being worked on for the scan tool but this is not available yet
• SPB states that at this point they will not make use of the scan tool as they first have to consolidate the instrument
• FXE is involved in the CAS tests

ITDM

• As mentioned before the DAQ version is still old and needs an update. This contains many improvements
• IBM discovered a bug in GPFS and published a patch. The bug could lead to data corruption
• Decision: DAQ update and GPFS patch will be installed today in the morning.

General

• There is currently no beam less than 24h before user operation is supposed to start in SPB
• The affects everybody (Instruments, Detector, Laser 

General

• User operation is delayed by 48h due to rocky startup of FEL.
• FEL still has not reached acceptable levels of stability and power

Instruments

• SPB:
  • Instrument was brought back to a somewhat running state
  • There are still problems with motors in the sample chamber
  • Alignment with beam is ongoing at this time
• FXE
  • New scan tool was tested
  • Currently working on user-specific alignment tasks
  • Laser was routed to the sample position
  • The beam is still drifting around

Tunnels

• FEL had trouble to find lasing
• There was a controled access on Wednesday to fix various problems. However, further problems appeared
• It was decided to roll back the changes made for a new injector laser, but this did not improve the situation
• At some point 400 uJ were reached and lost again
• There are many problems with the undulator system
  • Incorrect values in lookup tables
  • Difficult operation of the entire system
• FXE should be able to start up tonight
• SPB would follow Saturday morning since time is needed for alignment during the day today
• Shifts for photon diagnostics were largely cancelled due to FEL problems
• KMONO gave some interesting results pointing to the problems in the undulator system
• Next week FEL Imager and KMONO will be installed in SASE2

Laser

• Both SASE1 instruments have laser beam with local timing. 
• Tests with machine timing start today at lunchtime

Detectors

• AGIPD is in best possible state. 
• Firmware and DAQ were upgraded
• Some commissioning time with beam is needed today
• LPD had a cooling leak  and a power problem last night
• At 6 pm yesterday, this was still not fixed

AE

• Continuing small activities for user operation

ITDM

• A small bug was found in the DAQ that affects only slow devices  with certain train ids
• It is proposed to fix this during the next break. Decide on this Monday

CAS

• Dedicated squads for the setups of SPB/FXE, DOOCS integration, Scan Tool closed now
• Tools delivered but with significantly reduced scope
• All necessary things for DOOCS Karabo integration were delivered. Now waiting for DESY

General

• No user operation.
• During the week end the accelerator was not able to deliver reliable and stable beam for the user operation.
• The M2 mirror bender stopped working on Saturday morning and had to be replaced. The mirror and blender were replaced as a whole durig the week end. Normally it would have taker weeks but a "spare" system was already set up for SASE2 and could be used on a short notice. Pumping of the M2 section is ongoing and leak test be performed today.
• The non working M2 does not show apparent problems so far, and the reason why was not working is not clear.
• Some sections of the LINAC were warmed up due to some maintenance required for the cold compressors.
• LINAC is being cooled down and ready for operation today afternoon.
• The technical support groups were not informed about the problems until saturday afternoon. The communication channels in these cases have to be improved.

​Instruments

• SPB and FXE:
  • The goal is to be able to strat the secondo users run on Thursday (Day FXE, Night SPB).

Tunnels

• XTD2,9 Ready to take beam in the afternoon (if no leak in M2 is discovered).

ITDM

• A small bug was found in the DAQ that affects only slow devices  with certain train ids
• It is proposed to fix this during the next break. Decide on Wednesday

Dear all,

I received updates from DESY (M. Hüning) about how the access to the southern tunnels is being restricted in the next months, due to interlock tests etc., and included it in the TAS. Please check, if you have work to be done in these tunnels:

https://docs.xfel.eu/share/page/site/PSC/wiki-page?title=Tunnel_access_schedules

Cheers, Gerd

Instruments

• SPB:
  • Waiting for beam to come in a usable condition (500 uJ and pointing within aperture of optical elements)
• FXE
  • Similar

AE

• There was an interrupt of the vacuum loop in XTD2 due to a faulty pump controller. This was fixed by ZZ
• Rotational SmarACT stage and slits in the sample chamber for SPB are being worked on

CAS

• Due to change of beam schedule, now working on implementing more devices in DOOCS-KARABO for Jan Grünert

ITDM

• Will upgrade DAQ software today. Note: Still based on KARABO 2.1.16

General

• Beam is back with good intensity (up to 1 mJ) 
• Next user period starts on  23 Nov
• Time between now and then should be used to optimize FEL, beam lines and Instruments

Instruments

• FXE:
  • Measurements of pulse intensity were done using an XGM from DESY in the hutch
  • A  pulse intensity of ca 40 uJ was found. This is in rough agreement with the expectations
  • Spectrometer was aligned
  • GreatEyes and Gotthardt detectors are working fine
  • LPD still to be checked
• SPB:
  • Using beam to do alignment
  • Not such a good focus (too big)
  • Plans are to continue with instrument commissioning today and Saturday
  • Work is currently being done by Mahadi on the SmarACT rotary stages
  • FXE points out that most likely SPB operated with the FXE CRLs inserted into the beam
  • Note: The standard "switching script" from Harald should be used to make sure parameters and devices are properly adjusted when beam is switched from one instrument to the other
  • Detector calibration on Sunday

Tunnels

• Last Friday there was a Bender failure on mirror M2
• The mirror was removed from the tunnel during the weekend and the SASE2 spare was inserted 
• In the process one Beckhoff controller was found broken
• In the removal process a further problem was caused in one of the Beckhoff loops
• This problem could be fixed from outside the tunnel 
• After this 20h were spent to redo Beam-Based Alignment in 7 iterations
• The new orbit deviates from the old by up to 80 um.
• After using the new orbit, the FEL lased quickly with good intensity
• The replaced mirror M2 could be re-aligned 
• On Thursday beam was available to FXE
• Remark from J. Grünert: Instruments should make sure that most up-to-date DOOCS panel is loaded to see proper XGM info
• Update of DAQ caused DAQ in tunnels to be broken. This is now fixed according to K. Wrona
• HIREX commissiong is going in parallel 

Detectors

• AGIPD:
  • Before the bender failure a few X-Rays could be taken on AGIPD
  • It looks like 30 X-Ray pulse can be taken now
  • Additional flat field calibration is needed now
  • On Sunday a full shift is foreseen for commissioning with beam
• LPD:
  • One quadrant was disassembled to find the cooling leak.
  • Leak now appears to be fixed
  • No sign of dead spots on detector

Laser

• We are operating in shifts:  early / late, although experiments have been cancelled, performing shift change and other excercises for commissioning.
• Operating the laser with machine timing was so far hampered by the fact that the transmitted laser patterns from the machine were limited to the actual number of x-ray pulses, i. e. mostly non-deterministic 1 or 0. Optical alignment and characterization is not possible under those conditions. Therefore, on the weekend we have switched to local timing to give guys some power.
• Following this experience, we have decided to attempt local generation of laser patterns under machine timing in order to be independent of the machine and not have to switch back and forth between local and machine timing. This has been implemented and is under test today and over night. We still observe spurious long term drifts, which are under investigation.  However, if finally successful, this can only be a make shift solution for the coming weeks. Mid to long term we need proper implementation by DESY, allowing pattern changes on short notice and independent of the machine. This is in preparation (time line ca. mid 2018).
• Currently, beams are sent to FXE and SPB with 30 pulses each, 800nm broadband, should be compressible to 15fs. This is under machine timing but with locally generated patterns.
• There are other curiosities concerning machine timing, which are related to the increased T0 arrival time jitter compared with our local timing. Debugging is ongoing. We are also testing various features under above conditions (machine timing, local pattern):  phase shifter, trigger shifter, shift change. Time line: depending on outcome, daily updates will be provided.

AE

• When the breakdown of the interlock occured in conjunction with the broken Beckhoff terminal it became obvious that crucial emergency phone numbers are not available
• Better emergency procedures are needed
• The special EPS operation mentioned to align the mirrors should be written down so that it can be implemented an no override of the interlock system is needed
• The list of shutdown activities compiled by PSPO contains a lot of work for AE. This is now being discussed. 

ITDM

• Some problems occured this week with DAQ of slow data. This should have been fixed yesterday

CAS

• Doing 24/7 OCD support 
• During switch between instrument CAS comes in to provide support
• The cancellation of user operation last weekend was not properly communicated so that people came in unnecessarily
• Next week additional work for WP74 is being done

General

• Intensity and availability of beam was good over the weekend
• Large drifts and jitter were observed that are significantly higher than previously
• Beam has two horizontal lobes as if it is too large for the mirrors

Instruments

• SPB:
  • Many tasks were achieved during the weekend, e. g. instrument alignment, detector calibration
  • Issue with encoders whose cables are close to motor cables persists. These encoders are being disabled
  • Rotational SmarACT stages still do not work and are needed for this week's user experiment
• FXE
  • At shift start on Saturday after reloading beam parameters there was no beam in the hutch.
  • Beam had to be brought back by hand. This was a large effort
  • This situation did not repeat on Sunday
  • LPD detector had problems: Only one super module correctly recorded data. Comment from S. Hauf: FXE should have called the detector expert
  • PP laser synchronisation does not work properly yet: Increments when adjusting timing are too coarse. Scan range is limited to 18ns

Detector

• Detector group was not informed about the LPD issues in FXE. 
• AGIPD calibrations were successfully taken on Sunday

ITDM

• Still work is needed for slow data DAQ
• DAQ behaves differently in SPB and FXE. In FXE it appears overloaded and in SPB it works fine
• Comment from CAS: 
  • Many changes were made in the KARABO framework to allow asynchronous operation
  • These changes have to be also pulled through to DAQ. This will only be done during the winter shutdown
  • Until then, DAQ is very vulnerable to any delay in the system

CAS

• No further comments

AE

• Currently working on problems with rotational stages in SPB

General

• User experiments resume tomorrow morning
• Beam should be coming back late this morning or early this afternoon
• After the Operation Coordinaton Meeting H. Sinn, L. Samoylova and the instruments will align the mirrors
• Beam to the instruments should be available directly after this

Instruments

• SPB:
  • The last night with beam on Monday was used to take flat field calibrations with AGIPD
  • The timing for overlap with the optical laser was adjusted
  • 5 out of 6 SmarACT rotational stages are now operations. Some problems remain and AE is talking to the manufacturer
• FXE
  • Request for use of the beam after 9 PM
  • Can be given to SPB starting at 5 AM tomorrow for final setup 

Detectors

• Connection to the ONDA system was set up for the detectors
• Significant latency develops quickly an saturates at approximately 2 mins

ITDM

• Simultaneous use of the DAQ system by both instruments should be done carefullly

AE

• no further comment

CAS

• no further comment

General

• User operation ongoing in SPB/SFX and FXE

Instruments

• FXE:
  • User operation started during the night
  • Some concerns over running LPD as the sample chamber is from diamond and direct reflections could cause damage
  • Generally instrument worked very well
  • Synchronization with PP laser is now in a state where it can be tested
• SPB
  • Many data taken for detector calibration
  • Severe issues with KARABO
    • Response is very slow
    • Major crash this morning
    • Detector controls complex and instable
  • A failure of AGIPD related to FPGA not understood
  • Strange interlock issue: EPS interlocks appeared disabled

Tunnels

• L. Samoylova spent many hours on Wednesday to set up the DAQ in the tunnel
• With the help of many support people she managed to get the system to work
• The system is instable and also seems to record many more parameters than are needed
• Use of the system is very clumsy
• Comment from ITDM (L. Maia): The data sources were specified by the users. It is acknowledged that the system needs a lot more work
• Diagnostics is working with CAS in a squad to work down a list of open redmine tickets
• The DOOCS - KARABO bridge is now working. The FEL imager is routinely available in DOOCS to the BKR crew
• DESY set up a statistics server to help collect data about beam pointing instabilities

Laser

• Local pattern generation synchronized to machine timing is working
• Overlap finding together with FXE was done but has to be tested under real condition next week
• A few suddent shutoffs of the front end occured. The origin might be understood
• There are large timing drifts in the ADCs that are currently not understood (up to ms)

ITDM

• Next Wednesday there will be a 1h shutdown of the load balancer in the DESY RZ. This should affect mostly the web services
• It should be noted that the DAQ ran quite stably in SPB and FXE. More than 20 runs were collected in each case without a restart

CAS

• Two hot fixes are out (2.1.18.4/5). These address  very specific issues
• A major new release (2.2) is planned for the winter shutdown. This will be mandatory
• Comment from instrument: It must happen early so that wrinkles can be ironed out!)

General

• Preparation for the user's run starting tomorrow (until 7:00 Tuesday morning)

Instruments

• FXE:
  • On Monday night the DAQ seemed to be slow. This problem was caused by a misconfiguration in the DAQ which could be solved by ITDM once they were informed. In general both the Configuration understanding by the users and the communication with ITDM has to improve.
  • The Gotthard detector could not be used. The Greateyes detector is being used now and works properly.
  • PP laser is now synchronized but not commissioned yet. This is going on now (crucial point is the pulse width measure). It is not yet clear whether the PP laser will be used in the next users run. 
• SPB
  • AGIPD was moved 5 m downstream for the next experiments. Vacuum is OK. Alignment checked with the ALAS, will be checked with beam probably today evening.
  • A change in the DAQ configuration has to be (is being) done today (CAS/ITDM)

ITDM

• Upgrade switches today at 9:00 will be checked if there were problems right after this meeting.

General:

• User Operation ongoing until Tuesday 5 Dec 8 AM
• CAS shall present a release plan for KARABO 2.2 next week
• Dispatch meetings shall move to a Mo/Wed 13:00 schedule starting next Wedneday 6 Dec

Instruments

• FXE:
  • Scantool was successfully used to establish overlap with laser beam (Tangerine) 
  • Some problems with slow DAQ in particular also for tunnel components
  • Comment from ITDM: There are known issues with the slow data acquisiton. A new version exists that was intentionally not deployed before the end of UOP
• SPB
  • Just finished first shift of laser user experiment
  • No big problems
  • All goals in the optimistic schedule were achieved

Tunnels

• Currently preparing for the shutdown
• Sticking condition of mirror bender could be reproduced in the lab
• WP74 just installed the last missing pieces in XTD1

Laser

• Running
• temporal and spatial overlap with beam is achieved
• But now instruments do not intend to use the PP laser for the last experiment

Detectors

• Both detectors are working
• The DAQ for slow data is crashing occasionally
• Gotthard is fully operational but there problems getting the data out. Help from PSI is needed

AE

• No problems

ITDM

• No problems

CAS

• CAS-XPD Running:
  • We will finish next Monday
  • We have been addressing issues related to MCP, Karabo-DAQ, Cameras, and K-Mono
  • Deploy K-mono today
  • Review DAQ Groups in Karabo: Cyril (interface to ITDM), Liubov, Koch
  • Gotthard we have to address in a separate and more systematic discussion: trigger, operation, calibration...
• XGM memleak - we already have a fix to get it stable but have to check when we will be able to upgrade (only device).
• Internal discussion about CAS/Karabo priorities for the winter shutdown

General:

• User operation with good beam conditions over the weekend
• One interlock trip in the FXE branch in XTD9. Restored by CAS
• Shutdown starts tomorrow at 08:00.
• Next dispatch meeting takes place Wednesday at 13:00. A reminder will be sent.

Instruments

• SPB:
  • Started with the actual sample during the night shift
  • Some DAQ issues
  • 108TB data was recorded
  • Problems appear to occur between 01:00 and 02:00  and between 13:00 and 14:00 
• FXE
  • Running is ok. 
  • Temporal and spatial overlap with optical laser established
  • Currently taking pump-probe data
  • People on shift were uncertain about how to recover from interlock trip
    • Statement from Vacuum: Pumps can restarted/valves open if the respective button in KARABO can be clicked

CAS:

• Delay between data taking and "online" analysis in the "Hummingbird" software is about 0.3s

ITDM

• On Friday, there was a global network outage affecting the Office Network on the DESY site.
• The transponder system in the experiment hall was down
• SRP organised a guard at the hall entrance so that the hall had not to be evacuated
• The control network (XDAC) was not affected

General:

• Shutdown started.

Instruments

• SPB:
  • AGIPD prepared for exchange modules intervention
  • AE is in contact with SPB for crates issues fixing
  • The schedule for next year, especially when the experiment has to be ready before the users run has to be known as soon as possible.
• FXE
  • Starting loops update in the next days. Reason for the small delay is that FXE is checking all the default parameters to be able to restore them after the update.
  • In order to solve some DAQ issues, FXE requested to have their own topic. AE agrees, ITDM agrees, CAS is trying to find out how to do it and to do it before the next users run. Technical details have to be discussed offline with FXE.

ITDM

• in KW 51 (18-22 Dec.) the DAQ and online cluster will not be available due to recabling works in the balcony rooms.
• Update of the servers in the experiments has to be done. ITDM requested to know from CAS when the major update of Karabo is taking place to coordinate the effort and minimize the downtime. CAS will present a schedule concerning Karabo this Friday.
• 700 Tb of data are stored on the online cluster. It has to be communicated by the experiments what can be moved from there, since there is almost no space left.

Diagnostics

• Implementation of Gotthard detector in Karabo is still a problem. S.Hauf will invite the main groups involved for a meeting to find a solution.
• The data correlation with time stamps when using the Karabo DAQ has to be done carefully and contacting WP74 since there are still issues present. 

AE

• Activities on the Hirex in SASE1 will need the MOV loop to be brought down in the afternoon. An entry in the ELOG will be written.
• SPB loop 8 prepared. 
• Full list of new crates belonging to the motion loop in SASE3 tunnels has to be delivered to AE
• Full list of new crates belonging to the IPS loop in SASE3 tunnels has to be delivered to AE in order to be able to close the loop.
• Technosoft controller issues in SASE3 will be attacked once the SASE1 activities will allow it.

VAC

• VAC testing ongoing.
• 2 mirrors will be installed before Christmas
• Motion crates for the motion lopps and test of the VAC interlock will happen this week.

The Motion loop in SASE1 is totally operated again.

Please note following info from DESY:

Author: F. Saretzki



Ab sofort werden die Rack-Löschanlagen in den südlichen Tunneln in Betrieb genommen. Die Anlagen bleiben danach scharf.



 

General

• Starting from next week shutdown activities will be tracked using M. Aresté's list
• Dispatch meeting on Monday and Wednesday at 13:00
• Next Friday Readiness meeting also at 13:00 because of conflict with XDAC

Instruments

• FXE:
  • Currently focussing of implementing all default parameters into the next firmware release
• SPB:
  • Major detector work with AGIPD disassembly started

Tunnels

• Currently testing firmware and software in SASE3
• Installation of first mirror starts next week. Mirror installation takes about 1 wk/mirror
• CRL in XTD2 will be vented to align internal mask and fix stuck arm
• Drilling takes place at the end of XTD9 for the installation of the FXE diamond grating
• Mirror cooling will not be installed this shutdown. It is not needed for the planned bunch rates and might introduce unwanted effects/vibrations
• Activities of Diagnostics:
  • Photonics Science camera in SASE1 exchange.
  • Tests are being carried out in SASE3 together with J. Tolkiehn
  • Cabling is going on in XTD1 on the XGM
  • There is a noise problem on the Gotthardt detector in the HIREX. Need support from AE to find the origin

AE

• Motion loop in SASE3 will be split into mov1 and mov2. This will allow to sort components into more and less stable sets and keep one of the loops up most of the time

Laser

• Laser activities had to start late due to problems with AC and laser interlock (delays of 6 and 5 months respectively)
• Availability of trigger pattern from the machine was also a problem
• Specs could just be reached for the Nov user period
• Eventually the PP laser was not used for any of the experiments
• Work on the laser is far from finished and will require 3 more months to come to full funtionality
• Currently, the Karabo version is being upgraded and the most up-to-date PLC firmware is being installed
• These updates had intentionally been delayed until the end of the user period.

Detector

• Detector group not present at the meeting
• Information from SPB: major work is ongoing on AGIPD

AE

• Train builder is being tested as it was not used so far. This required re-cabling
• PLC motor loops are also being split in SASE1
• Many activities in SPB
  • Remaining modifications of modules have started
  • Export of input for firmware generation for loop 8 has been provided by EETF
• Waiting for go in FXE
  • Need to work on the interface for the EPS system
• No tasks currently from laser group
• New information available on SmarACT stages: The observed problems arise from a cable problem and can be solve by configuration
• The problems with movers in the sample chamber in SPB (non-functioning encoders) arise from cable cross talk at the vacuum feed through. This require hardware re-design
• In SASE2 there are tests coming up together with the diagnostics group

ITDM

• Train builder tests are scheduled next week
• Recabling of Infiniband network happens on 18/19/20 Dec. No DAQ and controls available on these days
• A summary of DAQ statistics was shown (see slides on INDICO)

CAS

• Release of KARABO 2.2 on 18 Dec
• Freeze of functionality on 21 Jan
• Discussion reveals that a discussion on specific deployment dates and needs of different groups is urgently needed (Instruments, Tunnels, DAQ)

FYI

Author: F. Saretzki

English version below

Ankündigung:

Aufgrund von Sachverständigenprüfungen wird es am folgenden Termin zu Auslösungen des Evakuierungsalarms kommen:

Gebäude: XHE2/XS2, XHE4/XS4, XSDU1, XTD1, 3, 5, 6, 7 und 8
Datum: 13.12.2017 ca. 09:00 Uhr – 11:00 Uhr


Sie brauchen bei ertönen der Sirene in diesen Zeiträumen das Gebäude nicht verlassen.

============================================================================
Announcement:

Due to expert testing there will be an evacuation alarm on the following date:

Building: XHE2/XS2, XHE4/XS4, XSDU1, XTD1, 3, 5, 6, 7 and 8
Date: 13th of December 2017 between 09:00 a.m. and 11:00 a.m.


You don’t need to exit the building while the alarm occurs.

 

General

• Meeting on Friday will take place at 1300 due to overlap with XDAC
• M. Areste shall ask for updates on task list on Thursday

Instruments

• SPB
  • Work on AGIPD is ahead of schedule. Closing of detector should happen at the beginning of next week
  • Additional installations for roughing pump system is going on and will probably finish this year

Tunnels

• Diagnostics group currently working in the Southern tunnels, cabling components and aligning imagers
• Colleagues from Russia are coming 3 - 20 Jan to work on MCP in SASE3
• KMono local tests in SASE3 should also happen in early Jan
• Diagnostics requests scenes for XGM in XTD1

AE

• Vacuum loop in SASE3 will be modified. Some components shall be moved to a separate loop, e. g. Soft Mono
• SPB tunnel loop will be re-deployed to include the attenuator
• The motion loop in SASE1 will also be split to allow more rapid updates for newly integrated components
• Z-motion crate for detector in SPB will be done until next week or early next year

CAS

• Currently working with AE and ITDM on refining release schedule for KARABO
• Testing the version for release next week. This will then first be used by ITDM and is needed for the firmware upgrade of AGIPD
• Work is going on on the XGM train ID problem
• FXE will be moved to a separate topic (broker) to disentangle some dependencies at least temporarily
• FEL imager performance problem was investigated. This runs on a server that is very busy and should be moved to a different server